A Danish company's English pages are almost always the last part of the site to be discovered, and a page that has never been crawled cannot rank, cannot be measured and cannot be fixed by better writing. Discovery comes before everything else.
The case study nobody could find
A cleantech supplier outside Copenhagen sells heat-recovery units to industrial buyers in Germany, Poland and the Netherlands. Not one of its customers is Danish. Last spring it published eleven English case studies, each documenting an installation with the equipment model named in the heading. Good pages, written by someone who understood the equipment.
Nine months later a specifying engineer in Hamburg searched for exactly that model number and found a distributor's thin product listing, a trade forum thread and nothing from the manufacturer. The pages were live. Anyone with the link could read them. Google had never fetched four of them, and had fetched three others once, months earlier, and had not returned.
Nothing in the company's analytics said so. The domain looked healthy, because the Danish side was being crawled steadily and the aggregate figures reported a site in good order. The commercially decisive half of the site was simply absent, and absence produces no signal at all — no impressions, no clicks, no line on a chart, nothing to notice.
Discovery, crawl and index are not one event
People speak about getting a page "into Google" as if it were a single act. It is three, and a page can fail at any of them independently.
- Discovery. A crawler has to learn the URL exists. That happens through a link from a page already known, through a sitemap entry, or through a direct submission. No route, no knowledge.
- Crawl. Knowing about a URL is not the same as fetching it. Known URLs sit in a queue and are fetched in an order the crawler decides, which may be tomorrow or may be never.
- Index. Having been fetched, the page still has to be judged worth storing. Thin, duplicated or near-identical pages are fetched and then quietly discarded.
- Serve. Only an indexed page can be retrieved for a query, and only then does anything you would recognise as optimisation begin to matter.
Almost all published advice addresses the fourth stage. The eleven case studies failed at the first and second, which is why every hour spent rewriting them would have produced exactly the same result.
The English section is discovered last, and for structural reasons
This is not bad luck. A bilingual Danish site is built in a way that systematically starves its English half of the signals discovery depends on.
The Danish ecosystem does not link to it
Trade press, industry associations, suppliers and local directories link to the Danish pages, because that is what they read.
- Danish URLs accumulate external references for years
- English URLs launch with none and often never gain any
A hundred URLs appear on the same afternoon
Translation projects finish all at once, so the whole section arrives as a single block rather than accumulating gradually.
- Bulk publication looks less like organic growth
- Nothing about the batch signals which page matters most
Reachable only through a flag icon
If the only path to the English tree is a language selector in the header, internal linking gives crawlers almost nothing to follow.
- Depth from the home page grows quickly
- Deep translated pages become effectively orphaned
Structurally identical to the Danish tree
Mirrored templates, mirrored navigation and mirrored page structure make the second tree cheap to dismiss.
- Near-duplicate layouts invite a shallow crawl
- Pages that differ only in body text are fetched less often
Each factor alone is survivable. Together they describe most English sections on Danish domains, and the result is a tree that a crawler knows about vaguely, visits rarely and re-visits later than anything else on the host.
Crawl budget is attention, not an allowance
The phrase misleads people into imagining a quota that can be topped up. It is closer to attention: how much fetching a crawler is willing to spend on your host, decided by how quickly your server responds, how much of what it has already fetched turned out to be worth keeping, and how much external interest suggests the site is worth revisiting.
Two consequences follow, and both are uncomfortable. The first is that the Danish and English trees compete for the same pool. Every fetch spent on a paginated Danish archive is a fetch not spent on an English case study. The second is that you cannot buy your way out with volume: publishing forty more English pages divides the same attention into smaller portions.
There is a third consequence particular to small markets. A Danish domain is rarely large and rarely heavily linked by international standards, so the pool being divided was modest before anyone split it in two. A company with a few hundred URLs and a handful of external references is not operating with the crawl attention of a national publisher, and the English branch is competing for slices of an already small allocation.
This is also where the hreflang pairing between the two language versions quietly fails. Those annotations tell a search engine that two URLs are alternates of one another, but they only do useful work once both URLs are actually indexed. An hreflang tag pointing at a page that was never fetched declares a relationship to something the engine has no record of, which is why a correctly configured pair can still leave the English version absent from results while the Danish one performs normally.
| Symptom | Likely gate | Wrong response | Right response |
|---|---|---|---|
| English page has zero impressions, ever | Discovery or crawl | Rewriting the copy | Check whether it was ever fetched, then submit it |
| Fetched once, never returned to | Index judgement | Submitting it again weekly | Make the page substantively different from its Danish twin |
| Danish twin indexed, English one not | Internal linking | Assuming a language penalty | Link the English page from somewhere other than the switcher |
| Whole section indexed slowly after launch | Bulk publication | Waiting patiently | Submit in batches with the commercially important URLs first |
| Indexed, but a thin distributor page outranks you | Serving | More submissions | Now, and only now, is this a ranking problem |
Putting URLs in front of a crawler deliberately
Waiting for organic discovery is a reasonable strategy for a site with a link ecosystem. For an English section that has none, deliberate submission shortens a wait that might otherwise not end. The indexing tools inside the Semalt panel exist for exactly that gap, and they are worth understanding precisely, because the limits are part of the design rather than an afterthought.
Submission with a per-URL record of what happened
For sites where a section of the tree has to be pushed rather than waited for.
- A daily allowance per account. 1,000 URLs a day, which forces the prioritisation an export site needs anyway: the eleven case studies before the eleven hundred archive pages.
- Bulk handling. Up to 10,000 URLs in a single batch, so a freshly translated section can go in as one operation rather than as a week of manual work.
- Delivered through IndexNow. Submissions travel over the IndexNow API, reaching GoogleBot and BingBot rather than a single search engine.
- A log per URL, not per job. Bot visit with a timestamp, status, error detail, plus live counters for submitted, found and failed URLs. This is the part that turns a guess into a fact.
The daily ceiling is worth thinking of as a feature. It obliges you to answer a question most sites never ask: if only a thousand of your URLs could be looked at today, which ones would you choose? For an export-led business the honest answer is rarely the ones the content management system would have listed first.
Sitemaps, parsed properly rather than trusted blindly
A sitemap is the other way to declare that URLs exist, and on a bilingual site it is usually the neglected one. Many Danish sites generate a sitemap covering the main tree and quietly omit or truncate the translated branch, which means the section most in need of announcement is the one being announced least. Sitemap handling in the Indexing Hub is built to expose that rather than assume the file is complete.
Recursive parsing to a documented depth
For anyone whose sitemap is really an index pointing at other sitemaps.
- Two ways in. Upload the file directly or hand over its URL, whichever suits a site whose sitemap is generated on the fly.
- Nested indexes followed. Parsing is recursive to three levels, which covers the index-of-indexes arrangement most shop and publishing systems produce.
- Volume per job. Up to 1,000 sitemaps in one job, enough for a portfolio of domains handled together rather than one at a time.
- Queue discipline. Two jobs run at once with up to twenty waiting, so a large submission does not block the small urgent one behind it.
The practical check takes ten minutes. Open your sitemap, count how many of the entries live under the English directory, and compare that with how many English pages you believe you published. On the sites where this article's problem is present, the two numbers rarely match, and the gap is usually larger than anyone expected. Sitemaps generated by a plugin that was configured before the translated section existed are the most common cause, and nobody revisits a file that produces no error message.
What the record proves, and what it does not
The per-URL log is the reason to bother with any of this. A recorded visit history converts a conversation that usually runs on assumption into one that runs on timestamps: this URL was visited on this date, by this bot, with this status. When somebody claims a page "isn't ranking", the log settles in seconds whether ranking was ever the question.
That last point deserves its own emphasis, because it is where the English half of a bilingual site is most exposed. A translated page that mirrors its Danish original in structure, length and argument, with the words swapped, gives an indexing system little reason to keep a second copy. The fix is editorial: different examples, different emphasis, content aimed at the foreign buyer who was described earlier rather than at the domestic reader whose article was translated.
- Check before you conclude. Confirm a fetch happened before treating anything as a ranking problem. Roughly a third of "we don't rank" complaints turn out to be discovery failures once someone looks.
- Submit in priority order. Revenue pages, then references and case studies, then supporting material. Archives and paginated listings can wait indefinitely.
- Re-submit after substance changes. A genuinely rewritten page is worth announcing again. A page with a new publication date and the same body text is not.
- Fix the linking while you wait. Submission gets a page looked at once; internal links are what bring a crawler back. Related-page links inside the English tree do more long-term good than any batch job.
| What the log shows | Reasonable reading | Next step |
|---|---|---|
| No visit recorded at all | The URL was never discovered | Submit it and check that it appears in the sitemap |
| Visit recorded, error status | A technical fault, not a content judgement | Reproduce the fetch yourself and fix the response |
| Visited once, long ago | Fetched and deprioritised | Differentiate the page, then add internal links to it |
| Visited repeatedly, still absent from results | Index judgement, not discovery | Treat as an editorial problem, stop submitting |
Pages a foreign buyer searches by name
Product and model pages, project references, case studies, technical documentation, contact routes for distributors.
- Named, specific and unlikely to exist elsewhere
- Losing them costs an enquiry, not a percentage point
Pages that exist because the system made them
Tag archives, paginated indexes, filtered listings, near-empty category pages and anything generated in bulk.
- They consume fetches and return no demand
- Better excluded than submitted
Common questions
How quickly does a submitted URL get visited?
Sooner than by waiting, and with a record either way, but no specific interval is promised by anyone and you should distrust any figure quoted as a guarantee. The useful change is that you stop guessing: the log tells you when a visit happened, so a fortnight of silence becomes an observation rather than an anxiety.
Should we submit the Danish and English versions together?
Submit them as separate, deliberately ordered batches. The Danish tree usually gets crawled adequately on its own, so spending the daily allowance on it wastes the mechanism on pages that did not need help. Give the allowance to the section without an inbound link trail.
Our English pages are close translations. Is that a problem for indexing?
It can be. A page that repeats its counterpart's structure and argument in another language is easier to fetch and drop than to keep. Translation quality is not the issue; distinctness is. Pages written for the foreign buyer, with their own examples and their own emphasis, survive the judgement better than faithful renderings.
What is a sensible daily submission routine?
Most weeks, nothing. Submission is for new material, genuinely revised material and pages the log shows were never fetched. A standing daily job that resubmits the same URLs consumes the allowance, teaches you nothing and does not improve the odds.
Is indexing a job for the developer or the marketing side?
Both, and the split is clean. Server response, canonical tags, redirects and sitemap generation are engineering work. Deciding which thousand URLs matter today, and whether a page is distinct enough to deserve keeping, is editorial. Sites fail here when each side assumes the other is watching.
Where to start this week
Run the smallest possible version of this check before committing to anything larger. Take five English URLs you would be embarrassed to be missing from — the ones a foreign buyer would search for by name — and establish, from records rather than intuition, whether a crawler has ever fetched them. That single exercise reorders most people's priorities.
If some of them have never been visited, the sequence is: submit those five, confirm the English directory is fully represented in the sitemap, add real internal links from pages that already get crawled, and remove the low-value URLs eating fetches. The engineering half of that list is covered in more depth under our technical SEO work, and the analytics side is discussed across the English articles.
The tools that make this legible are the URL tracker, the sitemap jobs and the per-URL visit log described above, and they sit inside one workspace with the Search Console and rank-tracking views, so a page's fetch history and its performance history can be read on the same afternoon. None of it is a substitute for pages worth keeping. To check your own English tree rather than take any of this on trust, sign in and submit your first batch of URLs, then read the log a week later and see which of your export pages had ever been looked at.
