Googlebot’s Crawl Quirks with Super-Sized Category Pages
Let’s start with this: Googlebot doesn’t love 100K+ URL category pages. Especially if your filters spit out half-indexable, half-canonical pages with JavaScript gluing everything together. I had a client once where one clothing category had 170k indexed URLs, most of which were paginated “show 24 / show 48 / sort by newest” permutations that didn’t even return results after page 7. Googlebot kept pinging those stale paginated pages like a fridge light that never turns off.
Here’s what you don’t often hear: when Googlebot gets stuck in pagination hell, it starts burning crawl budget on URLs that add no value. The more parameters you stack, the worse it gets. Not because parameters are bad, but because some of them falsely return a 200 with thin or duplicate content. Even if you’re canonicalizing appropriately, the bot still wastes time crawling them if discovery rates spike.
“Just noindexing the paginated pages didn’t help — the crawl rate dropped, not the crawl waste.”
You want to lean heavier on robots.txt disallows for parameter junk only after you figure out what Googlebot is chewing on, not before. The real game is understanding what it’s already seen and keeps coming back to unnecessarily. That means cracking into your server logs and Search Console’s Crawl Stats, not just adding random URL parameter rules in blind panic.
When Parameter Handling in GSC Doesn’t Do What You Think
Google Search Console’s legacy parameter handling tool used to do something — until it kind of didn’t. If you added “sort=price_asc” and told GSC to ignore it? Google would eventually sometimes listen… but only if the pages didn’t internally link to those versions.
There was one site I worked with where ?sort=newest had internal links across basically every facet. Even after setting a custom parameter handling rule to “No URLs: doesn’t affect page content,” GSC still showed those being crawled — and somehow increased crawl rate the week after. I suspect Googlebot treats in-sitemap and internally linked parameter variants as discoverable no matter what you tell it.
What started working better: stripping or replacing parameters server-side on deep navigations. Doing this with a bit of Cloudflare Worker voodoo helped more than any GSC setting. Bonus: use canonical URLs that don’t carry the parameters and ensure internal links point to that canonical version. GSC can say one thing and bot behavior shows another, so always verify against logs and live crawl fetches.
Edge-Case: JavaScript Sort Filters That Load Differently for Bots
Here’s a nasty one nobody tells you: if your front-end framework lazy-renders facets or sort menus (common in React or Vue builds) and they require client-side JS to populate, Googlebot might not see the full link structure at all — OR worse, it may see variations you didn’t know existed.
I ran into this while auditing an Angular-based shop that had a “Sort by Discount” option that rendered only after interactive state initiated. Even in rendered tests in Search Console, it looked fine visually. But from logs and rendered HTML snapshots (use a headless browser to double check), I saw that Googlebot got a default ?discount=true variant that wasn’t even selectable in the real UI.
Google’s rendering system may lag a few dozen seconds behind the layout paint, and if you’re injecting anchor tags after user interaction, those links go completely undiscovered. So yeah, that messes with crawl pattern discovery. We ended up re-writing the filters as server-rendered, no-JS fallback anchor tags and left the actual dynamic behavior to JS-enhanced handlers. Crazy how just seeing a live snapshot gave more clarity than a week of GSC console diving.
Sitemaps: Don’t Overload Just Because You Can
Big e-commerce builds tend to auto-gen sitemaps like they’re printing cash. One sitemap per variant, per language, per manufacturer. I looked at one that had over 300 individual sitemap files — and almost half didn’t get crawled in the last 90 days. That’s not “efficiently hinting at priority.” That’s wasting XML bandwidth.
Google will not crawl everything just because it’s in a sitemap. It prioritizes based on historical interaction, internal linking, and a bunch of signals we don’t control. Worse yet, a badly tuned sitemap setup can slow initial discovery because of how Google queues new ones in batches.
Sanity checklist for massive sitemap usage:
Compress and cache sitemap indexes on a CDN edge — not origin served
Cap each sitemap’s URL count to 25K, not the full 50K if you’re updating often
Group URLs functionally (e.g., /products/sale/, /brands/), not arbitrarily
Prune anything that hasn’t been updated or indexed in 6+ months
Use lastmod dates — and make damn sure they actually reflect product template or availability changes
A great sitemap is like shouting politely: not too much, and only when it matters. Resist the temptation to put every possible combination in there just because you can automate it.
Cloudflare and Edge Bot Rules That Actually Work
For sites under too much crawl pressure (I’m looking at you, Shopify with 9000 semi-identical tags), Cloudflare’s bot rules can drastically reduce unwanted crawl traffic without touching origin code. I deployed this on a BigCommerce monster store that had 1.8 million tag+cateogry parameter combos — 90% of which led to zero-converting product lists.
The setup:
Expression: (http.request.uri.path contains "/collections/" and http.request.uri.query contains "page=")
Action: Skip/JS Challenge for bots
This isn’t about blocking Googlebot — it’s about slowing non-legit bots from chewing your tail off. You can also selectively rate-limit anonymous requests to specific parameter-heavy paths.
Pro tip: set a high-frequency alert in Cloudflare Analytics for spikes in URLs that include 3+ query params. These usually signal bot loops or scraper tests gone rogue. A lot of unnecessary crawling can be caught here before you start hacking at sitemaps or rendering heuristics.
Internal Linking Depth Kills Crawl Efficiency More Than Most Realize
Back in the day, I assumed as long as a URL was in the sitemap, Google would find it. Then came the 10,000-deep product set where things beyond depth 4 just never got picked up unless manually pinged. Turns out, your internal link graph shape might be the quiet cause of half your crawl issues.
Googlebot seems to treat highly interlinked, shallow-depth pages as high-priority crawl locations — and only dips deeper if your site architecture shouts “yes there’s more gold down here.” That means your 50th variant of product/detail/sku?product_id=1238&p=3 ends up two folders too deep and zero contextually linked to anything closer to homepage strength.
So what helped:
Smart merchandising blocks with real product data, not random links
Breadcrumbs that also link to peer categories or top-selling sibling products
Rotating featured collections on home/category level with curated real products — not dynamically generated placeholder junk
Every product should link to at least one related item that isn’t just “seen together” logic. Manual curation wins crawl friendliness.
It isn’t just about “link juice.” It’s how visible your page URL is to the crawler’s session tree. If it’s hard to reach, it’s easy to ignore.
Unexpected Fix: Drop the Mega Menu JS Hydration
This one is subtle. Your header navigation might be one of the most important internal linking devices on your whole site — and if you mess it up with delayed JS hydration, you’re essentially hiding that powerhouse until too late in render.
I ran into this with a Magento headless build. The devs had moved the mega nav into a separate <MegaNav /> component that only hydrated after user interaction. Result: Googlebot didn’t see those category links early enough, and the key taxonomies had zero crawl weight. Rendered HTML during pre-hydration showed an empty <nav> shell.
We rolled back to a fully server-rendered nav with simple fallback anchor elements. Everything else could rehydrate progressively afterward. Crawl stats improved in a week. Clickthroughs too.
I won’t get into the whole isomorphic JavaScript debate here — but if your nav doesn’t render in curl -L, fail faster and fix it.
