81%
of Googlebot requests hit parameter URLs
Twelve weeks of server logs, 538,000 Googlebot hits. Session IDs alone took almost a quarter of the crawl; the 148 pages Granitefold actually wanted ranked shared 12%.
Finding 1 of 4
A fictional business. The figures are a modelled scenario, not a client result.
Granitefold Insurance · 35-agent home & auto insurance brokerage · Burlington
New pages indexed in 4 days, not 74, and +62% organic quote requests
11 months · North America · Signal plan · Crawl budget · URL parameter handling · Log-file analysis · XML sitemaps
of Googlebot’s requests reach real pages
Crawl barometer · Burlington
01The challenge
Its quote tool spun out endless parameter URLs that swallowed the crawl budget, so new coverage pages took months to reach Google’s index.
Granitefold’s quote tool was the best thing on the site for customers and the worst thing for Googlebot. Every ZIP code, vehicle, coverage level and deductible was a crawlable link, and every visit added a session ID. Googlebot kept following the combinations and rarely reached the pages that mattered: new guides on ice-dam claims, teen drivers and lake cottages sat undiscovered for weeks, then waited weeks more to be indexed.
One quote form, multiplied
= 32,568,000
possible quote URLs
Then a fresh session ID on every visit, and the count has no ceiling.
02Diagnosis
81%
of Googlebot requests hit parameter URLs
Twelve weeks of server logs, 538,000 Googlebot hits. Session IDs alone took almost a quarter of the crawl; the 148 pages Granitefold actually wanted ranked shared 12%.
Finding 1 of 4
412,000
parameter URLs known to Google
Search Console’s crawl stats and page indexing report listed hundreds of thousands of quote and calendar URLs as crawled or discovered, against 148 real pages.
Finding 2 of 4
74
median days to index a new page
Only 63 of the 148 coverage and local pages were indexed. The ice-dam claims guide, published in December, was first crawled in late February.
Finding 3 of 4
2,340
crawlable links into the quote tool
A “Quote in your ZIP” footer widget, vehicle shortcuts and a booking calendar with an endless “next month” link put GET links on every template.
Finding 4 of 4
03The strategy
Built on our Signal plan: the technical engagement for sites where crawl, rendering and architecture decide the outcome.
Search Console’s URL Parameters tool was retired in 2022, so every control had to live on the site itself: robots.txt, forms, links, canonicals and sitemaps.
I
Robots.txt rules for the quote tool and the calendar, written only after the logs showed which families wasted the crawl. A disallow stops the fetch; noindex would not have, because Googlebot has to crawl a page to read it.
User-agent: *
Disallow: /quote?
Disallow: /*?*date=II
The quote form now submits by POST, ZIP and vehicle choices live inside the form, sort links became buttons and internal tracking tags were stripped. Session IDs moved to a cookie, and old session URLs 301 to their clean twins.
<form method="post" action="/quote">
<link rel="canonical" href="/coverage/home/">III
Daily log ingestion for the whole engagement. Every sealed trap was checked within a week: requests to that family falling, requests to coverage pages rising. That reading drives the needle in the barometer below.
66.249.66.1 "GET /quote?zip=05401&sessionid=8f3a2c91"
200 "Googlebot/2.1"IV
One sitemap per section (coverage, local offices, guides) holding only canonical 200 URLs with honest lastmod dates, plus links from a coverage hub, so a new page was found within a day of going live.
<loc>https://granitefold.example/coverage/auto/teen-drivers/</loc>
<lastmod>2026-04-14</lastmod>04The barometer
Each switch seals one family of parameter URLs, in the order the work was done. The blued needle reads the share of Googlebot’s requests that reach real pages; the brass hand stays where it stood in month 1.
Reading
Stormy
Brass hand · month 1 · 12%
Stormy: 12% of the crawl reaches real pages, and a new page is indexed in about 74 days.
Session IDs in links24% of crawl
/quote?zip=05401&sessionid=8f3a2c91
FixSessions moved to a cookie; every old sessionid URL now 301s to its clean twin, so Googlebot learns to drop them.
ZIP-prefilled quote links19% of crawl
/quote?zip=05452
FixThe “Quote in your ZIP” widget became a form that posts, and /quote? is disallowed in robots.txt.
Vehicle combinations14% of crawl
/quote/auto?year=2019&make=subaru&model=outback
FixYear, make and model are picked inside the form, never offered as crawlable links.
An endless booking calendar9% of crawl
/agents/book?date=2031-02-01
FixThe calendar stops at 60 days, “next month” became a button and date= URLs are disallowed.
Sorted and paged reviews8% of crawl
/reviews?sort=oldest&page=212
FixSort options became buttons; review pages keep plain ?page= URLs with self-referencing canonicals.
Tracking tags on internal links7% of crawl
/coverage/home/?utm_source=newsletter&ref=footer
FixUTM and ref tags stripped from internal links; canonicals point at the clean URL.
05Month by month
Months 1–2
Twelve weeks of logs, Search Console crawl stats, a full site crawl and a parameter inventory: 41 parameter names in six families.
Months 3–4
Quote and vehicle links turned into forms, session IDs into cookies, and robots.txt rules for the quote tool and the calendar.
Months 5–6
Tracking tags stripped from internal links, sort links turned into buttons, canonicals checked on every template and 1,960 redirect chains collapsed to one hop.
Months 7–8
Sectioned XML sitemaps with honest lastmod dates, a coverage hub linking every guide, and new coverage guides published into a calm crawl.
Months 9–11
Weekly log checks so no new parameter could open a fresh leak; the needle held at Fair through the winter quote season.
06Results
Organic quote requests
+62%
410 → 664 organic quote requests a month
Crawl spent on parameter URLs
81% → 6%
Share of Googlebot requests, month 1 → month 11
Median days to index a new page
74 → 4
Median for pages published that month
Coverage pages indexed (of 148)
63 → 146
2 still waiting to be indexed
Modelled scenario for a fictional business — not a client result. Rankings and traffic are never guaranteed; see our Terms.
Is a filter, a calendar or a quote tool eating your crawl? Start with the Signal plan, or tell us what your logs show.