Illustrative case study

A fictional business. The figures are a modelled scenario, not a client result.

FARWANDER TRAVEL — Online travel agency, 2M+ pages · Seattle

How a 2-million-page travel marketplace got Google crawling the pages that make money

REVENUE-PAGE CRAWL SHARE

30% → 78%

Over 16 months on the Signal plan: Googlebot re-vectored from junk airspace to the pages that book.

CONTROLLER CONSOLE · SELECT PHASE

MONTH 1 · ATC → GOOGLEBOT

GOOGLEBOT, RADAR CONTACT. YOU ARE HOLDING OVER FILTERED-SEARCH AIRSPACE. SEVEN OF EVERY TEN REQUESTS ARE LANDING ON JUNK.

Readback: Holding. Destination pages last visited 19 days ago.

REVENUE CRAWL SHARE
30%
KEY-PAGE RECRAWL
19days

REVENUE PAGESJUNK · 70%

BRIEFING SHEET · FARWANDER TRAVELSEATTLE, UNITED STATES

01The challenge

Two million pages, and the crawler kept circling the wrong ones.

Googlebot spent 70% of its crawl on expired deals and filtered search pages. High-margin destination pages waited weeks to be recrawled after price changes.

On a travel marketplace, freshness is revenue. When a Lisbon package drops by $120 and Google keeps showing last month’s price, the click goes elsewhere. Farwander’s search filters, sort orders and ended deals had turned the site into congested airspace: Googlebot was busy, just never over the pages that book.

GOOGLEBOT HITS BY URL CLASS · 30-DAY LOG SAMPLE, MONTH 0

  • Filtered & sorted search41%
  • Expired deals (200 OK)22%
  • Session & print params7%
  • Revenue pages30%

02Diagnosis · notices to airmen

Six hazards on the chart, all found in the logs before a single template was touched.

  1. NOTAM 01LOG FILES

    11.8M Googlebot requests in a 30-day sample; 8.3M of them hit faceted search, expired deals and parameter duplicates.

  2. NOTAM 02FACETS

    1.4M crawlable filter combinations (price × stars × board × date), each self-canonical and linked from every results page.

  3. NOTAM 03EXPIRED DEALS

    212,000 ended offers still answered 200 OK with a “deal ended” banner — soft 404s Google kept revisiting.

  4. NOTAM 04SITEMAPS

    380,000 sitemap URLs were non-canonical or redirected; lastmod was stamped on every build, so it carried no signal.

  5. NOTAM 05ARCHITECTURE

    9% of destination pages were orphaned and the median revenue page sat 7 clicks deep from the homepage.

  6. NOTAM 06TEMPLATES

    428,000 programmatic “hotels near X” pages had under 80 unique words and no inventory — thin by any measure.

03The strategy · filed flight plan

Close the junk airspace, then build the corridors.

Four vectors, flown in order, on our Signal plan. Measure first, remove waste second, then give the crawler a shorter, better-signposted route to every page that can take a booking.

  1. VEC-1HDG 030

    Log-file analysis

    Server logs piped into a daily Googlebot report, verified by reverse DNS. Every URL class gets a crawl share, a status mix and a last-hit date, so each later change is measured in hits, not guesses.

  2. VEC-2HDG 060

    Crawl budget

    Sort, session and print parameters disallowed in robots.txt; facets beyond two filters made non-linkable; expired deals return 410 with a link to the live destination; sitemaps split by template, with lastmod only on real price changes.

  3. VEC-3HDG 090

    Programmatic SEO

    Templates rebuilt around live inventory: a page is only generated when it has at least 12 bookable options and unique price data. 640,000 thin pages consolidated or removed; the survivors gained fare tables and seasonal guides.

  4. VEC-4HDG 120

    Internal linking

    Region → country → city hubs, with price-change modules that link freshly updated deals from the homepage and hubs. Median click depth for revenue pages fell from 7 to 3.

04Timeline · flight progress strips

16 months, one strip per month.

  1. 01LOGCRAWL 30%Log pipeline live; 30-day Googlebot baseline captured
  2. 02AUDCRAWL 31%URL-class audit: 1.4M facet combos, 212k expired deals
  3. 03ROBCRAWL 33%Sort, session and print parameters disallowed
  4. 04410CRAWL 36%Expired deals switched from 200 OK to 410 Gone
  5. 05FACCRAWL 41%Facets beyond two filters made non-linkable
  6. 06MAPCRAWL 47%Sitemaps split by template; lastmod tied to price changes
  7. 07PRGCRAWL 53%Inventory threshold added to programmatic templates
  8. 08PRNCRAWL 58%First 450k thin pages consolidated or removed
  9. 09HUBCRAWL 62%Region → country → city hubs launched
  10. 10LNKCRAWL 66%Price-change modules on homepage and hubs
  11. 11TPLCRAWL 69%Fare tables and seasonal guides on surviving pages
  12. 12PRNCRAWL 72%Pruning complete at 620k; redirects mapped
  13. 13QACRAWL 74%Soft-404 sweep; orphan rate under 1%
  14. 14MONCRAWL 76%Recrawl latency alerting on top 5,000 pages
  15. 15TUNCRAWL 77%Final 20k thin pages retired; 640k total
  16. 16HLDCRAWL 78%Steady state: 78% crawl share, 2-day recrawl

05Results · landing report

The crawler now spends its fuel where the bookings are.

Modelled scenario for a fictional business — not a client result. Rankings and traffic are never guaranteed; see our Terms.

  • CRAWL SHARE ON REVENUE PAGES

    30% → 78%

  • AVG. RECRAWL TIME (KEY PAGES)

    19 days → 2 days

  • ORGANIC BOOKINGS

    +41%

  • THIN PAGES PRUNED

    640,000

REVENUE-PAGE CRAWL SHARE · MONTH 1–16BARS: KEY-PAGE RECRAWL, DAYS
0%25%50%75%100%M1M4M7M10M13M16
DESTINATION RANKINGS · GOOGLE US, MONTH 1 → 16▲ POSITIONS GAINED
QUERYLANDING PAGEBEFOREAFTERΔ
lisbon city break deals/destinations/lisbon#14#3▲11
kyoto ryokan packages/hotels/kyoto#22#5▲17
last minute greece holidays/deals/greece-live#18#4▲14
iceland northern lights tours/destinations/iceland#11#2▲9
all inclusive cancun from seattle/packages/cancun#27#6▲21
patagonia trekking tour/tours/patagonia#24#4▲20
amalfi coast hotels with pool/hotels/amalfi#19#7▲12
costa rica family vacation package/packages/costa-rica#31#8▲23

06Your clearance

Get a plan like this

Large catalogue, slow recrawls, a crawler stuck in your filters? We start where Farwander did: the logs.

See the Signal plan →Talk to us← Back to all case studies