Web Crawling

How a crawler traverses a site

A crawler begins with one or more seed URLs, fetches each page, and parses its HTML to extract the links it contains. Those newly discovered URLs are added to the crawl frontier, a queue of pages still to visit, and the crawler works through that queue, expanding it as it goes. Left unbounded, this walk would try to follow every link it finds, so real crawlers are governed by a small set of policies.

A selection policy decides which pages are worth crawling and which to skip. A revisit policy decides how often to return to a page to check whether it has changed. A politeness policy limits request rate and respects robots.txt so the crawler does not overload a server or fetch pages the site has asked bots to avoid. A parallelization policy coordinates multiple crawler processes so they do not fetch the same URL twice. Together these policies turn an open-ended graph traversal into a bounded, repeatable process.

Web crawling vs. web scraping

Crawling and scraping are complementary steps that are frequently confused because they usually run together. Crawling is discovery: it follows links to find which pages exist on a site, and its output is a list of URLs and the structure connecting them. Scraping is extraction: given a page that has already been fetched, it parses the HTML and pulls out specific structured fields, such as a price, a job title, or a publish date.

The two are typically chained. A crawler maps a competitor's site and hands the discovered URLs to a scraper, which visits each one and extracts the data of interest. Crawling determines coverage, which pages you know about and whether they have changed, while scraping determines content, the actual values on each page. You can crawl without scraping, as a pure search-engine index does, and you can scrape a single known URL without crawling at all, but a monitoring pipeline of any breadth needs both.

Focused crawling for competitor monitoring

General search-engine crawlers try to traverse as much of the open web as possible. Competitor monitoring uses the opposite mode: a focused, or targeted, crawl scoped to a single competitor's domain rather than the whole web. The crawler follows links from the homepage, known section pages, and the site's sitemap to enumerate the pricing pages, blog posts, job listings, and press releases that make up the surface being watched.

Because coverage is the point, the crawl runs on a recurring schedule and its results are compared against the previous run. New URLs surface newly published pages; URLs whose content has shifted surface edits. A CI platform typically diffs this discovered state and then scrapes only the pages that are new or modified, rather than re-extracting an entire site each cycle. That selective step is what keeps a change-detection pipeline efficient and is the mechanism behind "what changed on this competitor's site" alerting. meertrack's monitoring works this way: crawl to find changed URLs, then extract from the pages that actually moved.

Coverage limits and practical constraints

Crawling is bounded in practice by more than politeness. Pages behind forms, search boxes, or logins are hard to reach by following links, and content rendered entirely by client-side JavaScript may not appear in the raw HTML a simple fetcher receives, which is why some crawlers pair with a headless browser. By the late 1990s no single search engine was estimated to index more than a fraction of the publicly available web, an illustration that exhaustive coverage is difficult even at large scale.

For competitor monitoring the constraints are narrower but still real. A site that omits pages from its sitemap or hides them behind unlinked URLs can leave gaps in coverage, and aggressive politeness settings trade freshness for a lighter footprint on the target server. Knowing which pages a crawl can and cannot reach is part of trusting the change signal that comes out of it.

Frequently Asked Questions

Related terms