Web crawling is the automated, systematic process of navigating the web by starting from a set of seed URLs and following the hyperlinks on each page to discover and fetch more pages. It is a discovery mechanism, not a data-extraction one: the output is a set of URLs and a picture of a site's structure, not the specific fields you might later pull from those pages. The program that does the work is called a crawler, spider, or bot, and it operates from a crawl frontier, a queue of URLs waiting to be visited, expanding that queue as it parses the links on each page it fetches.
The technique dates to the early 1990s web. The World Wide Web Wanderer, built at MIT in 1993, is generally cited as the first automated crawler, originally created to measure the size of the web, and the World Wide Web Worm followed with a simple index of page titles and URLs. The generic term itself is usually traced to WebCrawler, the search engine Brian Pinkerton began building at the University of Washington in January 1994 and launched publicly that April as the first full-text search of web pages. Search-engine indexing remains the canonical example: Googlebot and Bingbot crawl continuously to find what exists on the open web. The same technique underlies competitor-monitoring tools, which crawl a target site to learn which pages exist and which have changed before extracting anything from them.
How a crawler traverses a site
A crawler begins with one or more seed URLs, fetches each page, and parses its HTML to extract the links it contains. Those newly discovered URLs are added to the crawl frontier, a queue of pages still to visit, and the crawler works through that queue, expanding it as it goes. Left unbounded, this walk would try to follow every link it finds, so real crawlers are governed by a small set of policies.
A selection policy decides which pages are worth crawling and which to skip. A revisit policy decides how often to return to a page to check whether it has changed. A politeness policy limits request rate and respects robots.txt so the crawler does not overload a server or fetch pages the site has asked bots to avoid. A parallelization policy coordinates multiple crawler processes so they do not fetch the same URL twice. Together these policies turn an open-ended graph traversal into a bounded, repeatable process.
Web crawling vs. web scraping
Crawling and scraping are complementary steps that are frequently confused because they usually run together. Crawling is discovery: it follows links to find which pages exist on a site, and its output is a list of URLs and the structure connecting them. Scraping is extraction: given a page that has already been fetched, it parses the HTML and pulls out specific structured fields, such as a price, a job title, or a publish date.
The two are typically chained. A crawler maps a competitor's site and hands the discovered URLs to a scraper, which visits each one and extracts the data of interest. Crawling determines coverage, which pages you know about and whether they have changed, while scraping determines content, the actual values on each page. You can crawl without scraping, as a pure search-engine index does, and you can scrape a single known URL without crawling at all, but a monitoring pipeline of any breadth needs both.
Focused crawling for competitor monitoring
General search-engine crawlers try to traverse as much of the open web as possible. Competitor monitoring uses the opposite mode: a focused, or targeted, crawl scoped to a single competitor's domain rather than the whole web. The crawler follows links from the homepage, known section pages, and the site's sitemap to enumerate the pricing pages, blog posts, job listings, and press releases that make up the surface being watched.
Because coverage is the point, the crawl runs on a recurring schedule and its results are compared against the previous run. New URLs surface newly published pages; URLs whose content has shifted surface edits. A CI platform typically diffs this discovered state and then scrapes only the pages that are new or modified, rather than re-extracting an entire site each cycle. That selective step is what keeps a change-detection pipeline efficient and is the mechanism behind "what changed on this competitor's site" alerting. meertrack's monitoring works this way: crawl to find changed URLs, then extract from the pages that actually moved.
Coverage limits and practical constraints
Crawling is bounded in practice by more than politeness. Pages behind forms, search boxes, or logins are hard to reach by following links, and content rendered entirely by client-side JavaScript may not appear in the raw HTML a simple fetcher receives, which is why some crawlers pair with a headless browser. By the late 1990s no single search engine was estimated to index more than a fraction of the publicly available web, an illustration that exhaustive coverage is difficult even at large scale.
For competitor monitoring the constraints are narrower but still real. A site that omits pages from its sitemap or hides them behind unlinked URLs can leave gaps in coverage, and aggressive politeness settings trade freshness for a lighter footprint on the target server. Knowing which pages a crawl can and cannot reach is part of trusting the change signal that comes out of it.