Web Scraping
Updated July 21, 2026
Automated extraction of data from websites by parsing HTML/DOM structures and converting unstructured web content into structured data.
Also known as: Web data extraction, Web harvesting, Data scraping, Screen scraping
Web scraping is the automated extraction of data from websites. A program fetches a page, reads its HTML or rendered DOM, and pulls out specific fields such as a price, a job title, a headline, or a feature list, converting unstructured web content into structured rows a database or spreadsheet can hold. It exists because most of the web is published for human eyes, not machines: the data is visible on the page but not offered as a clean feed, so scraping reconstructs the structure the site never exposed.
There is no single inventor or founding paper for web scraping. The practice grew alongside the early web. The first web robot, the World Wide Web Wanderer, appeared in 1993 to measure the web's size, and JumpStation the same year became the first crawler-based search engine, establishing the crawl-and-parse techniques scraping later built on. Tooling matured with libraries like Python's BeautifulSoup, released in 2004, which made HTML parsing routine. In parallel, platforms such as Salesforce and eBay began offering official APIs around 2000 specifically so developers would not have to scrape their HTML.
Today scraping underpins price comparison, search indexing, market research, and competitive intelligence. Any workflow that needs data from sites that publish it on a page but not through an API tends to reach for scraping. Competitor-tracking platforms are a clear case: they fetch a fixed set of competitor pages, parse them into structured fields, and compare each snapshot against the last to surface what changed.
How web scraping works
A scrape runs in stages. First a request fetches the page, returning its raw HTML. For static pages that is enough, and a parser walks the markup and pulls the target fields by their tags, classes, or XPath. Many sites, though, build their content with JavaScript after the initial HTML loads, so the data a user sees is not present in the raw response. Those sites require a headless browser such as Selenium or Playwright that renders the page, executes its scripts, and exposes the finished DOM before extraction happens.
Once the relevant elements are located, the scraper maps them to named fields and emits structured output as CSV, JSON, or database rows. Two roles are often separated in a pipeline. A crawler follows links to discover which URLs exist; a scraper parses each known page to pull specific values. Some tools do both, others just one. A price-monitoring job, for example, may skip discovery entirely and scrape a fixed list of URLs, while a search indexer leans heavily on crawling to find pages first.
Web scraping vs. web crawling
The two terms are routinely conflated, but they answer different questions. Crawling answers what exists: it starts from seed URLs, follows hyperlinks, and builds a catalog of pages, which is how search engines map a site. Scraping answers what a page says: it takes a known URL and extracts specific data fields from its markup.
In practice the two chain together, crawling to discover URLs then scraping each one, which is why tools and vendors often bundle them. But they are distinct steps with distinct designs. A crawler cares about link graphs, coverage, and avoiding traps; a scraper cares about selectors, field mapping, and handling pages whose structure shifts. Competitive-intelligence pipelines usually weight toward targeted scraping of a curated URL set rather than open-ended crawling, because the pages that matter, such as a rival's pricing, careers, and blog pages, are already known and only need to be re-read on a schedule.
How competitive intelligence teams use scraping
Web scraping is the collection layer beneath most competitor-tracking work. The pattern is consistent: fetch a competitor's page, parse the HTML or rendered DOM into structured fields, store that snapshot, then diff it against the previous one. A change in a parsed price, a new job posting, an edited headline, or an added plan tier becomes a signal a team can act on. Because the output is structured rather than a screenshot, the comparison can be field-level and precise instead of a vague visual sense that something moved.
The scope is usually narrow by design. Rather than crawling an entire competitor domain, CI tools scrape a fixed set of high-value URLs, namely pricing, product, careers, press, and blog, on a recurring interval. That keeps the pipeline focused, makes diffs meaningful, and limits load on the target site. It also concentrates parsing effort on a small number of page templates whose structure the tool can learn and maintain.
Legal and ethical boundaries
Scraping's legal status is context-dependent and, in places, unsettled. The most cited U.S. case, hiQ Labs v. LinkedIn, saw the Ninth Circuit rule in 2019 that scraping publicly available, non-password-protected data did not violate the Computer Fraud and Abuse Act. The Supreme Court remanded it in 2021 after a separate ruling narrowed the CFAA, and the dispute ended in settlement rather than a definitive high-court precedent on scraping.
The practitioner consensus is more stable than the case law. Scraping public, non-logged-in data carries comparatively low legal risk. Risk rises materially when scraping reaches behind logins or paywalls, collects personal data or copyrighted content, or violates a site's Terms of Service or robots.txt. For a CI vendor running its own collection jobs, staying in the lower-risk zone means respecting robots.txt and Terms of Service, avoiding logged-in or personal content, and rate-limiting requests so gathering intelligence does not strain the sites being watched.
Stop looking terms up. Start tracking them.
meertrack watches your competitors' websites, pricing, and hiring, then alerts you when something meaningful changes.
Frequently Asked Questions
Is web scraping legal?
It depends on what is scraped and how. Collecting public, non-logged-in data is generally treated as lower risk; in hiQ Labs v. LinkedIn the Ninth Circuit found scraping publicly available data did not breach the U.S. Computer Fraud and Abuse Act, though the case settled without a definitive high-court ruling. Risk rises sharply when scraping crosses logins or paywalls, gathers personal or copyrighted data, or ignores a site's Terms of Service or robots.txt.
What is the difference between web scraping and web crawling?
Crawling discovers URLs by following links and answers the question of what pages exist, which is how search engines map a site. Scraping extracts specific data fields from a known page and answers what that page says. They are often chained, crawl to find URLs then scrape each one, but they are separate steps. Targeted scraping of a known URL list, without broad discovery, is common in competitor tracking.
How does web scraping work technically?
A scraper fetches a page and reads its HTML, then locates target elements by their tags, classes, or XPath and maps them to named fields output as CSV, JSON, or database rows. Static pages parse directly from the raw markup. Pages that build content with JavaScript need a headless browser such as Selenium or Playwright to render and execute scripts first, so the finished DOM is available before extraction.
Is web scraping the same as using an API?
No. Scraping parses a page's HTML or DOM precisely because no structured feed is offered. When a site provides an official API, pulling data through it is not scraping in the technical sense, even though the structured result looks similar. APIs emerged around 2000 as a sanctioned alternative, and platforms like Salesforce and eBay offered them so developers would not need to scrape their HTML.
Can websites block web scraping?
Yes. Sites use rate limiting, bot-detection systems, required logins, and robots.txt directives to deter or restrict automated access, and dynamic or frequently changing markup can break a scraper's selectors. Responsible collection reduces friction and legal exposure: respecting robots.txt and Terms of Service, avoiding logged-in or personal data, and pacing requests so the target site is not overloaded keeps a scraping job in the lower-risk zone.
Related terms
Systematically navigating and fetching web pages by following links, often the first step before scraping specific data.
HTML ParsingExtracting structured data from raw HTML by traversing the DOM tree and selecting elements via CSS selectors or XPath.
Structured Data ExtractionConverting unstructured or semi-structured web content into clean, machine-readable format (JSON, database rows).
Headless BrowserA web browser without a graphical interface (e.g., Puppeteer, Playwright) used to render JavaScript-heavy pages for scraping and monitoring.
robots.txtA file on a website that declares which pages web crawlers are allowed or disallowed from accessing. Ethical scraping respects these directives.
Website Change DetectionAutomated monitoring of web pages to identify when content, structure, or visual appearance changes. The core technology underlying CI monitoring tools.
Rate LimitingControlling the speed of requests to a target website to avoid overloading the server or triggering anti-bot defenses.
Anti-Bot DetectionTechniques websites use to identify and block automated scraping (CAPTCHAs, IP rate limiting, browser fingerprinting).