HTML Parsing

How HTML parsing works

Browser-grade HTML parsing runs in two stages, as specified by the WHATWG HTML Standard. First, tokenization turns the raw character stream into a sequence of tokens: doctype, start tags, end tags, comments, character data, and end-of-file. Second, tree construction feeds those tokens into a finite-state machine whose 'insertion modes' (states such as 'in head', 'in body', and 'in table') decide where each node attaches, building the DOM. This machinery exists largely to handle imperfect markup: unclosed tags, misnested elements, and omitted optional tags all resolve to a predictable tree.

Once the tree exists, extraction is a query problem. Code walks the DOM and selects nodes using CSS selectors or XPath, then reads their text, attributes, or child structure. Outside a full browser, libraries such as BeautifulSoup, lxml, and html5lib in Python, or Cheerio and jsdom in JavaScript, expose the same read-and-query model against an HTTP response, and the browser-native DOMParser Web API does it inside the page.

CSS selectors vs. XPath

CSS selectors and XPath are the two dominant ways to target elements in a parsed DOM, and choosing between them is the everyday decision in extraction work. CSS selectors use the same syntax front-end developers already know (id, class, attribute, and positional matches) and are generally simpler to write and quick for basic targeting. They are the natural choice when the data lives in an element you can name by class or structure.

XPath is more expressive. It can select by text content, traverse upward to parents and ancestors, express sibling and conditional logic, and it works natively against generic XML as well as HTML. That power matters when the element you want has no stable class and must be located relative to a label, or when you need to walk back up the tree from a matched node. In practice many pipelines mix both: CSS for the common cases, XPath where structure or text matching demands it. Neither changes what parsing produces; they are just two query languages over the same tree.

HTML parsing vs. web scraping and crawling

The three terms are often used interchangeably but describe different stages. Web scraping is the full end-to-end pipeline: fetch a page, render it if necessary, parse the markup, extract the fields, then store or act on them. HTML parsing is only the parse-and-extract step within that pipeline. Web crawling is different again: it is the discovery and traversal problem of following links to find which pages exist, not what to pull from any single one. A crawler decides which URLs to visit; a parser decides what to read once a URL's content is in hand.

A related distinction is parsing versus DOM manipulation. Parsing produces and reads the tree; manipulation, as in a live browser via JavaScript, mutates it. Scraping libraries almost always read rather than modify, since the goal is extraction, not changing a rendered page.

Where HTML parsing meets competitive monitoring

Static HTML parsers only see what is in the raw HTTP response. Many modern sites render key content client-side with JavaScript, so a plain parser finds an empty shell where the price, feature list, or job posting should be. The common fix is to run the page in a headless browser (Chrome driven by Playwright or Puppeteer) so scripts execute and inject content first; the resulting DOM is then parsed with the same selector-based approach.

For change-detection and competitive-monitoring tools, a further discipline applies. Rather than diffing raw HTML, best practice is to parse and track specific fields (a price via a CSS selector, a plan name, a headline) because that isolates meaningful change from markup and class-name churn that alters the HTML without changing anything a person would see. Teams monitoring competitor pricing pages, product pages, and job listings rely on parsed, targeted selectors to keep signal high and false positives low.

Frequently Asked Questions

Related terms