Rate Limiting
Updated July 21, 2026
Controlling the speed of requests to a target website to avoid overloading the server or triggering anti-bot defenses.
Also known as: Request throttling, API rate limiting, API throttling, 429 Too Many Requests, Request pacing, Crawl pacing
Rate limiting is a traffic-control mechanism that caps how many requests a client can send to a server within a defined time window. The client can be identified by IP address, API key, user account, or session, and once it crosses the threshold the server usually rejects further requests until the window resets, most often with an HTTP 429 "Too Many Requests" response. The purpose is to protect the server from overload, allocate capacity fairly across clients, and blunt abuse ranging from denial-of-service attacks to aggressive scraping.
In web scraping the term points in two directions at once. It describes the limits a target website enforces against incoming crawler traffic, and it describes the deliberate pacing a scraper applies to its own outbound requests to stay under those limits and avoid tripping anti-bot defenses. The same threshold-based logic a site uses to fend off bots becomes the constraint a well-behaved crawler chooses to operate within.
As a networking idea, rate limiting predates the web. It grew out of traffic-shaping and traffic-policing work on packet and telecom networks in the mid-1980s. The leaky bucket algorithm, widely cited as the foundational technique, was described by Jonathan S. Turner in a 1986 IEEE Communications Magazine article and later formalized by the ITU-T as the Generic Cell Rate Algorithm for policing ATM network traffic. Those algorithms were adapted for HTTP and API traffic (the 429 status code was defined in RFC 6585 in 2012) and, by extension, for the anti-bot limits and scraper-side pacing common today.
How rate limiting is enforced
Most implementations rest on one of a few algorithms. The token bucket model refills a bucket with tokens at a fixed rate and spends one token per request; when the bucket is empty, requests are rejected or delayed until it refills, which allows short bursts up to the bucket's capacity. The leaky bucket model is its near-mirror image: requests enter a queue that drains at a steady rate, smoothing bursts into an even outflow. Given equivalent parameters, the two produce the same accept and reject decisions. Sliding-window counters offer a third approach, tallying requests over a rolling interval rather than a fixed clock boundary.
Whichever algorithm a server uses, it must decide what to key the limit on. IP address is the most common choice for anti-scraping, since it needs no authentication, but a site can also key on an API token, a logged-in account, or a session. When a client exceeds its allowance the standard signal is an HTTP 429 response, frequently accompanied by a Retry-After header telling the client how long to wait before trying again.
Rate limiting vs. throttling
The two words are often used interchangeably, and sources are inconsistent, but the most commonly cited distinction is about what happens to the excess. Rate limiting is usually framed as a hard cap: once the threshold is crossed within the window, further requests are rejected outright, typically with a 429. Throttling more often slows, queues, or delays the excess rather than refusing it, stretching a burst out over time so it still completes, just more slowly.
It is also worth separating rate limiting from the responses that tend to follow it. An IP block or ban is a longer or indefinite denial, often triggered after repeated limit violations, whereas rate limiting is a temporary, threshold-based restriction that clears when the window resets. CAPTCHA and behavioral bot detection act on entirely different signals (headers, TLS or JA3 fingerprints, mouse movement, JavaScript execution) and can fire independently of how fast a client is sending requests.
Self-imposed pacing in a scraper
For a crawler, rate limiting is not only something done to it but something it does to itself. The goal is to sustain long-running collection without overloading a target or being flagged, which means spacing requests deliberately rather than firing them as fast as the network allows. A naive approach of fixed, evenly spaced delays is a poor one: perfectly regular timing is itself a fingerprint, since human traffic is never that even. Scrapers therefore add randomized jitter to their delays to look less mechanical.
When a 429 does arrive, the disciplined response is exponential backoff (waiting progressively longer between retries), often reading the Retry-After header when the server supplies it, rather than hammering the endpoint until it blocks the IP outright. Honoring a published robots.txt Crawl-delay where reasonable is a further courtesy. Crawl-delay is a voluntary, publisher-declared pacing signal from the robots exclusion protocol, distinct from server-enforced rate limiting, that well-behaved crawlers may choose to respect whether or not any hard limit exists.
Why rate limiting shapes competitive-intelligence crawling
Competitive-intelligence tooling depends on polling many competitor properties (pricing pages, job boards, blogs, press rooms) on a repeating schedule. Rate limiting is the central operational constraint on that work. Every target site can enforce its own limits, and a crawler that ignores them earns a 429 at best and an IP block at worst, cutting off the source entirely.
This turns pacing into a genuine trade-off. Faster polling means fresher data but higher detection risk and more load on the target; slower, jittered, backoff-aware polling is gentler and more durable but less immediate. A monitoring system like meertrack that watches many competitor domains at once has to balance the two continuously, spacing requests per domain, backing off on 429s, and respecting reasonable crawl delays so that coverage stays broad and the crawl stays non-disruptive over the long haul. The same mechanism a target deploys to keep bots out is the discipline a courteous CI crawler adopts to keep its access alive.
Stop looking terms up. Start tracking them.
meertrack watches your competitors' websites, pricing, and hiring, then alerts you when something meaningful changes.
Frequently Asked Questions
What HTTP status code indicates rate limiting?
HTTP 429, "Too Many Requests," is the standard code a server returns when a client has sent too many requests in a given window. It was defined in RFC 6585 in 2012. A 429 response often includes a Retry-After header stating how long the client should wait before trying again, which a well-behaved crawler should read and honor rather than immediately retrying.
What is the difference between rate limiting and throttling?
The two labels blur together, but the common way to tell them apart is by what becomes of the extra requests. Rate limiting tends to be a firm ceiling: cross the threshold inside the window and additional calls get refused, usually returning a 429. Throttling, by contrast, generally slows down, buffers, or postpones the surplus instead of rejecting it, so those requests eventually go through, just at a gentler pace. Plenty of sources still treat the pair as synonyms.
How do you handle a 429 Too Many Requests error when scraping?
Stop hammering the endpoint and back off. The standard approach is exponential backoff: wait progressively longer between retries, and read the Retry-After header if the server provides one to set the wait precisely. Reducing your overall request rate, adding randomized jitter between requests, and rotating identifiers can all help you stay under the limit going forward and avoid escalating from a temporary 429 to a full IP block.
What is the token bucket algorithm?
Token bucket is a common rate-limiting algorithm. A bucket is refilled with tokens at a fixed rate up to a maximum capacity, and each request spends one token. Requests are allowed while tokens remain and are rejected or delayed when the bucket is empty. Because the bucket can hold a reserve, it permits short bursts up to its capacity while still enforcing an average rate over time. The leaky bucket algorithm is its near-mathematical mirror image.
Why do websites use rate limiting?
Websites use rate limiting to protect servers from overload, to allocate capacity fairly among clients, and to mitigate abuse such as denial-of-service attacks and aggressive scraping. By capping how many requests any single IP, API key, or account can make in a window, a site keeps one heavy client from degrading service for everyone else and gains a simple, frequency-based defense against automated traffic.
Related terms
Automated extraction of data from websites by parsing HTML/DOM structures and converting unstructured web content into structured data.
Web CrawlingSystematically navigating and fetching web pages by following links, often the first step before scraping specific data.
Anti-Bot DetectionTechniques websites use to identify and block automated scraping (CAPTCHAs, IP rate limiting, browser fingerprinting).
Proxy RotationUsing different IP addresses for each request to avoid IP-based blocking when scraping at scale.
robots.txtA file on a website that declares which pages web crawlers are allowed or disallowed from accessing. Ethical scraping respects these directives.
Polling Interval (Crawl Frequency)How often a monitoring system re-checks a target URL for changes. Shorter intervals = faster detection, higher resource cost.
HTML ParsingExtracting structured data from raw HTML by traversing the DOM tree and selecting elements via CSS selectors or XPath.
Structured Data ExtractionConverting unstructured or semi-structured web content into clean, machine-readable format (JSON, database rows).