Polling Interval (Crawl Frequency)

How the interval drives detection speed and cost

A polling monitor runs on a schedule. At each tick it requests the target URL, renders or parses the response, and diffs it against the last stored snapshot; if the diff clears a change threshold, it fires an alert. Nothing about that loop depends on whether the page actually changed, so most checks return no new data. Widely cited Zapier figures put the share of polling requests that find nothing at roughly 98.5 percent, which is the plain cost of the approach: you pay for every check to catch the occasional one that matters.

The interval sets the ceiling on how stale your view can be. With a six-hour interval, a competitor's price change goes live and is detected somewhere between instantly and six hours later, depending on where in the cycle it lands; on average you learn about it about half an interval late. Halving the interval halves that expected lag but roughly doubles request volume, bandwidth, and proxy spend, and doubles the exposure to rate limits. That linear trade is why interval choice is a deliberate decision rather than a default left at its lowest setting.

Setting interval by page type

Blanket frequencies waste effort on pages that rarely move and under-watch the ones that change fast. The common practice, recommended by competitor-monitoring tools such as Visualping and PageCrawl.io, is to tune the interval per URL by how quickly the page changes and how much a change matters competitively.

Pricing and product pages tend to get the shortest intervals, often five to sixty minutes, or at least daily, because pricing has high change velocity and directly affects deals in flight. Careers pages and blogs are usually checked every couple of days to weekly; a new role or post is worth catching but rarely time-critical to the hour. Structural elements like navigation, sitemaps, and About pages change seldom, so a monthly review is enough. Assigning intervals this way concentrates the polling budget where detection latency has the most strategic value, instead of spending it evenly across pages that mostly sit still.

Polling vs. webhooks

Polling is pull-based: the monitor asks the target on a fixed cadence whether anything changed, regardless of whether it did. Webhooks are push-based: the source system sends an event the instant something happens, so there is no interval, no wasted checks, and near-instant delivery. On paper webhooks win on both latency and efficiency.

The catch is that push requires the source to support outbound event delivery, and most public websites a CI team wants to watch do not offer webhooks for their pricing, careers, or press pages. That leaves polling as the default mechanism for external competitor monitoring: if the target will not notify you, the only option is to keep checking. Where a source does expose events, such as an internal API, a status page feed, or a platform integration, a webhook is the better tool. The two are often combined, with webhooks used where available and polling filling the far larger gap where they are not.

Related dials it is often confused with

Polling interval is the input, how often you check. Detection latency, or time-to-detect, is the output, how long after a real change occurs before you learn of it. A shorter interval lowers detection latency, but they are different metrics: latency also depends on where the change falls in the cycle and on processing time after the fetch.

In SEO, crawl frequency sits next to two adjacent terms. Crawl budget is a search engine's own allocation of how many and which URLs of a site it will crawl in a period, driven by crawl limit and crawl demand; it is a Google-side, indexing concern, not a monitor's re-check cadence. Crawl rate is the speed or concurrency of a crawler's requests, in requests per second, rather than how often it returns to a given URL over time. The three are routinely conflated even within SEO writing. Polling interval is also sometimes mixed up with cache TTL, which governs when a cached copy is considered stale, not the schedule on which active re-fetches fire.

Frequently Asked Questions

Related terms