A web data collection script can work perfectly during testing and still become unreliable within hours of entering production.<\/p>\n\n\n\n
A target website may start returning rate limits. A proxy can become unavailable. Network requests can begin timing out. A browser worker can consume more memory than expected. The pipeline may even report a successful run while collecting only a fraction of the expected data.<\/p>\n\n\n\n
That last scenario is particularly dangerous. A process being alive does not mean the data is correct or complete.<\/p>\n\n\n\n
Reliable web data pipelines are designed around this reality. They expect individual requests, dependencies, workers, and infrastructure components to fail. The goal is not to eliminate every failure but to isolate failures, recover from them, and keep successful work moving.<\/p>\n\n\n\n
A typical collection pipeline depends on several layers.<\/p>\n\n\n\n
The target website can introduce rate limits, return HTTP errors, change its page structure, or present a challenge instead of the expected content.<\/p>\n\n\n\n
The network layer can fail through connection timeouts, proxy outages, DNS problems, or poor connectivity.<\/p>\n\n\n\n
The processing infrastructure introduces another set of risks. Workers can run out of memory, storage can fill up, processes can crash, and scheduling problems can leave jobs stuck or prevent them from running altogether.<\/p>\n\n\n\n
These failures should not be treated identically.<\/p>\n\n\n\n
A temporary timeout may justify a retry. Invalid proxy credentials require intervention. A 404 response<\/a> is often a valid result rather than a technical failure. A 429 response indicates that the system needs to respect the target’s rate limit.<\/p>\n\n\n\n This classification determines what happens next.<\/p>\n\n\n\n A common failure pattern is to treat every unsuccessful request as a signal to immediately try again.<\/p>\n\n\n\n That can turn a temporary problem into a larger outage.<\/p>\n\n\n\n Suppose a product collection pipeline starts receiving HTTP 429 responses. If hundreds of workers immediately retry those requests, the system can generate another burst of traffic and extend the problem.<\/p>\n\n\n\n A better approach is controlled recovery.<\/p>\n\n\n\n First, record the failed request and relevant response information. Then classify the failure and determine whether it is safe to retry. Retryable requests should return to a controlled queue rather than blocking the worker.<\/p>\n\n\n\n Exponential backoff increases the delay between successive attempts. Adding jitter prevents many workers from waking up and retrying at exactly the same time.<\/p>\n\n\n\n Retries should also have boundaries. A maximum attempt count prevents one request from consuming resources indefinitely. An overall deadline provides another limit when a dependency remains slow for an extended period.<\/p>\n\n\n\n Concurrency matters as well. Per domain concurrency limits<\/a> can prevent one unstable target from consuming capacity needed by other jobs.<\/p>\n\n\n\n The objective is not to force every request through but to maintain predictable system behaviour when the target becomes temporarily unavailable.<\/p>\n\n\n\n A production collection system should assume that external dependencies will occasionally become unreliable.<\/p>\n\n\n\nHandling Rate Limits Without Making Them Worse<\/strong><\/h2>\n\n\n\n
Isolating Proxy Failures and External Dependencies<\/strong><\/h2>\n\n\n\n