Modern sites are built in ways that force concrete engineering choices. JavaScript executes on effectively all public sites, with usage well above 98%. That single fact means pure HTTP fetching is rarely enough for production work; you need controlled rendering, smart queueing, and a strategy for avoiding script-triggered blocks. Encryption is also the default, with more than 95% of page loads traveling over HTTPS. That raises the bar for connection reuse, TLS session resumption, and pooling, or you waste CPU time on handshakes instead of harvesting data. Finally, content management systems concentrate structure and anti-bot behaviors: one platform powers roughly 43% of all sites. That helps because you can standardize extractors by template, but it also means a small set of patterns guard a huge chunk of your targets, so detection risk compounds if you cut corners.
Plan with numbers, not hope
Before a single request leaves your crawler, do the math. Median web pages trigger more than 70 requests and weigh around 2 MB. If you need 500,000 detail pages a week, that is about 1 TB of transfer on the HTML alone, excluding images blocked by filters and any JavaScript assets required for rendering. At a modest 10 parallel browsers per machine, and a conservative 1.5 seconds per successful page including retries, you will process about 24,000 pages per node per day. Capacity planning from these baselines prevents half-built pipelines that buckle under peak load.
Measure the only outcomes that matter
Scraping is not about request counts; it is about decision-ready records. Track these five metrics at the dataset edge:
- Freshness lag: time between source change and your stored update
- Coverage: share of intended items actually captured
- Field-level accuracy: percent of fields that match source truth after validation
- Duplicate ratio: share of records that collapse under a stable key or content hash
- Cost per successful record: total spend divided by valid, deduplicated outputs
Everything else feeds these five numbers. If they are green, your pipeline works. If they are red, the fix is usually upstream: renderability, blocking, or brittle parsers.
Render responsibly
Given near-universal JavaScript, headless browsers are table stakes, but they must be tightly controlled. Limit the JavaScript surface area you execute to what is necessary for the DOM you need. Preload only essential resources, turn off images unless required for OCR, and set hard timeouts for network idles. This alone cuts bandwidth and raises throughput without sacrificing completeness. Because encryption dominates, enable HTTP connection reuse and tune keep-alive to avoid paying the handshake tax for every tab.
Structure-aware extraction beats brittle selectors
With a large portion of the web running on a handful of CMSs and storefront platforms, design extractors that key off semantic anchors rather than pixel positions. Target stable attributes, microdata, and patterns that persist across templates. Where possible, parse from embedded JSON blocks generated by the platform, which reduces breakage when visual layouts shift.
Identity, routing, and block avoidance
Anti-bot systems increasingly gate access based on traffic reputation, not just instant behavior. Spreading steady, human-paced sessions over clean IP space helps more than cranking up rotations. Static residential IPs mapped to realistic geos and ASNs read as normal traffic, which reduces friction on sites that tighten controls after a small number of suspicious requests. For programs expected to run for months, a long-term proxy solution for businesses is often simpler to maintain than reactive fixes after the blocklists grow.
Validation loops keep you honest
Do not ship raw HTML-derived values straight to downstream systems. Insert cheap, automated checks that mirror how analysts would sanity-check a feed:
Schema checks that enforce types, formats, and permissible ranges
Cross-field rules, such as price including tax equaling subtotal plus tax
Reference checks against known catalogs or prior snapshots
Content hashing to eliminate near-duplicates from paginated views and A/B variants
Canary URLs that you fetch on every run to detect silent parse drift
Run these validations before storage to cut reprocessing costs and keep cost per record stable.
Respectful crawling is durable crawling
Compliance is not just ethical; it is pragmatic. Read robots.txt, set clear contact details in your user agent, and honor crawl-delay guidance. Back off automatically on rising error rates or blockpage fingerprints. Sites that see considerate access patterns are less likely to tighten controls or block entire IP ranges, keeping your pipeline steady.
Bring it all together
The public web’s realities are measurable: near-universal JavaScript, encrypted transport, heavy pages, and concentrated platforms. Build around those facts. Budget bandwidth using median page weights. Right-size capacity from realistic per-page timings. Extract from structure rather than surface. Route traffic over steady identities that match human usage. Validate early and always. When your engineering choices flow from hard numbers, data collection stops being a brittle script and becomes an enduring part of your tech stack.