Web Scraping Services

Something that reads one page and returns the wanted values takes an afternoon. Almost all the effort in a web data extraction system goes elsewhere: to the sources that quietly restructure their markup, the pagination that behaves differently past the fiftieth page, the results that appear only after a script has run, and the site that starts refusing requests at exactly the frequency that made the data useful.

This is why a one-off script and a collection system are different things rather than different sizes of the same thing. A script that fails does not announce it. It returns fewer rows than yesterday, and by the time somebody notices, the gap in the history is weeks deep and cannot be backfilled, because those pages no longer say what they said. A system has to tell the difference between a source that changed and a source that broke.

The other half of the difficulty is that extraction does not produce data. It produces text. The same company arrives under four spellings, prices come back as strings with a currency symbol attached, and a third of the records describe something already stored. Every use the business had in mind — comparison, alerting, a trend over time — depends on work that happens after the page has been read.

A crawl decides what to fetch. Targets are defined per source — category pages, search results, an index, a sitemap — and the schedule follows how often that source genuinely changes rather than how often it could be asked. Politeness here is self-interested as well as courteous: the fastest way to lose a source is to hammer it.

Extraction reads each page for the defined values. Where markup is consistent that is selector work, which costs nothing per page and never invents anything. Where a site's layout differs by category or changes every few weeks, a model reads the page and returns the same shape, which is what makes an unstructured source collectable at all. The choice is made per source rather than once for the whole system.

Cleaning and structuring turn what was read into typed values: a price becomes a number and a currency, a date becomes a date, an address becomes parts. Deduplication then decides what is actually new, which is the step that determines whether the dataset is usable or merely large.

Enrichment adds what the source did not carry, from other sources or from what the business already knows. Delivery is a database plus whatever people actually read: a scheduled export, an API, or an alert when a watched value moves.

Headless browsers for sources that assemble their pages as they load, and plain HTTP fetching everywhere it suffices — the second is an order of magnitude cheaper to run and is chosen per source, not once. A scheduler and a work queue drive the runs, with backoff on failure, so one unreachable source does not stall the rest.

Postgres holds the result, because the questions asked of this data are relational ones: what changed, what is new, how does this compare with that. Models are used where markup is unreliable rather than everywhere, since a model reading every page of a well-structured site is expense with a chance of error attached.

Identity comes first. The system needs a durable way to say that this listing is the one it saw last week, or the record becomes a heap of near-duplicates and no change can be measured across it. That rule is specific to each source and it is the single thing most worth getting right before scale.

Change detection is what most businesses actually wanted. Not the catalogue, but the difference: which prices moved and by how much, what appeared, what was withdrawn. A dataset that only ever states the present answers far fewer questions than one that remembers.

Every run reports on itself — pages fetched, pages parsed, values that came back empty, counts against what that source normally returns. This is the part people leave out, and it is the part that matters most, because a source silently returning nothing looks exactly like a quiet week until somebody makes a decision on the strength of it.

It collects pages that are published openly. It does not sign in, work around a paywall, or defeat protection a site has put there to refuse it. That is a boundary rather than an engineering problem to be solved: a source that has clearly declined collection is one to license, to ask, or to leave alone.

It does not gather personal information about individuals. Scale is exactly what separates that from reading a page, and data protection law treats it as a separate activity whether or not each page was public.

And it does not replace an official feed. Where a source offers an API or licenses its data, that route is more reliable, cheaper to maintain and permitted. Scraping is what gets built when no such route exists, which is most of the time, but it is worth confirming rather than assuming.