Data quality checks every scraped dataset needs

· 3 min read · CrawlMine guides

Rows are easy. Right rows are the work.

A scraper that returns rows is not the same as a scraper that returns correct rows. The failures that hurt are quiet ones: a price picked up from the wrong element, half the products missing because a page changed, duplicates inflating counts. These do not cause errors. They cause wrong decisions.

The checks below catch most of them. Ask any vendor which of these run on every delivery, and apply them yourself if you build in-house.

1. Row counts against expectations

Every run should produce roughly the expected number of rows. Compare each run with the previous ones for the same source. A sudden drop, say a third fewer products than yesterday, usually means a page changed or a category failed to load. A sudden jump can mean duplicates.

2. Fill rates per field

For each field, measure the share of rows where it has a value. If price is normally filled on almost every row and today it is filled on far fewer, something broke. Track fill rates over time per source and alert on changes.

3. Value ranges and types

Check that values make sense:

  • Prices are positive numbers within a plausible range for the category.
  • Ratings are within the scale the site uses.
  • Dates are real dates, not in the future unless expected.
  • Text fields are not suspiciously short or long.

A price that is a hundred times the usual value often means a currency or unit problem, or a parsing error that joined two numbers.

4. Change against the last run

For recurring feeds, compare each value with the last run. Prices move, but most do not move by huge amounts overnight. Large changes on many items at once point to a collection problem rather than a market move.

5. Duplicates

The same item can appear twice: once in a category page and once in a search result, or across pages of results. Deduplicate on a stable key, such as the product URL or the site’s own ID, and report how many duplicates were removed. For listings that appear on several sites, deduplication needs matching, which we cover in product matching.

6. Freshness

Every row should carry a collection timestamp. Check that the delivery contains data from the current run, not cached results. Freshness checks also catch stuck jobs that keep delivering yesterday’s file.

7. Schema stability

Columns should not appear, disappear or change type between runs without notice. Validate every delivery against the agreed schema. If a change is needed, version it and announce it. Downstream dashboards depend on this.

8. Spot checks against the live site

Automated checks cannot catch everything. Regularly pick a small random sample and compare it with the live website by hand. This catches subtle errors, like collecting a crossed-out old price instead of the current one.

9. Coverage reports

For each run, report which sources and categories succeeded, which partially failed and which were skipped. Gaps should be visible in the data, not hidden. A missing day marked as missing is far better than one silently filled with old values.

10. Documentation

A data dictionary that defines each field, its unit and how it was collected is a quality control in itself. It prevents people from misreading data, and it makes errors easier to spot.

Putting it together

A good delivery comes with a short run summary: rows per source compared with the usual, fill rates, duplicates removed, any sources with problems and what was done. That one summary tells you whether to trust today’s data before anyone builds a chart on it.

This is how we deliver web data for AI and analytics and every recurring web scraping service. If you want to see what a quality report looks like on your sources, ask for a sample or email sales@crawlmine.com.

Talk to us directly

sales@crawlmine.com

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to sales@crawlmine.com
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

sales@crawlmine.com