CSV, JSON or API: choosing how to receive scraped data
Format follows the person using the data
The right delivery format is the one that lands the data where it gets used, with the least work. An analyst who lives in spreadsheets wants a file. A pricing engine wants an API. A data team wants tables in its warehouse. Start from who opens the data and how often.
CSV and Excel
CSV is the most common choice and for good reason. Every tool opens it, it is easy to inspect, and it loads into almost any database. It suits flat data: one row per product, listing or posting.
Excel suits teams who will read the data directly and want formatting, several sheets or a summary tab. It is less suited to automated loading.
Watch for: commas inside values (they need quoting), number formats (thousands separators can turn numbers into text), and dates. Agree a date format, ideally ISO such as 2026-01-10, and plain numbers without separators for machine use.
JSON and JSON Lines
JSON handles nested data well: a product with several variants, each with its own price and stock, or a listing with a list of amenities. Developers prefer it because it maps directly to objects in code.
JSON Lines puts one JSON record per line. It streams well and is the usual choice for large datasets and machine learning pipelines.
Parquet
Parquet is a column-based file format used by data warehouses and analytics engines. It is compact, typed and fast to query. For large recurring datasets going into a warehouse or a data lake, it is often the best option. See web data for AI for how we deliver datasets to data teams.
Cloud storage drops
Instead of emailing files, many teams receive them in a cloud storage bucket they control. Each run adds a dated file. Their own systems pick it up on a schedule. This is simple, reliable and keeps a history of every delivery.
API
An API lets your systems ask for data when they need it. It suits applications: a repricing service, a product feature, an internal dashboard. Good data APIs support incremental queries, for example “give me rows changed since yesterday”, so loads stay small.
Our custom web data API serves your feed through documented endpoints, with run status so your systems know when fresh data is ready.
Webhooks
A webhook is a message we send to your system when something happens, such as a run completing. It removes the need to poll. Often combined with an API or storage drop.
What to agree before the first run
Whatever format you choose, agree these in writing:
- Schema. Column names, types and units. Who can change them, and how changes are announced.
- Keys. Which field identifies a record across runs, so you can track changes.
- Timestamps. Collection time on every row, with a time zone.
- Empty values. How missing data is shown: empty, null or a marker.
- Full or incremental. Does each delivery contain everything, or only what changed?
- Delivery time. When the file or API update is ready each day.
These small decisions prevent most integration problems. Our guide to data quality checks covers what should be checked before each delivery.
Combining formats
Many teams want more than one: a daily CSV for analysts and an API for the product. Both should come from the same pipeline and schema so numbers always match.
A quick decision guide
| Who uses it | Good choice |
|---|---|
| Analysts in spreadsheets | CSV or Excel, emailed or in shared storage |
| BI dashboards | CSV or Parquet in storage, or direct database load |
| Data warehouse | Parquet or CSV in storage, incremental |
| Applications | API with incremental queries, plus webhooks |
| Machine learning | JSON Lines or Parquet with a data dictionary |
If you are not sure, start with CSV in cloud storage. It is easy to change later. Tell us how your team works in the quote form or at sales@crawlmine.com and we will suggest a setup.