How to choose a web scraping company
Start with your own requirements
The most common reason a scraping project disappoints is not the vendor. It is a vague brief. Before you talk to anyone, write down four things:
- Fields. The exact columns you want, with an example value for each.
- Sources. The websites, with an example URL for each page type.
- Volume. How many products, listings or pages, roughly.
- Schedule. One time, or how often fresh data is needed.
Add how you want to receive it and you have a brief any vendor can quote against. It also makes quotes comparable, because everyone is pricing the same job. Our guide on writing a web scraping brief has a template.
Ask for a sample from your own sources
A demo on a vendor’s favourite website proves little. Ask for a sample from your sources, with your fields. A paid pilot is fine and often better: it shows the vendor is serious and gives you real data to inspect.
When the sample arrives, check it properly:
- Open it in the tool your team actually uses.
- Compare twenty rows against the live websites by hand.
- Look for empty fields, wrong units and duplicates.
- Check that every row has a source URL and a timestamp.
If the sample is messy, production will be too.
Ask how breakages are handled
Every scraper breaks eventually, because websites change. The question is what happens next. Good answers sound like this:
- Monitoring detects the break on the same run.
- Someone is alerted and fixes it within an agreed time.
- Missed runs are re-collected where the source allows.
- You are told if any delivered data was affected.
Vague answers such as “our scrapers rarely break” are a warning sign. They all break. Read more in why scrapers break.
Ask about data quality checks and documentation
Getting rows is easy. Getting complete, consistent, deduplicated rows every day is the real work. Ask what checks run on every delivery: row counts against expectations, fill rates per field, value ranges, duplicate detection, freshness.
Ask for the schema documentation: field names, types, units and definitions. Ask whether the schema stays stable between runs, and how changes are announced. A dashboard built on a feed that silently renames columns breaks at the worst moment. Our list of data quality checks for scraped data is a useful reference.
Ask about compliance and NDAs
A responsible vendor raises legal questions before you do. Ask:
- Do you collect only publicly visible data?
- Do you bypass logins or access controls? (The right answer is no.)
- How do you handle personal data?
- Do you review website terms and tell us about restrictions?
- Will you sign an NDA before we share details?
Expect a clear “not legal advice” caveat, and a willingness to discuss the specifics of your sources. See our overview on whether web scraping is legal for background.
Understand the commercial model
Make sure you know what you are paying for:
- Is there a set-up fee, a monthly fee, or both?
- Are fixes, re-runs and infrastructure included?
- What happens to the price if volume grows?
- What is the notice period, and do you keep the data delivered so far?
- Can you get the code at the end if you want to bring it in-house?
Compare like with like. A cheap quote that excludes maintenance is not cheaper over a year.
Red flags
Walk away, or at least ask hard questions, if you see:
- Guaranteed unlimited scale. Every source has limits.
- No sample. If they will not show data from your sources, they may not be able to collect it.
- No maintenance plan. Someone has to fix things when sites change.
- Invented proof. Big numbers with no detail, logos you cannot verify, testimonials with no names.
- No questions about compliance. A vendor who never asks about personal data or site terms is passing that risk to you.
- Fixed answers before seeing the sources. Difficulty varies a lot. A serious vendor checks first.
A short scorecard
Rate each vendor from one to five on: quality of the sample, clarity of the breakage process, quality checks and documentation, compliance approach, and transparency of pricing. The best choice is rarely the cheapest. It is the one whose data you trust on a Monday morning.
Where to start
Write the brief, ask two or three vendors for a sample, and compare the data, not the sales pitch. If you want us to be one of them, send your sources and fields or email sales@crawlmine.com. Our web scraping services and hire a scraping developer pages explain how we work.