How to write a web scraping brief (with a template)

· 3 min read · CrawlMine guides

Why a brief saves time and money

A clear brief gets you an accurate quote in a day instead of a week of back-and-forth. It also makes vendors comparable, because everyone is pricing the same job. And it forces your own team to agree on what they need before anyone builds anything. One page is enough.

1. The goal

Start with one or two sentences on why you need the data and what decision it supports. For example: “We want to see competitor prices for our top 2,000 products each morning so the pricing team can adjust before noon.” The goal helps a vendor suggest a better approach than the one you assumed.

2. Sources

List the websites. For each, include one example URL per page type you need: a category page, a product page, a search result. Note anything you already know: the site needs a location, prices differ by city, some content loads after scrolling.

If you only know the kind of source (“the top five property portals in Pune”), say so. The vendor can propose sites.

3. Fields

List the fields as column names with an example value. This is the most important part of the brief.

Field Example Notes
product_name Trail Runner 2 as shown
price 4299 selling price, number only
list_price 4999 if shown
in_stock true
url https://… product page

Mark fields as must have or nice to have. Nice-to-have fields that are hard to get can be dropped to save cost.

4. Volume

Give a rough number: how many products, listings, pages or keywords. A range is fine. Volume drives cost more than almost anything, so a realistic number avoids surprises. If you need everything in a category, say roughly how big the category is.

5. Schedule

One time, daily, weekly or monthly? If recurring, what time should fresh data be ready? Are there periods, such as sale events, when you need more frequent runs?

6. Matching and cleaning

Do you need records matched to your own catalogue or list? If so, what identifiers do you have: SKU, EAN, model numbers? Do you need units normalised, duplicates removed, categories mapped to your own taxonomy? This work is often where the value is, so describe it.

7. Delivery

How should data arrive: CSV, Excel, JSON, Parquet, cloud storage, database or API? Who will use it first? Our guide on delivery formats helps you choose.

8. Constraints

Note anything that limits the work: a deadline, a budget range, internal security rules, a requirement to keep data in a region, or a need for an NDA. Budget helps more than people expect: it lets the vendor propose the most useful scope within it.

9. Compliance notes

Say whether you need any personal data. Usually you do not, and saying so up front simplifies everything. If a source requires a login, mention it; most responsible vendors will not collect behind logins without permission.

A copy-paste template

Goal:
Sources (with one example URL per page type):
Fields (name, example, must or nice to have):
Volume (approximate):
Schedule (one time / daily / weekly / monthly, ready by):
Matching and cleaning needs:
Delivery format and destination:
Deadline and budget range:
Compliance notes (personal data? logins?):
Contact:

What happens next

With a brief like this, a vendor can check the sources, estimate difficulty and send a fixed quote, usually with a suggestion for a small pilot. You can get a rough number before that from our cost estimator.

Send your brief through the quote form or paste it into an email to sales@crawlmine.com. We reply within one working day. Our web scraping services page shows what happens after the brief.

Talk to us directly

sales@crawlmine.com

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to sales@crawlmine.com
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

sales@crawlmine.com