How to write a web scraping brief (with a template)
Why a brief saves time and money
A clear brief gets you an accurate quote in a day instead of a week of back-and-forth. It also makes vendors comparable, because everyone is pricing the same job. And it forces your own team to agree on what they need before anyone builds anything. One page is enough.
1. The goal
Start with one or two sentences on why you need the data and what decision it supports. For example: “We want to see competitor prices for our top 2,000 products each morning so the pricing team can adjust before noon.” The goal helps a vendor suggest a better approach than the one you assumed.
2. Sources
List the websites. For each, include one example URL per page type you need: a category page, a product page, a search result. Note anything you already know: the site needs a location, prices differ by city, some content loads after scrolling.
If you only know the kind of source (“the top five property portals in Pune”), say so. The vendor can propose sites.
3. Fields
List the fields as column names with an example value. This is the most important part of the brief.
| Field | Example | Notes |
|---|---|---|
| product_name | Trail Runner 2 | as shown |
| price | 4299 | selling price, number only |
| list_price | 4999 | if shown |
| in_stock | true | |
| url | https://… | product page |
Mark fields as must have or nice to have. Nice-to-have fields that are hard to get can be dropped to save cost.
4. Volume
Give a rough number: how many products, listings, pages or keywords. A range is fine. Volume drives cost more than almost anything, so a realistic number avoids surprises. If you need everything in a category, say roughly how big the category is.
5. Schedule
One time, daily, weekly or monthly? If recurring, what time should fresh data be ready? Are there periods, such as sale events, when you need more frequent runs?
6. Matching and cleaning
Do you need records matched to your own catalogue or list? If so, what identifiers do you have: SKU, EAN, model numbers? Do you need units normalised, duplicates removed, categories mapped to your own taxonomy? This work is often where the value is, so describe it.
7. Delivery
How should data arrive: CSV, Excel, JSON, Parquet, cloud storage, database or API? Who will use it first? Our guide on delivery formats helps you choose.
8. Constraints
Note anything that limits the work: a deadline, a budget range, internal security rules, a requirement to keep data in a region, or a need for an NDA. Budget helps more than people expect: it lets the vendor propose the most useful scope within it.
9. Compliance notes
Say whether you need any personal data. Usually you do not, and saying so up front simplifies everything. If a source requires a login, mention it; most responsible vendors will not collect behind logins without permission.
A copy-paste template
Goal:
Sources (with one example URL per page type):
Fields (name, example, must or nice to have):
Volume (approximate):
Schedule (one time / daily / weekly / monthly, ready by):
Matching and cleaning needs:
Delivery format and destination:
Deadline and budget range:
Compliance notes (personal data? logins?):
Contact:
What happens next
With a brief like this, a vendor can check the sources, estimate difficulty and send a fixed quote, usually with a suggestion for a small pilot. You can get a rough number before that from our cost estimator.
Send your brief through the quote form or paste it into an email to sales@crawlmine.com. We reply within one working day. Our web scraping services page shows what happens after the brief.