Web scraping use cases, industry by industry

105 practical ways businesses use public web data, grouped by industry, with the fields each use case needs. Use it to find your starting point, then ask us for a sample.

Prefer email? Write to sales@crawlmine.com. We reply within one working day.

Retail and e-commerce

Full industry page
  • Competitive pricing

    Compare your price with each competitor for the same product, every day, and feed the gaps into your pricing rules.

    price · list price · discount · matched SKU

  • Assortment gap analysis

    List the brands, models and variants competitors stock that you do not, by category.

    category · brand · model · variant

  • Promotion calendars

    Record when competitors run sales and how deep the discounts go, so you can plan your own.

    promo label · discount · start seen · end seen

  • Stock-out opportunities

    See which competitors are out of stock on products you have, while there is still time to act on it.

    in stock · delivery estimate · seller

  • Review-driven merchandising

    Use public ratings and complaints to decide which products to promote, fix or delist.

    rating · review count · review text

Quick commerce and grocery

Full industry page
  • Cross-platform price comparison

    See which platform is cheapest for each product in each pincode, every day.

    pincode · platform · price · MRP

  • On-shelf availability

    List the pincodes where your product is missing while competitors are available.

    in stock · pincode · brand

  • Discount depth tracking

    Measure how deep each platform discounts your category and when.

    discount · offer label · collected at

  • Delivery promise benchmarks

    Compare delivery time estimates by platform and neighbourhood.

    ETA minutes · pincode · platform

  • Share of shelf and placement

    Track sponsored and featured placements in search and category pages.

    position · sponsored flag · category

FMCG and CPG brands

Full industry page
  • Price and promotion compliance

    Check that agreed prices and promotions run as planned on each retailer.

    price · promo · retailer · region

  • Online availability

    Measure in-stock rates by retailer, city and pincode.

    in stock · city · pincode

  • Search share

    See where your products rank for category searches against competitors.

    keyword · position · sponsored

  • Content accuracy

    Check titles, images and specifications against your master data.

    title · image count · attributes

  • Ratings and reviews

    Track ratings by product and retailer and read the complaints behind them.

    rating · review count · text

Consumer electronics

Full industry page
  • Launch price tracking

    Follow how quickly prices fall for a new model and its competitors.

    model · variant · price · date

  • Spec comparison

    Build a clean specification table for every model in a category.

    screen size · storage · battery · weight

  • Seller and MAP monitoring

    List every seller offering your models and flag prices below policy.

    seller · price · MAP gap

  • Bundle and offer tracking

    Record bank offers, exchange offers and bundles shown with the price.

    offer text · bundle items · effective price

Fashion and apparel

Full industry page
  • New arrivals tracking

    See what competitors launch each week, by category, colour and price.

    first seen · category · colour · price

  • Markdown monitoring

    Track when and how deeply competitors discount each line.

    original price · sale price · markdown %

  • Size sell-through signals

    Follow which sizes go out of stock first as a proxy for demand.

    size · in stock · days to sell out

  • Trend and material research

    Count how often materials, fits and colours appear across brands.

    material · fit · colour family

Beauty and personal care

Full industry page
  • Channel price checks

    Compare prices for each shade and size across retailers and apps.

    shade · size · price · channel

  • Ingredient landscape

    Map which ingredients and claims competitors use in a category.

    ingredients · claims · category

  • Review themes

    Group public reviews by issue such as texture, scent or skin reaction.

    rating · review text · date

  • Seller authenticity checks

    List the sellers offering your products and flag unknown ones.

    seller · price · listing URL

Real estate

Full industry page
  • Locality price benchmarks

    Compare asking prices per square foot by locality, type and size.

    locality · price · area · type

  • Rental yield analysis

    Combine sale and rent listings to estimate gross yields by area.

    rent · sale price · bedrooms

  • Inventory and absorption

    Track new, changed and removed listings to see how fast stock moves.

    first seen · last seen · status

  • Project launch tracking

    Follow new project launches, builders and launch prices.

    project · builder · launch price

  • Amenity and attribute research

    See which amenities are common at each price point.

    amenities · floor · parking

Travel and hospitality

Full industry page
  • Competitive set rate shopping

    See what nearby properties charge for the same room type and date.

    property · check-in · room · rate

  • Rate parity monitoring

    Compare your rate on each travel site with your own website.

    source · rate · taxes shown

  • Event demand detection

    Spot dates where prices rise across a city and plan for them.

    date · average rate · availability

  • Package and tour pricing

    Compare tour and package prices for similar itineraries.

    itinerary · nights · inclusions · price

  • Guest feedback tracking

    Track ratings and review themes across your properties and competitors.

    rating · review count · text

Recruitment and HR

Full industry page
  • Hiring demand by role and city

    Count open roles by normalised title, seniority and location.

    title · seniority · location

  • Skills demand tracking

    See which skills employers ask for and how that changes over time.

    skills · posted on · industry

  • Salary benchmarks

    Compare published salary ranges for the same role across markets.

    salary min · salary max · currency

  • Client prospecting for agencies

    Find companies hiring heavily in your specialism.

    company · open roles · role family

  • Employer branding research

    Compare how competitors describe roles, benefits and remote policies.

    remote status · benefits · employment type

Finance and investment

Full industry page
  • Pricing power signals

    Track price changes across a company's product range over time.

    product · price · date · company

  • Hiring momentum

    Count open roles by company, function and location.

    open roles · function · location

  • Store and outlet counts

    Track openings and closures from public store locators.

    store count · city · status

  • Published rate comparison

    Collect deposit, loan and card rates from bank and lender sites.

    product · tenure · rate

  • Disclosure feeds

    Receive new public announcements as structured records.

    entity · headline · published at

Insurance

Full industry page
  • Product feature comparison

    Compare cover, exclusions, waiting periods and add-ons across insurers.

    product · cover · exclusions · add-ons

  • Network coverage checks

    Map network hospitals or garages by city against competitors.

    provider name · city · network

  • Asset value inputs

    Use current vehicle and property listing prices as inputs to valuation models.

    model · year · price · locality

  • Distribution monitoring

    See how aggregators and partners present your products.

    site · rank · displayed premium

Automotive

Full industry page
  • Used vehicle valuation

    Price trade-ins and stock against current comparable listings.

    model · year · km · price

  • Dealer inventory tracking

    See what competing dealers stock and how fast it moves.

    dealer · stock count · days listed

  • New vehicle price tracking

    Track ex-showroom and on-road prices and offers by city.

    variant · city · price · offer

  • Parts and accessories pricing

    Compare parts prices across online parts sellers.

    part number · fitment · price

Healthcare and pharma

Full industry page
  • Medicine price comparison

    Compare prices for the same molecule, strength and pack across online pharmacies.

    molecule · strength · pack · price

  • Availability by city

    Track which products are in stock or out of stock by city or pincode.

    in stock · city · delivery estimate

  • Brand and generic landscape

    Map all brands of a molecule with manufacturer and price.

    brand · manufacturer · MRP

  • Provider network mapping

    List clinics, labs and hospitals by specialty and locality.

    provider · specialty · locality

  • Clinical trial monitoring

    Track new and updated records on public trial registries.

    trial id · condition · phase · status

Logistics and supply chain

Full industry page
  • Rate benchmarking

    Compare published freight and courier rates by lane and weight band.

    lane · weight band · rate · carrier

  • Schedule tracking

    Follow published sailing and service schedules and their changes.

    service · departure · arrival · status

  • Port and customs notices

    Receive new public notices from ports and authorities as structured records.

    authority · notice · published at

  • Supplier price and lead times

    Track prices and stated lead times on supplier and distributor sites.

    part · price · lead time · MOQ

Food and restaurants

Full industry page
  • Menu price benchmarking

    Compare your dish prices with similar restaurants nearby.

    dish · price · cuisine · area

  • Location and cuisine gaps

    Find areas with demand but few restaurants of your cuisine.

    area · cuisine · restaurant count

  • Offer and discount tracking

    Record competitor offers and how often they run.

    offer text · discount · days active

  • Supplier prospecting

    List restaurants by cuisine, area and size to target sales.

    restaurant · cuisine · rating count

Manufacturing and industrial

Full industry page
  • Distributor price monitoring

    See how distributors price your parts and competitors' equivalents.

    part number · distributor · price · quantity break

  • Catalogue and cross-reference data

    Build structured catalogues with technical attributes and equivalents.

    part number · specs · cross reference

  • Input cost tracking

    Track published prices for metals, polymers and other inputs.

    material · grade · price · date

  • Tender opportunities

    Receive public tenders for your product categories.

    buyer · category · deadline

Market research

Full industry page
  • Market sizing

    Count products, sellers, listings or businesses in a market.

    entity · category · region · count

  • Price studies

    Collect price points across brands, channels and regions for a fixed basket.

    product · channel · price · region

  • Competitive landscapes

    Profile every competitor's range, prices and positioning.

    company · products · price range

  • Consumer sentiment

    Analyse public reviews for a category across sites.

    rating · review text · source

  • Tracking studies

    Repeat the same collection monthly to show change over time.

    wave · metric · value

AI and machine learning

Full industry page
  • Domain-specific training sets

    Collect structured records from the sites that define your domain.

    record · category · source url

  • Evaluation and benchmark sets

    Build categorised, labelled samples to test model quality.

    input · label · source

  • Retrieval indexes

    Keep an index of public product, policy or specification pages fresh.

    url · title · updated at

  • Entity and attribute extraction

    Turn pages into entities with typed attributes for knowledge graphs.

    entity · attribute · value

Media and publishing

Full industry page
  • Coverage monitoring

    See every article on your topics or titles across outlets.

    headline · outlet · published at

  • Chart and bestseller tracking

    Record positions of books, apps or titles in public charts.

    title · chart · position · date

  • Retail price and format tracking

    Track prices for print, e-book and audio editions across stores.

    format · store · price

  • Topic and search research

    See which topics and formats rank for your audience's searches.

    keyword · position · url

Education and edtech

Full industry page
  • Course and fee benchmarking

    Compare fees, durations and formats for similar programmes.

    course · provider · fee · duration

  • Admissions calendar tracking

    Collect admission notices, intakes and deadlines in one feed.

    institution · intake · deadline

  • Skills gap analysis

    Compare course content with skills in job postings.

    skill · job count · courses covering

  • Online course market mapping

    List courses on public platforms with prices and ratings.

    platform · course · price · rating

Energy and utilities

Full industry page
  • Tariff comparison

    Compare published tariffs by utility, category and slab.

    utility · category · slab · rate

  • Fuel and energy price tracking

    Record published fuel and energy prices by city and date.

    product · city · price · date

  • Project pipeline tracking

    Follow announced solar, wind and storage projects and capacities.

    developer · technology · capacity · location

  • Regulatory and tender monitoring

    Receive new orders, consultations and tenders as they are published.

    authority · document · published at

Agriculture and commodities

Full industry page
  • Market price time series

    Build clean daily price series by commodity, variety and market.

    commodity · variety · market · modal price

  • Arrivals and supply tracking

    Track arrivals volumes as a supply signal.

    market · arrivals · date

  • Input cost tracking

    Record seed, fertiliser and agrochemical prices across sellers.

    product · pack · price · seller

  • Crop and weather reports

    Collect public crop progress and weather bulletins as structured data.

    region · crop · indicator · value

Construction and infrastructure

Full industry page
  • Tender discovery

    Receive every relevant tender, filtered by category, region and value.

    tender · buyer · value · deadline

  • Award and competitor tracking

    See published award results and who wins which work.

    tender · winner · award value

  • Material price tracking

    Track cement, steel and other material prices across sellers and cities.

    material · grade · city · price

  • Project launch tracking

    Follow new residential and commercial projects for supply opportunities.

    project · developer · location · stage

Marketing and agencies

Full industry page
  • Rank tracking for clients

    Track keywords by city and device in your own data warehouse.

    keyword · location · position · url

  • Competitor monitoring

    Report competitor pricing, offers and messaging changes each month.

    competitor · page · change

  • Review and reputation reports

    Track ratings and review themes across locations.

    location · rating · review count

  • Content and SEO audits

    Extract titles, headings and metadata from large sites.

    url · title · h1 · meta description

Talk to us directly

sales@crawlmine.com

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to sales@crawlmine.com
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

sales@crawlmine.com