Alternative data from the web for investment research

· 2 min read · CrawlMine guides

Nothing here is investment or legal advice. It describes data methods only.

What alternative data is

Alternative data is information outside traditional financial sources, such as filings and market prices, that helps explain how a business is doing. Much of it is published on the public web: what a company charges, how many people it is hiring, how many stores it runs, how its products are rated. Collected consistently, these become signals that move before reported results.

Common web signals

  • Prices and promotions across a company’s product range, as a read on pricing power and demand
  • Stock levels on its online store or retailers’ sites
  • Job postings by function and location, as a read on growth plans. See job postings data.
  • Store and outlet counts from public store locators
  • Listings for property or vehicles, as market activity indicators. See real estate data.
  • Ratings and review counts as a read on product reception
  • App rankings and release cadence for consumer apps
  • Published rates for banks and lenders

Point-in-time is essential

Research data must reflect what was knowable at the time. If a value is corrected later and the history is overwritten, a backtest will use information nobody had, and the signal will look better than it is.

So collect point-in-time: keep every value with its collection timestamp, never overwrite, and record corrections as new rows. Our financial and company data service works this way by design.

Mapping to entities

Web data arrives under brand names, store names and website domains. Research needs it mapped to companies and their identifiers. Keep the mapping as its own documented table, with dates, because brands are bought and sold and the mapping changes over time.

Backtesting pitfalls

  • Short history. Many web signals only exist from the day you start collecting. Plan collection early.
  • Coverage changes. If you add sources over time, aggregate signals shift. Track coverage alongside values.
  • Survivorship. Products, stores and companies that disappear must stay in history.
  • Seasonality. Hiring and pricing are seasonal. Compare like periods.
  • Overfitting. Many candidate signals, short history: be strict with out-of-sample tests.

Provenance and compliance

Compliance teams will ask where data came from and how it was collected. Keep:

  • A list of sources and what was collected from each
  • The method: public pages only, no logins, moderate request rates
  • Confirmation that personal data is excluded
  • Raw snapshots where agreed, to show what a page displayed

We collect publicly available business information only and document each dataset. Firms should apply their own policies on data sourcing.

Building a signal

Start with a hypothesis: for example, that a retailer’s discount depth predicts its margins. Collect the data that tests it, for long enough to cover a few cycles. Clean, map and validate before modelling. Many of the best signals are simple counts done consistently.

Getting started

Tell us the companies you cover and the signals you want to test, and we will propose sources and a schedule. Send your requirements or email sales@crawlmine.com. The finance and investment industry page lists more examples.

Talk to us directly

sales@crawlmine.com

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to sales@crawlmine.com
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

sales@crawlmine.com