NOLK Enterprises Inc.
Thousands of online stores indexed from spreadsheets, compared by traffic and social following
NOLK evaluates and buys e-commerce brands and needed to see which niches and stores perform best. We made its spreadsheets searchable first, then crawled every store for page data, follower counts and traffic rank, and put the comparison in dashboards its team uses directly.
Data & AnalyticsStore intelligence index and dashboardsBuilt from zero
- What shipped
- Searchable store index, crawler and dashboards
- Scale
- Thousands of online stores, compared store by store
- Signals
- Traffic rank, Instagram and Facebook followers
- Rating
- 5 star client review, then a second contract
The challenge
An e-commerce investor held thousands of online stores in four large spreadsheets, with no way to search them or compare stores on traffic and social following.
What we did
We loaded the spreadsheets into Elasticsearch, crawled each store with Scrapy for page data, follower counts and traffic rank, and built Kibana dashboards.
The outcome
The team searches and compares stores by niche in dashboards it runs itself. The client rated the first phase five stars and hired us again for ongoing crawling.
The starting point
NOLK Enterprises Inc. is a startup that evaluates and buys e-commerce brands. Its team needed to judge which niches, for example jewelry, and which stores within them perform well across the whole internet.
What had to change
The store list lived in four large Excel files with no way to search or compare stores. Performance signals such as traffic and social following sat on each store's own site and public profiles.
What could not break
- Start from the client's own four spreadsheets
- Only public store pages, follower counts and traffic data
- Compared store by store, never product by product
- Searchable and visual for a team of non engineers
The calls we made
We started from the data the client already had, made it searchable first, and only then added outside signals.
- 01
Searchable before enriched
Loading the four spreadsheets into Elasticsearch first gave the client a working index and dashboard to judge in a small first phase, before any crawling.
- 02
Compare stores, not products
We compared stores on signals every store has in public: traffic rank, Instagram and Facebook followers, plus page title and description.
Ruled outComparing at product level.
- 03
Dashboards the team runs itself
Kibana lets the client's team filter by niche, for example jewelry, and follow store growth over time from the same index.
How the system fits together
What we built
Store index
The client's four Excel files imported into Elasticsearch, so every store is searchable.
Enrichment crawler
A Scrapy crawler that visits each store and adds page title, description, Instagram and Facebook follower counts and Alexa traffic rank to its record.
Niche and store dashboards
Kibana dashboards for comparing niches and stores and following their growth over time.
How it works
Import
Store lists from the client's spreadsheets are loaded into Elasticsearch.
Crawl
The crawler visits each store's site and public profiles for page data and follower counts.
Enrich
Traffic rank and social signals are added to each store's record in the index.
Compare
The team filters by niche in Kibana and tracks how stores grow over time.
What changed
NOLK could search and compare thousands of stores on traffic and social signals instead of scrolling spreadsheets. The client rated the first phase five stars and hired us again to keep crawling and analyzing at a larger scale.
Under the hood
- Datics role
- Product built by Datics
- Data sources
- The client's Excel files, public store websites, public Instagram and Facebook follower counts and Alexa traffic rank.
- Search and analytics
- Elasticsearch as the index for every store, with Kibana dashboards on top.
- Boundary
- Store level data from public pages and public follower counts, with no product level data.
Data and retrieval
Apps and APIs
Also in the stack
How it shipped
Fixed first phase
The four spreadsheets imported into Elasticsearch with a searchable Kibana dashboard, then a first crawl to enrich each store.
Ongoing crawling and analysis
A second engagement kept the crawler running at a larger scale and extended the dashboards for niche and store analysis over time.
A small fixed first phase
The first phase was tightly scoped, an import, a crawl and a dashboard, so the client could judge the result quickly.
Then ongoing data work
After that delivery the client hired us again to keep crawling and analyzing at a larger scale.
What this unlocked
The same index, kept current by the crawler, let the client track store growth over time rather than taking a single snapshot.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.