An audience data company
Website logs stream through Spark into Elasticsearch and MySQL, ready for recommendations
An audience data company streamed website logs through Kafka into Spark and Elasticsearch to feed a recommendations engine. We worked in that environment as it stood, analyzing the logs in Spark and writing results to Elasticsearch and MySQL.
Data & AnalyticsWeblog analytics pipelineModernized a live product
- Scope
- Engineering on an existing weblog stack
- Stream
- Kafka into Spark Streaming
- Outputs
- Elasticsearch and MySQL
- Purpose
- Data for a recommendations engine
The challenge
The company already ran Kafka, Spark and Elasticsearch to collect website logs for recommendations, and needed that data analyzed and written where its recommendations work could use it.
What we did
Spark analysis of the log stream on the company's own installed versions, with reads and writes to Elasticsearch and MySQL.
The outcome
The company kept its streaming stack, with log data landing in the stores its recommendations work used, over a long running engagement.
The starting point
An audience data company collected website log data with Kafka, Spark and Elasticsearch already installed. The data was meant to feed machine learning for recommendations.
What had to change
The company needed weblog data analyzed in Spark and read from and written to Elasticsearch and other outputs, including MySQL. It needed that over a long stretch, not as a one off fix.
What could not break
- Spark 2.1 and Elasticsearch 6.5 as installed; no platform swap
- Log data keeps arriving through the Kafka stream while we work
- Results land where existing consumers read them: Elasticsearch and MySQL
The calls we made
We framed the job as keeping an existing data platform useful for recommendations, working inside it rather than around it.
- 01
Work on the installed versions
Kafka, Spark 2.1 and Elasticsearch 6.5 were already collecting the logs, so we worked with the company's own versions and setup.
Ruled outReplacing or re-hosting the platform.
- 02
Write where each consumer reads
Search and analytics needs sat in Elasticsearch and relational needs in MySQL, so the Spark work wrote to both from the same log data.
Ruled outA single store for every consumer.
How the system fits together
What we built
Weblog analysis in Spark
Spark work on the website log data arriving through the Kafka stream, on the company's own Spark install.
Outputs to Elasticsearch and MySQL
Reads and writes between Spark, Elasticsearch and MySQL, so processed log data reached the stores that used it.
How it works
Collect
Website logs stream in through Kafka.
Process
Spark Streaming and Spark jobs analyze the log data.
Store
Results are written to Elasticsearch and MySQL.
Use
The prepared data feeds the client's recommendations work.
What changed
The company kept its existing streaming stack, and its weblog data was analyzed in Spark and written to Elasticsearch and MySQL for the recommendations work the stack was built for.
Under the hood
- Datics role
- Engineering delivered by Datics
- Streaming
- Kafka feeding Spark Streaming on the client's Spark 2.1 install.
- Stores
- Elasticsearch 6.5 in the ELK stack, plus MySQL.
- Machine learning
- The processed data was prepared for the company's recommendations engine.
Data and retrieval
Also in the stack
How it shipped
On the company's own environment
We worked on the company's installed stack and versions, not on a copy we hosted.
Long running
The work ran as an ongoing engagement on the live stack rather than a single project.
What this unlocked
Clean, streaming behavioral data is the groundwork that recommendations and other machine learning features depend on.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.