Real time page analytics built for scale
In 2021-2022, I led a team of super talented engineers at Qwilr to build the backend analytics system
aimed at enabling our customer insights for their customer engagement. We built a system that could
handle 1000s of requests per second, and scale to 10s of millions of events per day, on the back of
kafka and druid. We used Kafka, Druid, and a custom-built API to handle the ingestion and querying of
data. We also built a custom dashboard to allow users to view their data in real time.
The learnings from this project were the importance of choosing the right tool for the job, and while druid was probably overqualified for this job at that time, but it is very much built to work with apache kafka, facilitated growth for the future and provided a fairly simple route to migrating old data. The promise of druid is fast queries coupled with high concurrency, it was a great learning experience, and the right set of tools in this case.
The learnings from this project were the importance of choosing the right tool for the job, and while druid was probably overqualified for this job at that time, but it is very much built to work with apache kafka, facilitated growth for the future and provided a fairly simple route to migrating old data. The promise of druid is fast queries coupled with high concurrency, it was a great learning experience, and the right set of tools in this case.