The Problem
ActiDash’s clients — e-commerce businesses with complex, multi-channel operations — were making decisions on data that was hours or days old. Reporting pipelines ran on batch schedules: sales data from the previous day, marketing performance from the previous hour, consumer behavior from the previous session batch. By the time the data surfaced in dashboards, the decisions it should have informed had already been made.
The problem compounded at scale. As transaction volumes grew, batch processing windows lengthened. A system designed for 100,000 daily events struggled at 1 million — and the gap between data freshness and business need grew wider. ActiDash needed an architecture that could handle current and projected volumes without batch-processing trade-offs.
The Constraints
Fault tolerance at streaming scale. A system that processes billions of events cannot have single points of failure. Kafka topic replication, consumer group failover, and partition leadership recovery needed to be engineered into the architecture from the start — not added after the first outage.
Integration with existing on-premises systems. ActiDash clients had existing operational infrastructure that could not be replaced or disrupted. The streaming platform needed to ingest from on-premises sources and serve analytics to on-premises consumers while operating from cloud infrastructure.
Highly available, independently scalable. Kafka throughput, processing worker capacity, and serving layer performance all scale differently. Each layer needed to scale independently based on its own bottleneck — not as a monolithic cluster upgrade.
Our Approach
The architecture is built on a Kafka cluster of 100+ brokers, partitioned by data domain (sales events, marketing events, behavioral events, inventory events). Each domain’s producers write to their topic partitions independently; downstream consumers can subscribe to individual domains without coupling to others.
Stream processing workers (Node.js consumers, containerized on Kubernetes) apply real-time transformations to the event stream: sessionization, attribution modeling, behavioral aggregations, and metric roll-ups. Processed outputs write to a serving layer optimized for the dashboard query patterns each metric domain requires.
Angular dashboards serve aggregated metrics at near real-time latency — refreshing from the serving layer on sub-minute schedules. Sales, marketing attribution, consumer behavior, and inventory dashboards each consume the outputs of their corresponding processing domain.
Terraform and Ansible provision the infrastructure as code; Jenkins manages CI/CD for both the processing workers and the dashboard application. The entire cloud infrastructure integrates with on-premises data sources through secure, authenticated connectors rather than requiring on-premises system modification.
The Outcome
- Platform handles billions of events across 100+ Kafka brokers in production
- Near real-time dashboards — business stakeholders see metric changes within seconds of the underlying events
- Seamlessly integrated with existing on-premises operational systems
- Delivered ahead of schedule — no scope reduction, no timeline extension
Client feedback: “The team was highly responsive to our requirements and accommodated changes promptly as they came up. Notably, they delivered ahead of schedule.” — Deanna Shepard, ActiDash
Team
Engagement: 6 months, 5 engineers (1 tech lead, 2 data engineering, 1 backend, 1 DevOps).
Stack: Angular, Node.js, Apache Kafka, Kubernetes, Terraform, Ansible, Jenkins, AWS