Every login event, routed to the right stream, in real time.
A Kafka pipeline that ingests user login events, validates each one, normalizes its timestamp, and routes it by device platform to iOS, Android, or missing-data topics.
Live routed stream, running now. Scroll for the full control room.
/ live control room
Watch every login event route itself.
A live simulation of the pipeline at scale. Tune the producer rate, partition count, and consumer parallelism, then fire chaos at it. The real router.py on the right highlights the line executing for each event.
1from confluent_kafka import Consumer, Producer2import json3from datetime import datetime45TOPICS = {"iOS":"ios-user-login", "android":"android-user-login",6 "missing":"missing-data-login"}7EXPECTED_FIELDS = 789consumer.subscribe(["user-login"])10while True:11 msg = consumer.poll(1.0)12 raw = json.loads(msg.value())1314 if len(raw) != EXPECTED_FIELDS:15 producer.produce(TOPICS["missing"],16 value=json.dumps(raw))17 continue18 if raw.get("user_id") is None:19 continue2021 raw["timestamp"] = str(22 datetime.fromtimestamp(raw["timestamp"]))2324 topic = TOPICS.get(raw.get("device_type",""),25 TOPICS["missing"])26 producer.produce(topic,27 value=json.dumps(raw))
Click any node in the topology to inspect its code and live counters. Slide produce rate past consumer capacity to grow lag.
/ engineering calls
Decisions, with reasons.
The same tradeoffs the repo documents, surfaced as architecture decisions.
One generic consumer, not three?
platform_consumer.py replaces ios / android / missing scripts. Same logic, takes the topic as a CLI arg. Less duplication.
Route in a consumer, not Kafka Streams?
Routing is a 10-line if/else on one field. A plain consumer and producer have far fewer dependencies than a Streams app for logic this simple.
Docker for infra, local for consumers?
The broker and producer need consistent networking, so Docker fits. Consumers are lightweight Python you want to watch and iterate on locally.
What I would add at scale
/ fan-out in 3d
One stream becomes three.
The router inspects each event and fans it to the matching topic. Drag to orbit. At scale, partitioning by user_idhash keeps every user's events ordered within a topic, so downstream consumers see each user in sequence.
/ stack
A small, deliberate stack.
Each tool does one job. The broker and producer are containerized for consistent networking, while the router and consumers stay as lightweight Python you can watch and iterate on locally.
$ docker-compose up -d
zookeeper kafka my-python-producer started
$ python consumers/router.py
Router listening on 'user-login'
[ IOS] user=abc123 dev=iOS v2.3.0
[ANDROID] user=def456 dev=android
[MISSING] {user_id:ghi789, ..incomplete}Session analytics
opt-in, local to your browser, nothing leaves the pageToggle tracking to measure your session: total events routed, peak throughput you pushed the pipeline to, route distribution, and how many chaos interactions you triggered. Useful for the same reason real pipelines need telemetry: you cannot improve what you do not measure.
Run the pipeline locally.
Bring up Kafka, Zookeeper, and the producer in containers, then start the router and watch events classify and route in real time.
$ docker-compose up -d $ python consumers/router.py