Healthcare
Health app data pipeline designed for 10 users or 1 million
The app launched with about 100 users on a Google Cloud pipeline that merges two data streams
Real-time data streaming and processing

Who we worked with
Healthcare client launching a mobile app
A healthcare client was preparing a mobile app for launch, with an audience nobody could forecast. It brought us in to build the data pipeline behind the app.
What we did
Building and load-testing the pipeline
Ten users or millions, and no telling which
The client commissioned a mobile app that estimates the likelihood of an infection from the phone owner's geolocation history. With the traffic unknown, the data streaming had to scale with whatever the app got. The location history was stored without names or phone numbers, and only the pipeline's own service accounts could read it.
Python and Apache Beam on Google Cloud
We built the pipeline in Python and Apache Beam, running on Dataflow. Data comes in through Pub/Sub and goes out to BigQuery, Cloud SQL and Cloud Storage, with Airflow on Cloud Composer orchestrating the jobs. With the load unknown, managed services are the sensible bet. Dataflow adds workers as traffic grows and removes them when it drops, and Pub/Sub is built to scale horizontally.
What load testing turned up
Before launch we put the pipeline under load. Dataflow started too many workers, and the bill grew with them, so we capped the number of workers. Row-by-row streaming writes into BigQuery ran into cost and limits, and we moved to writing in batches. A design load is only a claim until traffic goes through it, and on a managed cloud the bill is one of the results.

The merge
Merging two streams with a tool new to the team
Why two live streams are hard to join
Joining two live streams is harder than joining tables, because events arrive late and out of order. The standard Apache Beam approach groups events into time windows and waits for the late ones. A watermark is the pipeline's estimate of how far the data has got in event time, and a window closes when the watermark passes its end. Wait too long and results lag. Close too soon and records that belong together never meet.
An experimental tool, and a load test to contain it
Our team used an experimental tool for the merge, because it matched events as they arrived. None of the team had used it before. The team contained that risk by putting the merge through the load test before launch. The app then went live on the pipeline.
Google Cloud
The Google Cloud services behind the pipeline
Pub/Sub
Dataflow
BigQuery
Cloud SQL
Cloud Storage
Cloud Composer
The pipeline in numbers
About 100
Users when the app launched
10–1M
Users the pipeline is designed for
2
Live streams joined as events arrive
Can't forecast your data load?
Tell us what your app or product will stream and what you can't predict about it. We answer within 24 hours.



