Health app data pipeline designed for 10 users or 1 million

The app launched with about 100 users on a Google Cloud pipeline that merges two data streams

Real-time data streaming and processing

Health app on a smartphone: a map of today's location history, a likelihood estimate and the places visited, stored without name or phone number

Who we worked with

Healthcare client launching a mobile app

A healthcare client was preparing a mobile app for launch, with an audience nobody could forecast. It brought us in to build the data pipeline behind the app.

Client under NDA

What we did

Building and load-testing the pipeline

Ten users or millions, and no telling which

The client commissioned a mobile app that estimates the likelihood of an infection from the phone owner's geolocation history. With the traffic unknown, the data streaming had to scale with whatever the app got. The location history was stored without names or phone numbers, and only the pipeline's own service accounts could read it.

Python and Apache Beam on Google Cloud

We built the pipeline in Python and Apache Beam, running on Dataflow. Data comes in through Pub/Sub and goes out to BigQuery, Cloud SQL and Cloud Storage, with Airflow on Cloud Composer orchestrating the jobs. With the load unknown, managed services are the sensible bet. Dataflow adds workers as traffic grows and removes them when it drops, and Pub/Sub is built to scale horizontally.

What load testing turned up

Before launch we put the pipeline under load. Dataflow started too many workers, and the bill grew with them, so we capped the number of workers. Row-by-row streaming writes into BigQuery ran into cost and limits, and we moved to writing in batches. A design load is only a claim until traffic goes through it, and on a managed cloud the bill is one of the results.

Diagram: two data streams from Pub/Sub merge in Apache Beam on Dataflow and land in BigQuery, Cloud SQL and Cloud Storage

The merge

Merging two streams with a tool new to the team

Why two live streams are hard to join

Joining two live streams is harder than joining tables, because events arrive late and out of order. The standard Apache Beam approach groups events into time windows and waits for the late ones. A watermark is the pipeline's estimate of how far the data has got in event time, and a window closes when the watermark passes its end. Wait too long and results lag. Close too soon and records that belong together never meet.

An experimental tool, and a load test to contain it

Our team used an experimental tool for the merge, because it matched events as they arrived. None of the team had used it before. The team contained that risk by putting the merge through the load test before launch. The app then went live on the pipeline.

Google Cloud

The Google Cloud services behind the pipeline

Pub/Sub

Dataflow

BigQuery

Cloud SQL

Cloud Storage

Cloud Composer

The pipeline in numbers

About 100

Users when the app launched

10–1M

Users the pipeline is designed for

2

Live streams joined as events arrive

Can't forecast your data load?

Tell us what your app or product will stream and what you can't predict about it. We answer within 24 hours.

Case studies

More healthcare and data projects

Healthcare

Blood pressure app with 70% less error than paper logs

Arterio blood pressure app shown on a smartphone