Fintech company takes analytics off its production database

A Vertica warehouse and BI dashboards built from scratch, designed to migrate easily later and to keep control of the data in-house

Data warehouse development

Management overview dashboard on a laptop: transactions per day by product and by channel, new accounts per week and published reports

Who we worked with

Fintech company

Our client is a fintech company whose reports ran on the same production database as its transactions. It wanted a data warehouse and BI built from scratch.

Client under NDA

What we did

From production queries to a warehouse

Reports were running on the production database

Analytics ran as queries straight on the production database. Reports were slow, and running them disrupted transactions. Processes were unregulated and BI expertise in the company was thin, and our client wanted its data to stay fully under its own control.

A proposal with every tool's trade-offs on paper

We started with an architecture proposal. It listed the candidate tools with the advantages and disadvantages of each, and we agreed it with top management. Then we led the build against the agreed requirements. A team of seven ran it over nine months, with two senior data engineers, a data engineer, two BI developers, a DevOps engineer and a project manager who also worked as the business analyst.

The warehouse, its pipelines and the dashboards

The warehouse is Vertica and SQL. The pipelines are Python and Apache Airflow on Kubernetes (Amazon EKS). Our DevOps engineer set up the cluster, CI/CD and infrastructure as code. Three new processing systems were integrated. Dashboards in Tableau and Apache Superset give top management the numbers its decisions rest on. Requests for data and reports now go through Jira Service Desk, and after it came in, the team handling them and the reporting had about 20% less operational workload.

Diagram: the production database and three new systems feed Python and Airflow into Vertica, then Tableau and Superset

How it was designed

A warehouse built to be moved later

Picking a first platform means guessing at future needs

A first warehouse platform is picked before anyone can see what reporting will need in a few years, so the design should make a later move cheap. The transformation logic lives in Python and Apache Airflow, outside Vertica, and the model is plain SQL. A later move means repointing the pipelines and leaving their code as it is. Control of the data stayed with the client. The warehouse and its pipelines ran in its own AWS account, and our engineers worked through named accounts it issued, each with only the access its role needed.

What runs through the warehouse now

All regular reports now run through the warehouse, and it kept running without issues long after the engagement ended.

The stack

Warehouse, pipelines, BI and the request desk

Vertica

SQL

Python

Apache Airflow

Kubernetes

Amazon EKS

Tableau

Apache Superset

Jira Service Desk

The project in numbers

0

Analytical load left on production

3

New processing systems integrated

9 months

Engagement, with a team of seven

Reports slowing down your production database?

Tell us what your reports run on today and who depends on them. We reply to every inquiry within 24 hours.

Case studies

Other data projects we delivered