Logistics company raises demand forecast accuracy from 70% to 92%

Unstable Python scripts replaced with Spark ML on Databricks and Azure, and a forecast that reads more than the shipment history

Machine learning and predictive analytics

Demand forecast dashboard on a laptop: weekly shipments with the forecast and its range ahead, the model's inputs, and the busiest regions and lanes

Who we worked with

Logistics company with a forecast that didn't scale

Our client is a logistics company whose demand forecasting system did not scale with what was asked of it. It wanted better forecast accuracy and efficiency, lower operating costs and more on-time deliveries.

Client under NDA

What we did

From fragile scripts to a forecasting platform

Unstable scripts and a forecast prone to errors

The forecasting system already ran on Azure, as Python scripts on Azure Kubernetes Service. It was unstable and prone to errors.

Databricks notebooks on Azure

We replaced the Python scripts with Databricks notebooks that use Spark ML, written in Scala. Azure Data Factory schedules and runs the notebooks. Azure Blob Storage holds the data they work on, the shipment history and the outside factors. The CI/CD for the notebooks and data pipelines is ours as well.

Cleaning the shipment history

For accuracy, we cleaned and pre-processed the historical shipment data before the model used it.

Diagram: shipment history and outside factors go via Blob Storage to Spark ML in Databricks, run by Azure Data Factory

How the forecast was rebuilt

A new platform, and more for the model to see

We replaced the Python scripts outright.

Outside factors next to the shipment history

Shipment history tells a model what demand looked like. It can't say why. Demand moves with the season, with the economy and with what competitors do, and a model that sees only past shipments meets each of those as a surprise. Ours is gradient-boosted tree regression in Spark ML. Beside the shipment history it takes in fuel prices, economic indices, the holiday calendar and competitors' freight rates.

Tech stack

Forecasting stack on Databricks and Azure

Databricks

Spark ML

Scala

Azure Data Factory

Azure Blob Storage

Our client's results

70% to 92%

Forecast accuracy, before and after

15%

Lower operating costs

18%

More on-time deliveries

25%

Higher logistics partner satisfaction

Demand forecast stuck on fragile scripts?

Tell us what your forecast predicts today and where its data comes from. We reply to every inquiry within 24 hours.

Case studies

Related data and design projects