In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving, training, CI/CD, feature serving, and model monitoring systems.
On the real-time front, LyftLearn supported real-time inference and input feature validation. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.
While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft.
Lyft is a real-time marketplace and many teams benefit from enhancing their machine learning models with real-time signals.
To meet the needs of our customers, we kicked off the Real-time Machine Learning with Streaming initiative. Our goal was to develop foundations that would enable the hundreds of ML developers at Lyft to efficiently develop new models and enhance existing models with streaming data.
In this blog post, we will discuss some what we built in support of that goal and the lessons we learned along the way.
Capabilities of Real-time Machine Learning
One of the first questions we asked ourselves is — what are the general use cases within the ML ecosystem that can leverage streaming data?
We identified three main capabilities of real-time ML applications which could leverage streaming:
- Real-time Features → Computing features with real-time streaming data
- Real-time Learning → Training models with real-time streaming data
- Event Driven Decisions → Making decisions, e.g. retraining a model, triggering an alert, or running an inference call, with real-time streaming data