12-04, 17:00–17:30 (UTC), Data/ Data Science Track
Anomaly detection is hardly a new problem, nor is the progress in it as rapid as the LLM blast we’re witnessing today. But it is pressing.
In this talk, we’ll talk about a realtime anomaly detection pipeline on time series data and discuss the nitty-gritties of the algorithm knobs that help us build an unbiased and reliable system, which includes 1) using NeuralProphet, an open source framework, to forecast for time series data and 2) using robust techniques to detect true anomalies using forecasting errors.
With the boom of data, we’re seeing real-time systems cropping up and that means thousands of data streams flowing that power every experience across the internet. But with this, arises the critical need to monitor data quality and metrics, and alerting systems that help escalate unexpected behavior with high confidence.
In this talk, I’ll discuss anomaly detection for time series data, especially set in the context of real-time systems. Whether you’re a data engineer hoping to implement a build time monitoring pipeline or an ML engineer working on model monitoring systems, this talk has something for you.
We’ll focus on one approach to detect anomalies, which is to forecast for a period in the future and alert if the ground truth is far from this prediction by some amount. Easy-peesy, right? But for this to work, we need
- A performant and efficient forecasting algorithm, that works well for all kinds of data, and,
- An unbiased, and dynamic approach to detect anomalies and not noise
For the first, we’ll leverage NeuralProphet - which is an open-source framework for ML based forecasting that combines the interpretability of linear models with the accuracy of deep neural networks.
For the second, we’ll discuss implementing some robust techniques and how they’re useful in the real-time production environment, to prevent anomalous data biasing our system as early as one second later.
At the end of the talk, the audience will
- have knowledge of an end-to-end demo highlighting NeuralProphet’s capabilities
- be equipped with handy tips and techniques to implement automated anomaly detection using open-source tools.
No previous knowledge expected
I'm a data scientist at Intuit in California, USA and I work on the anomaly detection capability that tracks authentication and business health metrics at Intuit. I was previously building NLP models at GoDaddy, but I enjoy working with data in general. I'm a Python enthusiast and enjoy sharing my learnings with the community - I've previously presented at the Grace Hopper Conference, PyCon US, EuroPython, and GeoPython. When not opposite a screen, I can be found frolicking in nature and exploring new trails.