I'm transitioning from a traditional Software Development role into a team that focuses on data science. I understand CI/CD, but I'm hearing that managing an ai workflow requires a completely different mindset. What are the specific "gotchas" in MLOps that I should prepare for to keep our models from drifting?
3 answers
The biggest difference is that in DevOps, you only version code. In an ai workflow, you must version the code, the model, and the data. If the data changes slightly (data drift), your code might be perfect, but your results will be garbage. You need automated monitoring to alert you when the statistical distribution of your input data shifts. This is a core part of MLOps. We use tools like MLflow to track every run of our ai workflow so we can roll back to a specific model version if things go south.
Are you planning to use a centralized model registry for your ai workflow? It really helps in keeping track of which model is in staging versus production.
Don't forget about "Concept Drift." Your ai workflow needs to be retrained periodically because the real world changes, even if your data format stays the same.
Larry is right. A model trained on 2023 data might fail in 2024. Continuous retraining is the only way to keep an ai workflow accurate over time.
We haven't picked a registry yet, Kenneth. We were just going to use S3 buckets for the ai workflow artifacts, but I'm starting to see how that could get messy once we have dozens of different model versions and experiments running simultaneously.