Once a model is deployed, monitoring for performance decay is critical. Does Hugging Face provide native features to detect data drift, or is it strictly a model repository and training framework?
3 answers
Currently, the ecosystem is heavily focused on the "build and share" aspects of the lifecycle. While they have introduced "Inference Widgets" and "Spaces" to showcase models, they don't have a full-blown observability suite like Arize or WhyLabs integrated directly into the Hub. You can version your datasets and models, which helps in retraining once you've identified drift elsewhere, but the actual detection of that drift—comparing live feature distributions against training data—is something you'll likely need to handle with a separate MLOps monitoring tool for now.
Are you planning to automate the retraining trigger, or are you looking for a manual dashboard that alerts your data science team when the F1-score drops?
It’s mostly for hosting and sharing. For the "Ops" part of MLOps, you’ll definitely need to look into external integration or custom scripts to track your production metrics.
Exactly, Lisa. While Hugging Face simplifies the model access, the responsibility of maintaining the model's health in a live environment still falls on the traditional MLOps stack you build around it.
Jason, we are ideally looking for an automated trigger. If the model starts misclassifying due to a shift in user behavior, we want the system to pull the latest dataset from the Hub and start a fine-tuning job automatically. It seems like we can use GitHub Actions or a similar CI/CD tool to bridge the gap between our monitoring software and the Hugging Face training scripts.