Managing code with Git is easy, but how do I handle versioning for 10GB datasets and the resulting pickle files of my models? I’m finding it impossible to keep track of which dataset version produced which model accuracy. Are there industry-standard tools for "Git for Data"?
3 answers
You are hitting a common bottleneck in the MLOps lifecycle. You should look into DVC (Data Version Control). It works alongside Git by storing the actual large files in a remote bucket (like S3 or Azure Blob) while keeping small .dvc pointer files in your Git repository. This allows you to "checkout" specific versions of your data just like you do with code. For tracking the relationship between the data, code, and model performance, MLflow is the current industry leader. It lets you log parameters, metrics, and the model artifact itself in a centralized dashboard, ensuring that every experiment is 100% reproducible.
DVC sounds powerful, but isn't the learning curve a bit steep for a small team? Would just using Git LFS (Large File Storage) be enough if our datasets aren't changing that frequently, or is the integration with ML pipelines worth the extra setup?
Keep it simple to start: use a naming convention for your S3 buckets and log everything in a shared Google Sheet or a simple MLflow instance.
While the spreadsheet method works for a week, it fails quickly. I’d take Sarah's advice on MLflow but skip the manual logging—automate it so you don't forget!
Kevin, Git LFS is okay for storage, but it lacks the "pipeline" awareness that DVC offers. DVC tracks the dependencies—meaning if your data changes, it knows exactly which processing scripts need to be re-run. For a serious Data Science team, that automation is worth the initial learning curve.