We are currently setting up a pipeline that spans both on-prem servers and AWS. How does handle artifact storage in such a fragmented environment? Are there specific configurations to ensure we don't lose data when training across different geographic regions?
3 answers
For a hybrid setup, the key is using a centralized artifact store like S3 or an Azure Blob that all environments can reach. is quite flexible here; you just need to set the MLFLOW_S3_ENDPOINT_URL or equivalent. In our global setup, we used a centralized tracking server but kept local caches for training data to reduce latency. The tool handles the metadata globally, so a scientist in London can see the results of a run performed in a New York data center instantly. The most important thing is ensuring your IAM roles or access keys are correctly mapped across your on-prem and cloud environments to avoid "access denied" errors during logging.
That makes sense for storage, but what about tracking latency? If my tracking server is in the US and I'm training in Asia, will slow down my training loops significantly due to the network round-trips?
The setup is very robust for multi-region work. Just make sure your database backend for the tracking server is high-availability, otherwise a single outage can stop all your logging.
Great point, Joseph. We used a managed SQL instance for the backend and it has been rock solid for over a year now. The reliability is definitely there for enterprise use.
Michael, the logging is usually asynchronous or very lightweight, so it shouldn't impact your training time much. However, if you are logging massive artifacts like 10GB model files, that’s where you’ll feel the bottleneck. A good strategy is to log metrics to the central server but save large artifacts to a regional bucket first, then sync them later. This keeps your experiments moving fast without sacrificing the global view of your model performance.