I am running a series of hyperparameter tuning jobs for a deep learning model, but I’ve noticed that calling the MLflow API for every single batch is significantly increasing my epoch time. Is there a recommended way to buffer these metrics or use asynchronous logging to ensure that the tracking overhead doesn't become a bottleneck during intensive training phases?
3 answers
The performance lag you're seeing is usually due to the synchronous nature of standard HTTP requests to the tracking server. To mitigate this, you should avoid logging at the batch level and instead aggregate your metrics to log only at the end of each epoch. Furthermore, you can utilize the mlflow.log_metrics() batch API, which sends multiple values in a single request, reducing the network round-trip time significantly. If you are using a remote tracking server, ensure your network latency is low. For extreme cases, consider logging to a local file system first and then syncing the artifacts to the central server after the training run completes to keep the compute focused on the gradients.
Have you looked into the fluent API's autologging features, or are you manually instrumentation every step of your training loop?
I usually set a step frequency for logging. Instead of every batch, I log every 50 or 100 steps. This keeps the UI clean and the training fast.
I agree with David. Sampling your metrics is the way to go. You don't need 10,000 data points for a single epoch's loss curve to see the general trend of your model's convergence!
That’s a valid question, Michael. While autologging is convenient, it can sometimes be the culprit behind unexpected overhead because it captures a lot of metadata by default. If Susan is using MLflow with high-frequency logging, switching to manual logging for only the most critical scalars like 'loss' and 'accuracy' at larger intervals would definitely help reclaim those lost seconds per epoch.