My team is developing custom LLMs and we find that traditional Waterfall is too rigid, but standard Scrum feels a bit messy for long training cycles. How do you balance the experimental nature of R&D in Deep Learning with the need for clear milestones and stakeholder reporting in a corporate setting?
3 answers
For AI projects, a hybrid "Agile-Research" framework often works best. You can use Scrum for the software engineering components (API development, UI, integration) but utilize Kanban for the data science and model training phases. This allows for the "discovery" phase of Deep Learning where results are unpredictable. Set "Research Sprints" with the goal of answering a specific hypothesis rather than delivering a feature. Use MLflow or DVC to track experiments as if they were code versions, ensuring that your stakeholders see progress through data metrics.
Are you finding that your training bottlenecks are causing "idle time" in your Sprints? How do you currently handle the handoff between data engineers and model researchers?
We switched to a "Lean AI" approach. We focus on getting a baseline model into production as fast as possible, then use iterative loops to improve accuracy based on real user data.
That's the way to go, Megan. The "MVP" for AI is often overlooked, but it's the only way to validate if the model actually solves the business problem before spending months on fine-tuning.
Steven, the bottleneck is real. We often have developers waiting for the GPU clusters to finish a run. To fix this, we've started overlapping tasks—engineers work on the next feature's infrastructure while the researchers analyze the previous model's output. It’s not a perfect flow, but it reduces the downtime significantly compared to our previous sequential approach.