Our current structure has data scientists in one silo and software engineers in another. It's making it impossible to do true agile MLOps. How do successful companies organize their teams? Should everyone be a cross-functional engineer, or do we need dedicated roles for ML infrastructure? I am looking for advice on how to foster collaboration and speed up our deployment velocity.
Agile MLOps teams are most effective when organized into cross-functional squads containing embedded data scientists and MLOps engineers, supported by a central platform team that manages infrastructure and CI/CD standardization.
5 answers
To bridge the gap between research and production, you should implement a standardized organizational framework that ensures both skill sets are represented in every delivery cycle.
- Embed at least one data scientist and one dedicated MLOps engineer within every squad to ensure shared ownership of the model lifecycle.
- Establish a centralized platform engineering team to manage the underlying Kubernetes clusters and CI/CD tooling, providing common abstractions for the product squads.
- Rotate members periodically to ensure that cross-functional expertise is distributed rather than concentrated in singular silos.
- Standardize all experimentation tracking and model registries so that production deployments remain reproducible regardless of which team pushed the update.
Cross-functional squads organized around specific business domains are the only way to effectively achieve agile MLOps. Segregated siloes inevitably lead to a lack of accountability regarding model decay and infrastructure drift because the feedback loop between production performance and model training is severed.
I remember back at a previous consultancy where we had data scientists throwing black-box models over the wall to engineers who had no idea how to interpret the telemetry. We were shipping disasters until we finally broke the silos and forced the scientists to sit in the same stand-ups as the infra team.
It was messy for the first three sprints as everyone learned the other side's vocabulary, but our deployment frequency quadrupled once the engineers could actually build pipelines that the scientists understood. You don't need everyone to be a polymath, but they do need to share the same metrics and the same sprint goals.
The choice between generalized cross-functional roles and specialized infrastructure teams is a matter of scale versus agility. A fully cross-functional model allows for rapid prototyping and higher feature velocity because researchers have full control over the stack, yet this often leads to significant technical debt and inconsistent deployment patterns as the codebase grows. On the other hand, relying on a dedicated infrastructure team ensures high reliability and compliance with enterprise security standards, but it frequently introduces bottlenecks that stifle the iterative nature of machine learning development.
The optimal approach usually involves balancing these two extremes by creating internal platforms that allow data scientists to self-serve their infrastructure requirements while keeping the core pipeline hardened by experts. This hybrid model prevents the total loss of control associated with complete decentralization while avoiding the crushing latency of a legacy gatekeeper structure.
Stop treating data science as an isolated research function and start treating it as a production engineering discipline. You need cross-functional teams that share the pager, because if the data scientist doesn't feel the pain of a 3 AM model failure, they will never write maintainable production code. Dedicated infra teams are fine for scaling, but if they aren't embedded in the product squads, you are just building another silo with a different name. Combine the teams or watch your deployment velocity remain stagnant.
