Our global data team is a bottleneck. Every time a regional department wants a new data pipeline, they have to wait months. We are considering moving to a Data Mesh architecture where each domain owns its own data lake. Is this just a buzzword, or does it actually solve the problem of data ownership and scalability?
3 answers
Data Mesh is definitely more than a buzzword, but it’s a cultural shift as much as a technical one. In a mesh, the "Sales" team owns their data lake and provides it as a "Data Product" to others. This stops the central team from being a bottleneck. However, you must have a "Self-Service Infrastructure" team to provide the tools, or you'll end up with ten different ways of storing data. We moved to a mesh in late 2023, and while it empowered our domains, it took six months just to agree on the "Federated Governance" standards so everyone could still join data together.
Do your domain teams actually have the data engineering talent to manage their own pipelines, or will this move just shift the bottleneck from IT to the business units?
A Data Mesh is basically "Microservices for Data." It’s great for speed, but you need a very strong central catalog so people can actually find the products.
Thomas makes a vital point. Without a central "Marketplace" for these data products, the decentralization will just lead to a massive increase in duplicated datasets.
Patrick, that is our biggest fear. Our marketing team knows GA4 and SQL, but they don't know Spark or Kubernetes. If we go the Data Mesh route, we’ll likely have to hire "Domain Data Engineers" for each unit. The cost will be higher, but the speed of delivery for insights should theoretically triple. We are trying to figure out if the ROI on that extra headcount actually balances out in the long run.