The modern data landscape is not comparable to how it was 10 or 20 years ago. A new enterprise data architecture that is a hot trending topic in almost every data conference is the data mesh. It combines domain-driven design and product thinking to solve data and analytics problems.
In 2018, the worldwide business analytics and big data industry was worth US$168.8 billion. According to Statista, big data has an expected growth of up to US$274.3 billion by 2022. One of the trends predicted to disrupt and dominate the data market is data mesh.
But what is data mesh, and why are more companies looking to implement it in managing their data?
A Deep Dive into Data Architecture
Data is a crucial driver of innovation in most companies today. It is the source of valuable and actionable insights that allow companies to grow. However, data collection and management have been critical problems that companies face in achieving their goal.
For a long time, companies have relied on centralized and monolithic systems that are based on a data lake. Despite the enormous investments made into such platforms, organizations have found average and unpromising results.
As organizations grow, their dependence on data to make competent business decisions also grows. There are numerous sources of data and plenty of different data consumers. All the data flows through one platform. This creates a management problem for ETL engineers when they want to keep up with the sources. At the same time, the systems need constant maintenance with every slight change.
The centralized architecture of data lakes and warehouses may also cause traffic between data engineers and domain experts. Only the data producers have the domain expertise and can change how the data is shaped. Data consumers have plenty of interest in the data since they understand its importance and potential.
The data engineers fall in-between the two parties. They are responsible for ensuring that the data delivery is reliable and of high quality. However, they cannot influence the data created in any way. This results in frustrated teams that feel disconnected from the data. Such problems make the system fail to meet its deliverable on ensuring that an organization becomes data-driven.
Therefore, there is a need to assimilate to a data management architecture that can solve the problem. Here is where a data mesh comes in.
The Data Mesh Solution
Data mesh is the Holy Grail for organizations looking to decentralize their data. It introduces a new organizational perspective and does not rely on any specific technologies. A data mesh is different from a traditional monolithic data infrastructure in handling the consumption, transformation, storage, and output of data. Monolithic data infrastructures use one central data lake.
A data mesh supports domain-specific data consumers where each domain has an independent data pipeline. It is increasingly necessary when a company wants to scale. A traditional monolithic data infrastructure uses only one team to manage all the data ingestion and transformation before serving it to all the potential stakeholders. Such a system may lead to scaling issues.
With data sources increasing every single day, organizations should always consider their best options for scaling. Data mesh provides organizations with the best of both worlds. Businesses get a distributed data lake with domains responsible for their pipeline.
As a result, scaling becomes effective since the mesh can break down data architectures into smaller, domain-oriented components. It also helps streamline data management and governance by allowing the data producers and consumers to work together as closely as possible.
An almost perfect organization is where both teams produce and consume the same data. This allows for the equal sharing of interest and responsibility. However, the practicality of such a system may not be possible.
The data-producing team is overloaded with so many responsibilities that they cannot own a data-consuming application. However, getting rid of a middleman and ensuring direct communication are equally significant, and the business should redirect its resources towards this endeavour.
Data mesh helps to streamline an organization by creating bilateral relations between data producers and consumers. It ensures that a particular domain’s responsibilities belong to one team, thereby reducing friction while also increasing ownership.
Data Mesh Cons
But even with all the benefits brought about by a data mesh, particularly its decentralized property, there are several setbacks that it also faces. Managing several data products may not be as easy as it sounds. You may end up with a messed-up data pipeline if you are not careful.
Here are some barriers that you may have to overcome when handling data mesh:
Data duplication
Data mesh repurposes data from the main to serve a new domain’s business needs that differ from the source domain. As a result, redundancy may occur, and this may affect the utilization of resources and data management costs.
Data governance
When the data products and pipelines are independent, it is easy to neglect the quality principles leading to technical debt. As a precaution, there should be an appropriate way to identify and govern the data.
Change in management
The adoption of data mesh requires a change in the management. Plenty of effort is needed to execute the change.
You do not necessarily need to rely on data mesh if you want to scale to many teams. Companies can still use a centralized data platform to scale to large operations.
A data mesh is much more suited for companies whose culture makes it difficult for them to scale. If the organization can coordinate its data management, it can still maintain a centralized data platform to avoid the overheads that come with decentralization.
Conclusion
It is clear that the value of adopting such an architecture for your organization is immense. If you have a goal of having a data-driven organization, then having a data mesh should be one of your major priorities.
A data mesh empowers your company to intelligently provide your clients with the best customer experience that is based on data. It also reduces your operational cost and minimizes the misinterpretation of data and communication problems. Do not hesitate to reach out to Softlanding if you would like to learn more.
Written By:
softlanding
Softlanding is a long-established IT services provider of transformation, professional services and managed IT services that helps organizations boost innovation and drive business value. We are a multi-award-winning Microsoft Gold Partner with 13 Gold Competencies and we use our experience and expertise to be a trusted advisor to our clients. Headquartered in Vancouver, BC, we have staff and offices in Toronto, Montreal and Calgary to serve clients across Canada.
More By This Author
