Data Mesh vs. Data Fabric: Which Approach Wins?

There have been many great rivalries over the years. Yankees vs Red Sox. Barcelona vs Real Madrid. Tyson vs Holyfield. And now, arguably the greatest rivalry the world (well, at least the data community) has ever witnessed: Data Fabric vs Data Mesh! Grab the popcorn. We’ll save you the $80 pay-per-view fee and give you a front-row seat into this exciting match up.

Data mesh vs. data fabric: Whats the difference?

In simple terms, data mesh emphasizes the significance of people and processes, whereas data fabric is an architectural approach that effectively manages the intricacies of data and metadata in a cohesive manner. Both approaches have their own advantages and limitations, and it’s crucial to evaluate them carefully to determine which one is best for your organization.

Gartner calls data fabric the Future of Data Management¹. Thoughtworks says data mesh is key to moving beyond a monolithic data lake. But which one is right? Which one is better? Spoiler alert: data fabric and data mesh are independent design concepts that are, in fact, quite complementary.

Data fabric has captured most of the limelight; it focuses on the technologies required to support metadata-driven use cases across hybrid and multi-cloud environments. Data mesh, on the other hand, takes a more people- and process-centric view. Data mesh forgoes technology edicts and instead argues for “decentralized data ownership” and the need to treat “data as a product”. This approach, Thoughtworks argues, overcomes the bottlenecks and disconnects that are typical of data lake and data warehouse environments — disconnects that arise as data engineers play middle-men between data producers and consumers.³

Moreover, data catalogs play a central role in both data fabric and data mesh. Gartner is explicit that an augmented data catalog is foundational to a data fabric. Indeed, a data catalog plays a crucial role in extracting and analyzing metadata from an organization’s data sources to fuel the data fabric. (See diagram below.)

Data fabric describes an interwoven technology stack; an augmented data catalog is a key foundation.

What Is a Data Fabric?

Well, it depends on who you ask. There are vendors out there that will have you believe their product is an example of a data fabric — some even have ‘Data Fabric’ in their product name. All of this is no doubt well-intentioned, but it does confuse the market. The definition we are going with here is Gartner’s and, to them, there is no single vendor that addresses the complete set of needs required to build a data fabric (at least not today). Gartner defines data fabric as a “design concept that serves as an integrated layer (fabric) of data and connecting processes.”

A data fabric utilizes continuous analytics over existing, discoverable and inferenced metadata assets to support the design, deployment and utilization of integrated and reusable data across all environments, including hybrid and multi-cloud platforms.⁴

We’ll dig into this definition in a bit. For now, note these important key words: integrated and reusable data. This, in essence, is the goal of a data fabric. At the core of data fabric is the intelligent analysis of metadata supporting a smarter system of integration, enabling trusted and reusable data to be leveraged by the widest possible group of consumers – humans and machines alike.

Let’s turn now to the rest of the definition. Gartner says a data fabric is a design concept. In other words, a data fabric is not a single thing or product (though a single product could support elements of the entire data fabric stack). It is instead composable, made up of a set of integrated technologies — read this for a deeper dive on those technologies — that accelerate value from enterprise metadata. Gartner also acknowledges that data is sitting everywhere today in hybrid and multi-cloud environments (which, at this point, should go without saying.)

Data fabric examples

The adaptability of a data fabric architecture provides numerous benefits. Examples of how data fabric is being used include:

Improving machine learning (ML) models

Providing the correct data to AI models in a timely manner can significantly enhance their learning capabilities. Machine learning calculations can actively screen data pipelines and suggest logical connections and integrations. By efficiently extracting data from the data fabric, they can thoroughly analyze all business data, inspecting it to identify and distinguish relevant associations and connections.

Customer insights

Organizations have the opportunity to leverage a data fabric to gather data from client activities and discover how interactions with customers can offer even greater value. This may involve consolidating real-time data from various sales activities, analyzing the time it takes to acquire a customer locally, and measuring customer satisfaction through KPIs.

Data fabric benefits

Here are some advantages of using a data fabric approach:

• Simplified data access: Data fabric provides a unified view of data, making it easier for organizations to access and use their data.
• Real-time data access: Data fabric enables organizations to access data in real time, allowing faster and more informed decision-making.
• Improved data integration: Data fabric allows for seamless data integration from different sources and formats, eliminating data silos and enabling better collaboration between teams.
• Automated data management: Data fabric automates many data management tasks, such as data integration, data quality, and data governance, reducing the workload on IT teams and allowing them to focus on strategic initiatives.
• Scalability: Data fabric is designed to be scalable, allowing organizations to easily add new data sources and users as their needs grow.
• Enhanced security and compliance: Data fabric provides a secure and compliant way to manage data, ensuring that sensitive data is protected and that organizations meet regulatory requirements.

In the grand scheme, data fabric platforms provide an array of impressive benefits. With its unique ability to integrate data from multiple sources and formats, data fabric empowers organizations to gain valuable insights and make well-informed decisions based on a comprehensive view of their data. Let’s put it all together by articulating the rationale for why a data fabric is needed in the first place.

The data fabric

With so much complexity emerging from data landscapes, people need a means to find trusted data alongside guidance on how to use it. The key is to capture wisdom in the community. Both data fabric and mesh enable people to use and reuse data by making the most valuable assets the most visible for wider use. This empowers newcomers to leverage insights and integrations others have built before them. It also ensures that established knowledge (and valuable processes) are woven into the system of data distribution.

But how do you identify the best data, and best practices for using it? The key is metadata. Metadata is the key to fueling data intelligence use cases across the board, including data search & discovery and data governance. Metadata (or “data about data”) captures the who, what, where, when, and how of every asset – to flesh out its “why” – and helps newcomers understand and use that asset more quickly. But accessing and making sense of metadata is extremely challenging in today’s environment.

A big reason is that metadata is everywhere. It’s in all types of data management systems, from databases to ERP tools, to data integration software. And metadata could be sitting in many different locations, including on-premises, in the cloud, and everywhere in between.

Humans are hard-pressed to find relevant metadata, let alone make sense of it. Data fabric, says Gartner, is the answer. By using technologies to automate the discovery and continuous analysis and reuse of metadata, organizations will overcome the challenges associated with its proliferation and reduce the error-prone manual efforts that go with making sense of it. These technologies are broadly categorized as data intelligence solutions.

In fact, data intelligence technologies support building a data fabric and realizing a data mesh. As a discipline, data intelligence weaves together “the traditional categories of metadata management, data quality, data governance, master data management, data profiling, and data privacy while incorporating intelligence derived from active metadata.”⁷

Let’s turn our attention now to data mesh.

The Data Mesh

For data mesh, the experts agree a “self-serve data platform” is essential to ensure teams can autonomously own their data products. A data catalog is not specified by name since the data mesh is technology-agnostic. But make no mistake: A data catalog solution addresses many of the underlying needs of this self-serve data platform, including the need to empower users with self-serve discovery and exploration of data products.

Zhamak Dehghani of Thoughtworks is widely credited with having conceived of data mesh in a blog post back in May 2019. Follow-up blogs clarify architectural aspects of data mesh, but all remain true to the founding vision and approach first introduced in 2019.⁸ Vendors are now putting their own spin on data mesh, which will no doubt introduce some confusion. Yet these vendors universally cite the work of Dehghani and Thoughtworks as the basis for their “take” on data mesh.

The origin story is clear, but a concise definition is harder to come by. (Most of Deghani’s public write-ups focus on motivating the data mesh and key principles of the data mesh architecture.) Fortunately, Arif Wider, also at Thoughtworks, offers a clear definition:

The data mesh paradigm is a strong candidate to supersede the data lake as the dominant architectural pattern in data and analytics. Importantly, the data mesh mainly introduces a new organizational perspective and is independent of specific technologies. Its key idea is to apply domain-driven design and product thinking to the challenges in the data and analytics space. Comparable to the introduction of a DevOps culture, establishing a data mesh culture is about connecting people, creating empathy, and about creating a structure of federated responsibilities. This way, generating business value from data can be scaled sustainably.⁹

Before we take a closer look into why organizations need to adopt a new architectural approach, let’s go over how a data mesh is used and what key benefits it can offer to your organization.

Data mesh examples

Data mesh can support a range of analytical and operational use cases in various domains.

Here are a few examples:

Customer views

By leveraging data mesh, client care can reduce the average handle time, increase the resolution of first contact, and enhance overall consumer satisfaction. In addition, marketing teams can use a single view of the customer for predictive churn modeling or next-best-offer decision-making.

Hyper-segmentation

Data mesh allows marketing teams to precisely target the right clients at the right time and through the right channels to deliver the most effective messaging.

Data privacy management

To ensure compliance with emerging regional data privacy laws such as VCDPA, data mesh can protect customer data before it is accessible to data consumers in various business domains.

Data mesh benefits

Below are some benefits of implementing a data mesh approach:

• Decentralization of data governance: With data mesh, individual teams can take ownership of their data domains and decide on data quality, access, and privacy. This allows for a decentralized approach to data governance.
• Scalability: Data mesh treats data as a product, which enables organizations to scale their data capabilities to meet growing demands while reducing duplication and redundancy of effort.
• Agile development practices: Data mesh promotes agile development practices, which means that teams can iterate quickly and experiment with new data products without being held back by centralized data governance.
• Democratization of data: Data mesh makes data accessible to all teams, allowing them to derive insights and make data-driven decisions without relying on centralized data teams.
• Encouragement of innovation: Data mesh fosters a culture of experimentation and collaboration, which enables teams to share insights and knowledge to promote innovation.

A data mesh provides an array of impressive benefits and can be used to gain a competitive edge over your competition, but as previously mentioned, Wider calls for a new architectural approach, one that will supersede the data lake. But why?

Why Do You Need a Data Mesh?

Much has been written about how data lakes have failed us all. How they’ve turned into data swamps due to lack of organization, governance, and accessibility. For Wider, the underlying issue with data lakes is straightforward and can be captured in one word: centralization.

A central team is responsible for maintaining the central infrastructure (AKA the data lake). This team is usually disconnected from the needs of data consumers and often lacks the domain expertise of data producers. Yet here they are, forced to play middle-men between consumers and producers because the prevailing data lake architecture forces the teams to be organized this way. The end result is a team that doesn’t scale and data being served up to consumers which may or may not meet their quality needs.

Data mesh inverts this model with domain-driven design and product thinking. Responsibilities are distributed to the people who are closest to the data. These product owners are responsible for delivering data as a product and, as such, they are accountable for objective measures, including “data quality, decreased lead time of data consumption, and general data user satisfaction…” ¹⁰

In other words, data mesh is all about people, calling for a shift in responsibilities to ensure high quality data is put in the hands of data consumers faster and more efficiently. As we’ll see in parts 3 and 4 of this series, however, technology does play a very important enabling role.

Data Mesh vs. data fabric: Choosing the best approach

There are many things to consider when choosing the best approach for your organization, including:

1. Evaluating your organization’s specific needs and goals for data management. Consider factors such as agility, innovation, scalability, data quality, and data governance.
2. Researching and gaining a thorough understanding of both data mesh and data fabric approaches. Learn about the benefits and limitations of each approach and how they can address your organization’s needs.
3. Assessing the current state of your organization’s data infrastructure, data governance, and data operations. Determine the level of complexity and potential challenges in implementing each approach.
4. Considering the resources required for implementing each approach, including time, budget, and expertise. Evaluate whether your organization has the necessary resources to implement and maintain each approach.
5. Determining the level of integration required for your organization’s data. Consider whether a unified view of data is necessary or if decentralized data management is more suitable.
6. Consulting with experts in the field and other organizations that have implemented either approach. Gain insights into their experiences, successes, and challenges.
7. Developing a roadmap for implementing the chosen approach. Consider the necessary steps, resources, and timeline for successful implementation.
8. Continuously evaluate and monitor the effectiveness of the chosen approach. Make necessary adjustments to ensure the approach continues to meet your organization’s evolving needs and goals.

In conclusion: The final tale of the tape

Choosing between data mesh and data fabric is an important decision that requires careful consideration of your organization’s unique needs and goals. At Alation, we’re committed to helping organizations make informed decisions about their data management strategies.

New to data catalogs? This white paper, How to Evaluate a Data Catalog, walks you through what to look for.

---

1. https://www.gartner.com/document/3994025

2. https://martinfowler.com/articles/data-monolith-to-mesh.html

3. https://www.thoughtworks.com/en-us/insights/blog/data-mesh-its-not-about-tech-its-about-ownership-and-communication

4. https://www.gartner.com/smarterwithgartner/data-fabric-architecture-is-key-to-modernizing-data-management-and-integration

5. Though a single product could support elements of the entire data fabric stack

6. See blog 2 for a deep dive on the technologies involved.

7. https://www.forbes.com/sites/forbestechcouncil/2021/01/28/what-is-data-intelligence-and-how-can-it-help-your-organization/?sh=46313f1e4033

8. https://martinfowler.com/articles/data-monolith-to-mesh.html

9. https://www.thoughtworks.com/en-us/insights/blog/data-mesh-its-not-about-tech-its-about-ownership-and-communication

10.https://martinfowler.com/articles/data-mesh-principles.html#DomainOwnership