Alation and dbt Unlock Metadata and Increase Modern Data Stack Visibility
By Mark Kwiatkowski
Published on October 18, 2022
Alation is pleased to be named a dbt Metrics Partner and to announce the start of a partnership with dbt, which will bring dbt data into the Alation data catalog.
Yet every dbt transformation contains vital metadata that is not captured – until now. Making this data visible in the data catalog will let data teams share their work, support re-use, and empower everyone to better understand and trust data.
Data Transformation in the Modern Data Stack
Data engineering plays a critical role in distributing data to a wide audience. Yet for raw data to become information, it not only must move from source systems to analytics, it must be transformed. Transformation can include combining data from multiple systems, normalizing data so it can be consistently used, and preparing data for analysis. Data engineers, analysts, and data scientists often collaborate to transform data.
But what does this look like in practice? As an example, purchase date may be represented in several columns and even formats. A data engineer will use dbt to combine all purchase date data into a single field and apply a single format so that it can be consistently used. Data engineers will also use dbt to safeguard data quality; in this example they would ensure all of the values are indeed dates.
Taking this a step further, this same data can also be transformed to generate business metrics. Purchase date represents one customer touch point. But what if you could combine that information with usage metrics? You could derive insights that help you improve customer experience. In many organizations, these key metrics are calculated in many independent ways, leading to inaccuracy and delays as multiple versions of the same metrics need to be rationalized.
To address this problem, dbt metrics provides a path to standardize metrics under version control in dbt projects. “By abstracting metrics calculations out of pre-aggregated tables or specific BI tools, dbt metrics can be defined once and used everywhere,” the website states. Making these metrics widely known is essential – as it will lead to better adoption and utilization of these centrally defined metrics.
Benefits of the Alation and dbt Partnership
The new dbt Semantic Layer makes this information, previously locked in the dbt processes, accessible. When combined with the dbt metadata API, a rich set of data, capturing its transformation history, can now be added to the Alation data catalog.
Bringing this valuable information into the Alation data catalog will:
Expand the visibility and value of defined dbt metrics
Users can now quickly search, discover and understand the metrics within the data catalog.
Increase trust by granting data analysts and engineers
Users enjoy visibility into the transformations and metrics shaping key data for analytics.
Accelerate data processing and engineer productivity
Data teams will enjoy greater visibility into the data they are working on and transformations in use across the organization, helping them be more productive.
Improve data analysis accuracy
By giving data teams ongoing visibility into business metric definitions and governance, including endorsements, collaboration, and lineage visibility, this partnership will support analysis accuracy.
How will these updates manifest in the data catalog? When this rich data from dbt is combined with the Alation APIs, the Alation data catalog will be able to show:
Curated dbt descriptions for models, sources and metrics accessible for all data consumers in the data catalog
Lineage between dbt sources, models, and metrics
dbt transformations used on tables and columns found in the data catalog
Business metric definitions, including the description, configuration and calculation criteria
These insights will prove invaluable to business users and analysts seeking to better understand data’s path to the data catalog. By granting all data users visibility into metric definitions, people better can understand what key business terms actually mean.
“Our partnership between Alation and dbt Labs yields an incredibly powerful tool for any data user looking to improve data quality and make better decisions faster,” said Raj Gossain, Chief Product Officer at Alation. “We forged this strategic partnership to increase data intelligence across the modern data stack, making metadata about transformations and metrics visible, trusted, and usable to all joint Alation and dbt Labs customers. Together, we’re changing how data leaders and data consumers collaborate by increasing visibility, trust, and accuracy for more people across organizations.”
Data Intelligence for the Modern Data Stack
This combination of Alation and dbt will help bring much-needed data intelligence to the modern data stack. As data teams move data through pipelines and into flexible data platforms like Snowflake and Databricks, the transformation of that data, and preparing it for analytics and creating business metrics, is a critical step.
A lot can happen in the course of that transformation. Raw data may be cleaned, erroneous data may be deleted, and new metrics may arise. How did the data transform exactly? Who was involved? Why did they make the choices they did? These are key details. But if the information on these transformations stays in isolation, data teams may recreate the same transformations or interpret key business metrics in different ways.
Making this metadata visible to everyone is a much needed step to ensure that everyone creating new data sets can understand the work already in progress across the organization and consistently apply the common practices and metrics. As teams rapidly evolve data practices, this data intelligence will help them stay on track and even accelerate their work.
“dbt Labs is pleased to name Alation a ‘Metrics Ready’ partner that empowers data analysts and engineers to better understand and trust data,” said Nikhil Kothari, Head of Technology Partnerships at dbt Labs. “Alation is a valued data catalog and data intelligence partner in the modern data stack that helps everyone in an organization find, understand, and trust data. With dbt transformation and metrics data in Alation, information previously contained in the dbt is now accessible to all data consumers.”
dbt’s new features are a key element to increasing data intelligence across the modern data stack. Making metadata about transformations visible to everyone in Alation will be a powerful tool for any organization looking to speed their time to decision with data while ensuring that everyone is working together to safeguard data quality and trust.
This is just the first step in a growing partnership. Stay tuned for more updates about joint Alation-dbt capabilities. We’re looking forward to great things to come!
Curious to learn more? Read the press release on this partnership.
- Data Transformation in the Modern Data Stack
- Benefits of the Alation and dbt Partnership
- Data Intelligence for the Modern Data Stack