Alation and dbt Unlock Metadata and Increase Modern Data Stack Visibility

By Mark Kwiatkowski

Published on October 18, 2022

Someone pointing to a virtual lock near their laptop screen.

Alation is pleased to be named a dbt Metrics Partner and to announce the start of a partnership with dbt, which will bring dbt data into the Alation data catalog.

In the modern data stack, dbt is a key tool to make data ready for analysis. Data analysts and engineers use dbt to transform, test, and document data in the cloud data warehouse.

Yet every dbt transformation contains vital metadata that is not captured – until now. Making this data visible in the data catalog will let data teams share their work, support re-use, and empower everyone to better understand and trust data.

Data Transformation in the Modern Data Stack

Data engineering plays a critical role in distributing data to a wide audience. Yet for raw data to become information, it not only must move from source systems to analytics, it must be transformed. Transformation can include combining data from multiple systems, normalizing data so it can be consistently used, and preparing data for analysis. Data engineers, analysts, and data scientists often collaborate to transform data.

But what does this look like in practice? As an example, purchase date may be represented in several columns and even formats. A data engineer will use dbt to combine all purchase date data into a single field and apply a single format so that it can be consistently used. Data engineers will also use dbt to safeguard data quality; in this example they would ensure all of the values are indeed dates.

Taking this a step further, this same data can also be transformed to generate business metrics. Purchase date represents one customer touch point. But what if you could combine that information with usage metrics? You could derive insights that help you improve customer experience. In many organizations, these key metrics are calculated in many independent ways, leading to inaccuracy and delays as multiple versions of the same metrics need to be rationalized.

To address this problem, dbt metrics provides a path to standardize metrics under version control in dbt projects. “By abstracting metrics calculations out of pre-aggregated tables or specific BI tools, dbt metrics can be defined once and used everywhere,” the website states. Making these metrics widely known is essential – as it will lead to better adoption and utilization of these centrally defined metrics.

Benefits of the Alation and dbt Partnership

The new dbt Semantic Layer makes this information, previously locked in the dbt processes, accessible. When combined with the dbt metadata API, a rich set of data, capturing its transformation history, can now be added to the Alation data catalog.

Bringing this valuable information into the Alation data catalog will:

  • Expand the visibility and value of defined dbt metrics

Users can now quickly search, discover and understand the metrics within the data catalog.

  • Increase trust by granting data analysts and engineers

Users enjoy visibility into the transformations and metrics shaping key data for analytics.

  • Accelerate data processing and engineer productivity

Data teams will enjoy greater visibility into the data they are working on and transformations in use across the organization, helping them be more productive.

  • Improve data analysis accuracy

By giving data teams ongoing visibility into business metric definitions and governance, including endorsements, collaboration, and lineage visibility, this partnership will support analysis accuracy.

  • How will these updates manifest in the data catalog? When this rich data from dbt is combined with the Alation APIs, the Alation data catalog will be able to show:

  • Curated dbt descriptions for models, sources and metrics accessible for all data consumers in the data catalog

  • Lineage between dbt sources, models, and metrics

  • dbt transformations used on tables and columns found in the data catalog

  • Business metric definitions, including the description, configuration and calculation criteria

These insights will prove invaluable to business users and analysts seeking to better understand data’s path to the data catalog. By granting all data users visibility into metric definitions, people better can understand what key business terms actually mean.

 Alation Data Catalog showcasing dbt Metric - spend for closed transactions.
Zooming into Alation Data Catalog showcasing how quickly you can find and understand dbt Metrics.

“Our partnership between Alation and dbt Labs yields an incredibly powerful tool for any data user looking to improve data quality and make better decisions faster,” said Raj Gossain, Chief Product Officer at Alation. “We forged this strategic partnership to increase data intelligence across the modern data stack, making metadata about transformations and metrics visible, trusted, and usable to all joint Alation and dbt Labs customers. Together, we’re changing how data leaders and data consumers collaborate by increasing visibility, trust, and accuracy for more people across organizations.”

Data Intelligence for the Modern Data Stack

This combination of Alation and dbt will help bring much-needed data intelligence to the modern data stack. As data teams move data through pipelines and into flexible data platforms like Snowflake and Databricks, the transformation of that data, and preparing it for analytics and creating business metrics, is a critical step.

A lot can happen in the course of that transformation. Raw data may be cleaned, erroneous data may be deleted, and new metrics may arise. How did the data transform exactly? Who was involved? Why did they make the choices they did? These are key details. But if the information on these transformations stays in isolation, data teams may recreate the same transformations or interpret key business metrics in different ways.

Making this metadata visible to everyone is a much needed step to ensure that everyone creating new data sets can understand the work already in progress across the organization and consistently apply the common practices and metrics. As teams rapidly evolve data practices, this data intelligence will help them stay on track and even accelerate their work.

“dbt Labs is pleased to name Alation a ‘Metrics Ready’ partner that empowers data analysts and engineers to better understand and trust data,” said Nikhil Kothari, Head of Technology Partnerships at dbt Labs. “Alation is a valued data catalog and data intelligence partner in the modern data stack that helps everyone in an organization find, understand, and trust data. With dbt transformation and metrics data in Alation, information previously contained in the dbt is now accessible to all data consumers.”

Conclusion

dbt’s new features are a key element to increasing data intelligence across the modern data stack. Making metadata about transformations visible to everyone in Alation will be a powerful tool for any organization looking to speed their time to decision with data while ensuring that everyone is working together to safeguard data quality and trust.

This is just the first step in a growing partnership. Stay tuned for more updates about joint Alation-dbt capabilities. We’re looking forward to great things to come!

Curious to learn more? Read the press release on this partnership.

    Contents
  • Data Transformation in the Modern Data Stack
  • Benefits of the Alation and dbt Partnership
  • Data Intelligence for the Modern Data Stack
  • Conclusion
Tagged with