Business Results
Improves Time-to-Market
Product development is faster when people can find information quickly
Improves Data Literacy
People know how and where to find critical data
Speeds Data Quality Issue Resolution
Faster root-cause analysis with manual lineage feature addresses problems sooner
The Problem: Tapping into 50 Years of Siloed Data
Airlines Reporting Corporation (ARC) is a small but mighty powerhouse in the travel industry. ARC is a key intermediary between airlines and travel companies such as Expedia, settling transactions including refunds, exchanges, and cancellations. In recent years, they have strategically positioned themselves as a data and analytics company, selling business intelligence and other data products to airlines, travel management companies, and other third-parties. ARC develops their data products using aggregated data from their settlement transactions and from the International Air Transport Association (IATA).
As one might imagine, after 50 years in business, ARC had accumulated a massive amount of data. The data was stored in on-premises servers, largely siloed by the company’s seven business domains.
The company wanted to bring new products to market faster by making data more accessible across the business. According to Leonard Kwok, Senior Data Analyst at ARC, the company’s existing product development processes relied heavily on the tribal knowledge of domain experts. This required direct coordination between multiple people to find and tap into the right data to create new solutions. When employees left the company, they took their expertise with them.
In 2018, the company decided to embark on a digital transformation initiative, which included modernizing business processes and migrating all their data to an AWS S3 Data Lake and Snowflake Data Cloud. By moving to the cloud, ARC would make data more accessible across domains, reducing the bottlenecks created by relying on individual expertise during product development. To ensure that folks could find the data they needed, ARC sought to implement a data catalog solution for their migrated cloud data.
The Solution: Create a One-Stop-Shop for Data
After evaluating several data catalog options, ARC chose the Alation Data Catalog. In addition to having the most intuitive, refined, and mature user interface, several Alation features led to the choice, including:
Trust Flags to deprecate unused data, communicate which data should be used for what purpose, and increase trust in data across the organization
Compose to create a unified SQL querying experience
Data profiling to show the technical, business metadata, and enable users to see a sampling of the data without needing to access multiple tools
ARC uses the Snowflake Data Cloud on AWS S3, where both third-party and internal data is stored. The company is curating and cataloging their data in Alation as they migrate it to the cloud. The data cataloged in Alation is purely cloud-based; ARC did not connect Alation to their on-premises systems.
Alation makes the data curation, cataloging, and governance process simple, even for a small company like ARC. Their 300 or so employees often wear multiple hats, so having a large group dedicated to the data catalog wasn’t feasible, according to Kwok. “Curating and cataloging our Snowflake data with Alation has been really beneficial to us as a small company,” he says. “So much is automatic – the metadata extraction, curation, labeling, query log ingestion, and building out the lineage – it’s a big help.” A small group of data stewards – experts in their domain data – holds responsibility for data curation and cataloging in Alation.
To help users from the wider company better understand where the data is and how to use it, ARC is encouraging the adoption of Alation as the one-stop shop for data access and analytics. Compose, the SQL query editor in Alation, will be critical for connecting people to the data they need without requiring them to access it directly through Snowflake. “We’re creating and sharing queries through Compose rather than having to train everyone on Snowflake,” says Kwok. “That’s easier and faster than telling people they need to use multiple tools. They just need to become familiar with Alation.”
"So much is automatic – the metadata extraction, curation, labeling, query log ingestion, and building out the lineage – it’s a big help."
Senior Data Analyst, ARC
ARC can use Alation to launch and curate Tableau reports. Users can also leverage Alation to search, with context, for those reports. “We’re using the seamless Alation connection to Tableau to deliver internal reports and for interactive reporting to our customers,” says Kwok. Once again, a single tool, Alation, provides access to the data the company needs for analytics. ARC also plans to use Tableau for data quality scoring, and then bring that analysis into Alation.
ARC implemented the manual lineage function in Alation to trace the connections more easily from the S3 data to Snowflake, and ultimately to Tableau for reporting and analysis. “The movement from the data lake to Snowflake is usually in Python,” explains Kwok. “But sometimes it’s in SnoSQL, which would require a parser for Python to get the lineage.” The Alation lineage feature eliminates that step and allows ARC to further extend that lineage to their Tableau workbooks.
Kwok is working closely with the ARC security teams to use the lineage feature to ensure that the company does not expose PII data to customers or individuals who don’t have access. The masking feature in Snowflake, which carries over into Alation, can also minimize the risk of exposing sensitive information.
The Result: Faster Time to Market with Data Self Service
Alation helps data users at ARC quickly find and connect to the data they need. Gone are the days of “ten meetings with ten people to get the right data for product development,” according to Kwok. His goal: as data is moved to the Snowflake Data Cloud, users will naturally and automatically use Alation to access it. He acknowledges this will be a cultural shift for some in the organization, but the time savings will be worth it.
Self-service through Alation will ultimately speed time-to-market of new products. Project teams can use Alation to conduct their own research into the data in the S3 data lake and Snowflake instead of learning to use multiple tools or contacting multiple people and information architects to find the data they need. As users become more familiar with what data is available and how to find it, Kwok expects data literacy at ARC to improve.
ARC can capture individuals’ domain expertise in the catalog, eliminating the loss of knowledge to the whole company that results when people retire or leave the company. Alation’s query log ingestion and data lineage mean that if a product developer leaves the company, ARC doesn’t have to trace the code to figure out how something works.
In addition to reducing time-to-market, ARC expects to more quickly identify and resolve data quality or production issues arising from third party data. “Manual lineage gives us a quick and easy way to document relationships and trace where data came from,” says Kwok. “We can use manual lineage to perform root cause analysis. This means we can identify problems from our data suppliers in minutes instead of hours or days. That translates to faster product development and remediation for the business.”
"We can use manual lineage to perform root cause analysis. This means we can identify problems from our data suppliers in minutes instead of hours or days."
Senior Data Analyst, ARC
The Forecast: Clear Skies for the Future
ARC continues to curate, migrate, and catalog its data. They are also connecting Salesforce to their data cloud, further enhancing and improving data literacy at the company. Once the catalog has reached a higher level of maturity, ARC intends to connect specific business partners, and possibly customers, to the Alation catalog.
Despite a couple of turbulent years through the pandemic, the future looks bright for ARC. Their cloud migration will help the lean company better serve their customers and create new and innovative data products faster thanks to the best-in-class technology stack they’ve chosen with AWS S3, Snowflake, Tableau, and, of course, Alation.
About Airlines Reporting Corporation
For more than half a century, ARC has been a trusted provider of airline settlement solutions. They help airlines and travel agencies connect, grow, and thrive by delivering flexible distribution solutions, innovative technology and access to the world’s most comprehensive air transaction data. In 2019, ARC managed more than $97.4 billion in transactions between airlines and travel agencies, representing more than 302 million passenger trips. Their membership includes more than 200 airlines and 9,400 travel agencies, and they have 13,000 points of sale.
