Customer Case Study:
 
Spark New Zealand

The second challenge, common to any B2C company today, was the massive increase in customer and other data that would only grow with the advances in 5G and internet of things (IOT) technology. Leaders saw that some of their on-premises systems would soon need to be upgraded or replaced, and that scaling their Hadoop cluster would be time consuming and costly. For these reasons, Spark decided to migrate their data to the cloud.

For Langham, an ideal solution would help Spark bring together the required data in the cloud and build advanced analytical models that derive customer insight. He also believed that data access should not be limited to his data science team. He wanted to democratize that data, making it available to business units across the organization so they, too, could derive insights to support the company’s strategic goals.

Building a Complete Cloud Data Solution

Spark chose the Alation Data Catalog to promote faster data search and discovery. This empowers their data engineers and analysts to derive insight from customer and other company data, no matter where it’s located. Alation brings the metadata and business understanding from all of Spark’s different systems together.

Spark then evaluated various cloud data platforms and chose Snowflake because it offered genuine separation of compute and storage, and the performance and scalability they needed. All the data now sits on a common technology platform. This stands in stark contrast to the siloed data landscape of the past, where a separate data warehouse, a separate data lake, and an analytics appliance accommodated different uses—but slowed productivity.

Finally, Spark wanted to run Snowflake on a public cloud. They chose the Microsoft Azure cloud computing service, in part for its compatibility with a number of Microsoft solutions already in place at Spark.

Harnessing Data for Insight

As Spark began their Snowflake migration, they briefly evaluated other compatible data catalogs but chose to stick with Alation. “When we looked at the features that were available, the compatibility with Snowflake and, obviously, the fact that we were existing Alation users and were already getting a lot of value out of it, it was a fairly obvious choice for us to continue with Alation,” says Langham.

Today, approximately 110 people at Spark regularly use Alation to find and interpret data stored in Snowflake and the remaining data in its on-premises systems. To improve the company’s agility and foster collaboration, Langham has embedded his data engineers and architects into teams from other parts of the business, spurring data democratization. This includes teaming analysts and data scientists with the company’s customer “tribe” to improve marketing campaigns, customer engagement, and products. The analysts’ work ranges from reporting to building new analytical and machine learning models.

Spark leverages Snowflake for a range of use cases. Today, Spark uses Snowflake as its data lake and data warehouse for their structured and semi-structured data. On top of that, Spark is boosting analyst productivity by using Snowflake as a “feature store” to save analytic features that can be reused across models. In the past, each time a data scientist created an analytical model, they would start from scratch. Now they can create and keep features in a common feature store in Snowflake and reuse them across multiple models. Alation catalogs the available features, making them visible and accessible to data scientists and analysts who are creating new models.

Spark is now in the process of moving data from their on-premises systems to Snowflake. Instead of a “lift-and-shift” migration of all data, they are using the popularity and lineage features in Alation to prioritize which data to move. This means they only move valuable data, that which is actively used or connected to other critical data.

Speeding Cloud Adoption and Analytics

Both Spark customers and employees have benefited from their innovations in data. By implementing Alation + Snowflake on the Microsoft Azure cloud, Spark boosted their ability to use customer insight to deliver the right products at the right time, a key pillar of its business strategy. Employees today enjoy speedier access to data. Using the Alation Data Catalog has reduced the time that data analysts and scientists need to find data by at least 50%, freeing them up to work on higher value activities. The catalog also provides far greater consistency around data definitions and usage.

The second challenge, common to any B2C company today, was the massive increase in customer and other data that would only grow with the advances in 5G and internet of things (IOT) technology. Leaders saw that some of their on-premises systems would soon need to be upgraded or replaced, and that scaling their Hadoop cluster would be time consuming and costly. For these reasons, Spark decided to migrate their data to the cloud.

For Langham, an ideal solution would help Spark bring together the required data in the cloud and build advanced analytical models that derive customer insight. He also believed that data access should not be limited to his data science team. He wanted to democratize that data, making it available to business units across the organization so they, too, could derive insights to support the company’s strategic goals.

Building a Complete Cloud Data Solution

Spark chose the Alation Data Catalog to promote faster data search and discovery. This empowers their data engineers and analysts to derive insight from customer and other company data, no matter where it’s located. Alation brings the metadata and business understanding from all of Spark’s different systems together.

Spark then evaluated various cloud data platforms and chose Snowflake because it offered genuine separation of compute and storage, and the performance and scalability they needed. All the data now sits on a common technology platform. This stands in stark contrast to the siloed data landscape of the past, where a separate data warehouse, a separate data lake, and an analytics appliance accommodated different uses—but slowed productivity.

Finally, Spark wanted to run Snowflake on a public cloud. They chose the Microsoft Azure cloud computing service, in part for its compatibility with a number of Microsoft solutions already in place at Spark.

Harnessing Data for Insight

As Spark began their Snowflake migration, they briefly evaluated other compatible data catalogs but chose to stick with Alation. “When we looked at the features that were available, the compatibility with Snowflake and, obviously, the fact that we were existing Alation users and were already getting a lot of value out of it, it was a fairly obvious choice for us to continue with Alation,” says Langham.

Today, approximately 110 people at Spark regularly use Alation to find and interpret data stored in Snowflake and the remaining data in its on-premises systems. To improve the company’s agility and foster collaboration, Langham has embedded his data engineers and architects into teams from other parts of the business, spurring data democratization. This includes teaming analysts and data scientists with the company’s customer “tribe” to improve marketing campaigns, customer engagement, and products. The analysts’ work ranges from reporting to building new analytical and machine learning models.

Spark leverages Snowflake for a range of use cases. Today, Spark uses Snowflake as its data lake and data warehouse for their structured and semi-structured data. On top of that, Spark is boosting analyst productivity by using Snowflake as a “feature store” to save analytic features that can be reused across models. In the past, each time a data scientist created an analytical model, they would start from scratch. Now they can create and keep features in a common feature store in Snowflake and reuse them across multiple models. Alation catalogs the available features, making them visible and accessible to data scientists and analysts who are creating new models.

Spark is now in the process of moving data from their on-premises systems to Snowflake. Instead of a “lift-and-shift” migration of all data, they are using the popularity and lineage features in Alation to prioritize which data to move. This means they only move valuable data, that which is actively used or connected to other critical data.

Speeding Cloud Adoption and Analytics

Both Spark customers and employees have benefited from their innovations in data. By implementing Alation + Snowflake on the Microsoft Azure cloud, Spark boosted their ability to use customer insight to deliver the right products at the right time, a key pillar of its business strategy. Employees today enjoy speedier access to data. Using the Alation Data Catalog has reduced the time that data analysts and scientists need to find data by at least 50%, freeing them up to work on higher value activities. The catalog also provides far greater consistency around data definitions and usage.

The increased productivity is not limited to experienced data scientists and analysts. A data catalog also speeds the onboarding of new personnel. It previously took up to two months for engineers to gain a comfort level with all the different systems and data stores. With Alation, new hires can become self-sufficient and begin using the data within a couple of weeks, or 75% faster.

The Alation catalog not only makes it easier to find the right data, but also boosts the use of the data on Snowflake.

“Alation has absolutely accelerated the adoption of Snowflake amongst our users,” Langham says. “It has enabled them to get up and running on Snowflake a lot quicker than they otherwise would have been able to do.”

As more data is migrated, the metadata and definitions are immediately updated in Alation, so users always know where the data is and how to use it. Alation’s intelligent SQL editor, Compose, makes it easy for even beginner users to query the catalog to find the right data. Compose also allows users to publish queries that can be reused by others, further facilitating the use of the data catalog for business users.

Langham notes that curating data in Snowflake and providing an intuitive data catalog with Alation make for a simple user experience, “It’s exactly the same as the simple and intuitive experience we’re trying to provide to our customers, but for our employees,” he shares.

Expansion on the Horizon

Alation + Snowflake on Azure has become a critical foundation of Spark’s larger business strategy. “I think it’s a really exciting time in terms of what we’re doing with advanced analytics and AI, machine learning and data democratization,” Langham shares. “It is part of a broader success story around the adoption of data and analytics, and Alation’s certainly been a great contributor to that.”

Early success has prompted plans for expansion. Spark plans to expand the number of people using Alation and Snowflake to up to 500 in the future as it moves toward greater data democratization by a wider set of data consumers. This expansion will likely include business users who consume data through Power BI dashboards or reports.

To ensure successful data democratization, Spark is turning to Alation and Snowflake’s governance features. Spark already has standards for classifying and tagging data, including personal identifiable information (PII) and other sensitive data, and is creating appropriate usage policies. Some of the permissions are implemented directly in Snowflake, but Spark is also looking at increasing the use of Alation to ensure governance while allowing access to the data that business units need.

About Spark New Zealand

Spark New Zealand Limited, more commonly known as Spark, is New Zealand’s largest telecommunications and digital services company. Operating fixed-line telephone services and a mobile network, Spark is also an internet service provider (ISP) and a major information and communications technology (ICT) provider to New Zealand businesses. Previously known as Telecom New Zealand, the company was rebranded as Spark in 2014. In 2020, Spark had more than 2.5 million mobile and 709,000 broadband connections. Headquartered in Auckland, New Zealand, Spark employs approximately 5,200 people.

• Cloud service: Azure
• Cloud data warehouse: Snowflake
• On-premise: Db2
• Additional: Hadoop
• Analytics: Power BI
• Current Alation users: 150 seats, 110 active users
• Current Snowflake users: 150 seats, 110 active users