Troubleshooting

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Core Connector Core connectors are included with all Alation platform tiers (subject to each tier’s connector limits) and are fully supported by Alation.

Issues with Metadata Extraction

Problem: MDE fails with error “iterating schema children”

The issue may occur with older connector versions that use the legacy Spark driver. This driver sometimes cannot keep up the connection long enough for MDE to complete.

Solution

This issue was resolved starting with connector version 2.0.2.6259, where the driver was changed to the Databricks JDBC driver. See more in Build the JDBC URI.

Problem: Fetch Schemas fails with error “unable to instantiate HiveMetastoreClient”

The issue may occur when Hive metastore is not instantiated. Hive Metastore is the default Databricks connection catalog.

Solution

Include the ConnCatalog and ConnSchema parameters into the connection JDBC URI:

  • ConnCatalog - Specifies the catalog which contains the metadata schema.

  • ConnSchema - Specifies the schema inside the catalog in which metadata is stored.

Example:

ConnCatalog=en_dlake_cat;ConnSchema=data_governance;

MDE Pre-check Fails with Warning “User is not the owner of external connection”

Problem

You are configuring or running metadata extraction (MDE) for a Databricks Unity Catalog data source. During the MDE process, Alation runs pre-checks to validate the service account permissions.

On the data source Settings > Metadata Extraction > Job History tab, you see the following warning message in the MDE job run report:

User is not the owner of external connection

The MDE job may complete with warnings, but external connection metadata may not be extracted correctly. This prevents you from seeing complete metadata for external data sources connected to your Databricks instance.

Cause

This warning occurs when your Databricks instance has external connections configured, but the service account used by Alation does not have ownership privileges for those connections.

Databricks Unity Catalog requires the OWNERSHIP privilege on external connections to enable complete metadata extraction. Without this privilege, Alation can connect to the data source but cannot extract metadata about external connections.

This is a Databricks Unity Catalog permission requirement, not an Alation limitation.

Solution

To resolve this warning and enable complete metadata extraction:

  1. Identify the service account configured in Alation:

    1. Navigate to your Databricks Unity Catalog data source in Alation.

    2. Go to the General Settings tab on your data source Settings page.

    3. Copy the username in the Service Account or Username field.

  2. Grant OWNERSHIP privilege in Databricks:

    1. Log into your Databricks workspace as an administrator.

    2. Navigate to the Data section and locate the external connections.

    3. For each external connection, grant OWNERSHIP privilege to the Alation service account by running:

      GRANT OWNERSHIP ON CONNECTION <connection_name> TO `<service_account>`;

      Replace <connection_name> with your external connection name and <service_account> with the username from step 1.

  3. Re-run metadata extraction in Alation:

    1. Return to Settings > Metadata Extraction in your Alation data source.

    2. Click Run Extraction Now.

    3. After the job completes, verify that the warning no longer appears in the Job History.

The metadata for external connections should now be extracted successfully.

Issues with Lineage Extraction

Problem: After upgrading the connector to version 1.0.3, MDE fails at ingestion

Connector logs will show the error “system.access.table_lineage table does not exist”.

Solution

Direct lineage extraction from system tables is a beta feature. The lineage system tables are not enabled by default and need to be enabled by your Databricks admin. See more in Lineage Extraction.

Problem: No lineage is generated after MDE

Solution

Verify that lineage data is available in the system lineage tables in Databricks.

Issues with QLI

Problem: After running QLI, some queries don’t appear in the History section of the Queries tab

Explanation

Some SQL statements are currently not supported by the Databricks SQL parser that is used to parse SQL statements during QLI, for example:

  • Partially qualified names in SQL statements are not parsed. Object names in query history must be fully qualified to be ingested.

  • The USE statements are not parsed.

  • Queries containing CTEs are not parsed.

  • Queries containing comments (COMMENT, COMMENT ON) are not parsed

This is a limitation with the current GSP and a known issue.

Problem: After QLI, parts of a query appear as separate queries in the catalog

Explanation

To support proper query parsing, accurate execution times count, and lineage resolution, batched queries are split into individual query statements before ingestion.

Problem: Queries that appeared in Preview don’t appear in the catalog after QLI

Explanation

The Preview is not directly tied to queries that will appear in the catalog. It returns 100 statements which may be a subset of all imported queries. For queries to appear in the catalog after QLI:

  • Queries must contain fully qualified object names with three-level namespace (catalog.schema.table).

  • The objects referenced in queries must have been cataloged with MDE.

  • Queries must not duplicate the queries that were ingested previously. Duplicated queries are not resolved.