Dynamic Variables in Data Quality Monitors

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Dynamic variables allow you to define parameterized data quality monitors that you can reuse with different inputs at runtime. Instead of hardcoding filter values or thresholds directly into a monitor, you define named placeholders that are supplied each time the monitor runs.

For example, instead of a hardcoded date filter like "filter": "ts_created >= '2024-01-01'", you can use a dynamic variable: "filter": "ts_created >= '${MIN_TS}'".

Dynamic variables are an optional feature. If you do not define any variables, monitors behave as they normally do.

Using dynamic variables provides several benefits:

  • Reuse monitors: Run a single monitor for different time windows, regions, or environments without creating duplicates.

  • Integrate with orchestration tools: Use tools like Airflow or dbt to run monitors with different runtime values, enabling event-driven data quality.

  • Automate quality checks: Integrate parameterized monitors into CI/CD pipelines and other data pipeline workflows.

Refer to the following use cases to use dynamic variables:

Use Case

Example variables

Description

Date range filtering

MIN_TS, MAX_TS

Run the same monitor against different time windows, such as daily or weekly, without duplicating the monitor configuration.

Environment targeting

ENV, SCHEMA_NAME

Reuse a monitor definition across development, staging, and production environments.

Threshold parameterization

NULL_THRESHOLD, MAX_DUPLICATES

Adjust quality thresholds at runtime based on the dataset’s criticality or other context.

Orchestration integration

BATCH_ID, LOAD_DATE

Pass pipeline-specific context from orchestration tools when running checks after a data load.

Define a Dynamic Variable

You define variables at the monitor level when you create or edit a monitor.

  1. Navigate to step Configure Checks when you Add a monitor.

  2. In the upper-right corner of the Select and Configure Checks section, click Dynamic Variables.

  3. In the modal that appears, click Add Variable.

  4. Configure the variable’s fields:

    • Variable Name: Enter the identifier for the variable. You reference this name in check definitions using the ${VARIABLE_NAME} syntax. The name is case-sensitive.

    • Data Type: Select the data type for the variable: String, Number, Date, or Boolean. The system uses this type to validate input values at runtime. If a provided value does not match the declared type, the monitor run fails before it executes.

    • Default Value: (Optional) Enter a fallback value to use if no value is provided when the monitor runs. If a variable has no runtime value and no default value, the monitor run fails.

    • Description: (Optional) Enter a human-readable explanation of the variable, such as its purpose or where it is used.

  5. Click Save.

Use a Variable in a Check

After you define a variable at the monitor level, you can reference it inside check definitions using the ${VARIABLE_NAME} syntax.

You can use variables in the following parts of a check configuration:

  • Filters

  • Threshold values

  • Configuration fields, such as missing or invalid values

  • Custom SQL checks

For example, you can use two variables to define a filter for a specific time range: "filter": "ts_created >= '${MIN_TS}' AND ts_created < '${MAX_TS}'"

Trigger a Monitor with Runtime Values

When you run a monitor, you supply values for each defined variable. You can trigger a monitor through the UI or with an API call.

Before the monitor executes, the system validates the provided values against each variable’s declared data type. If validation passes, the system substitutes the values into the check definitions and the monitor runs.

The following example shows an API payload that provides runtime values for the MIN_TS and MAX_TS variables.

{
   "MIN_TS": "2024-01-01T00:00:00Z",
   "MAX_TS": "2024-01-02T00:00:00Z"
}