Always On Orchestration with Dynamic Pipeline Runtime Variables

Always On Orchestration with Dynamic Pipeline Runtime Variables

Keep data flowing with pipelines that run the moment upstream processes finish—powered by runtime variables for seamless handoffs and dynamic logic.

Mage’s Always On triggers and runtime variable system make it easy to build responsive, intelligent workflows that react in real time to upstream events—and share context along the way.

Always On Triggers

Enable Always On mode to automatically trigger a pipeline as soon as the previous run finishes. No need to schedule or manually coordinate—pipelines flow continuously, ensuring rapid feedback loops and low-latency processing.

🔁 Trigger Downstream Pipelines with Context

Use orchestration blocks or API calls to trigger other pipelines and pass runtime variables—such as dates, environment flags, model versions, or partition keys. This ensures each downstream pipeline runs with the exact inputs it needs, dynamically and securely.

from mage_ai.orchestration.triggers.api import trigger_pipeline


@custom
def run(*args, **kwargs):
    trigger_pipeline(
        'downstream_pipeline',
        variables={'execution_date': kwargs['execution_date'], 'env': 'prod'},
        check_status=True
    )

📦 Runtime Variables = Shared Pipeline Context

Mage gives you access to global runtime variables that are available in every block of your pipeline—similar to Airflow’s context. Use them to:

  • Share metadata across blocks

  • Control conditional logic dynamically

  • Pass information between pipelines

  • Simplify parameter management


🔒 Block-Level Isolation When You Need It

If you need encapsulated logic, use block-level variables—scoped to a single block and invisible to others. Perfect for internal calculations, one-off configs, or reusable components across pipelines.

With Mage, orchestration is not just reactive—it’s context-aware, stateful, and completely programmable. Build pipelines that know when to run and how to carry forward what matters.

Your AI data engineer

Power data, streamline workflows, and scale effortlessly.

Power data, streamline workflows, and scale effortlessly.