Reusable Sensor Blocks for Smart Pipeline Orchestration

Reusable Sensor Blocks for Smart Pipeline Orchestration

Trigger downstream tasks only when upstream conditions are met—use sensors to orchestrate dependencies across pipelines with precision.

Sensor blocks in Mage enable intelligent, event-driven orchestration across pipelines. These reusable blocks continuously evaluate a condition—such as waiting for another pipeline or block to complete—before allowing downstream tasks to proceed.

Sensors are ideal for cross-pipeline dependencies, external event coordination, and ensuring data readiness before triggering computation.

⚙️ How it works

  • Add a @sensor block to your pipeline that checks the run status of another pipeline or block.

  • Define custom logic using Python to determine when the condition is satisfied.

  • Downstream blocks won’t execute until the sensor evaluates to True.

  • Sensors poll automatically every 60 seconds and stop when the condition is met or a timeout is reached.

🧩 Fully Reusable & Composable

  • Sensors can accept input from upstream blocks like any other block type.

  • Use them across pipelines to standardize readiness checks or sync coordination logic.

  • Avoid unnecessary runs and improve the reliability of multi-stage workflows.

🔁 Example use cases

  • Wait for a daily ETL pipeline to finish before triggering enrichment logic.

  • Sync with an external data ingestion job before running transformations.

  • Coordinate multi-team workflows where dependencies span projects.

With Mage’s sensor blocks, you gain fine-grained control over execution flow—without sacrificing modularity or reusability.

Your AI data engineer

Power data, streamline workflows, and scale effortlessly.

Power data, streamline workflows, and scale effortlessly.