Build Once, Reuse Everywhere: Global Data Products for Intelligent Pipeline Orchestration
TLDR
Create, share, and reuse data across pipelines without redundant processing.
Mage Pro’s Global Data Products feature allows you to generate a data output once and reuse it across any pipeline in your project. Whether the data is an in-memory DataFrame, a serialized JSON object, or a database table, it can be registered globally, referenced anywhere, and intelligently refreshed only when needed.
Instead of re-running expensive pipelines or duplicating data across workflows, Global Data Products act as shared, lazy-evaluated building blocks—ensuring consistent outputs, reducing compute costs, and minimizing redundant processing.
Global Data Products are registered once, versioned automatically, and lazily triggered only when their data is outdated or requested by another pipeline, ensuring maximum efficiency at scale.
How it works
Promote any pipeline output into a Global Data Product by assigning it a unique ID (UUID) through the global registry.
Use any Global Data Product as a block inside any pipeline. Other blocks can depend on it just like any other in-pipeline block.
A Global Data Product only runs when its data is requested and the data is outdated—preventing unnecessary recomputation.
Configure how long a data product stays fresh (e.g., 12 hours, 1 week) and define when it should become eligible for reprocessing based on specific times or dates.
Customize how much historical data to retrieve by setting partition windows when consuming a Global Data Product.
Pipelines referencing a Global Data Product can override freshness thresholds, output block selections, and partition ranges for precise control.
If multiple pipelines request a Global Data Product at the same time, Mage Pro ensures it runs once and shares the output, avoiding duplicate executions.
Why it matters
Global Data Products transform your pipeline architecture from siloed workflows into a modular, reusable system. They allow teams to:
- Build once, reuse everywhere across pipelines
- Reduce compute costs by preventing redundant executions
- Centralize key data assets with guaranteed freshness controls
- Accelerate downstream workflows with precomputed, ready-to-use datasets
- Simplify dependency management across complex multi-pipeline projects
With Mage Pro, data pipelines aren’t just about transformation—they become living ecosystems of reusable, intelligent data products.
