Pipeline execution

Effective management and execution of pipelines require flexibility to accommodate different use cases, operational needs, and business requirements. Some workloads demand on-demand execution, while others need to be scheduled, event-driven, or externally triggered to integrate seamlessly into broader workflows.

To provide maximum adaptability, Fyrefuse supports four distinct execution methods, ensuring pipelines can be triggered in the most suitable and efficient way:

1. Manual Execution – On-demand runs initiated by users.
2. Cron-Based Execution – Predefined, time-based scheduling.
3. External API Trigger – Remote execution via API calls.
4. Event-Based Execution – Automated triggering based on system events.

Each of these methods offers unique advantages, allowing users to balance control, automation, and responsiveness in managing the workloads. Below is a detailed explanation of each approach, including its ideal use cases and key benefits.

Manual execution

Manual execution provides users with the ability to trigger a pipeline whenever needed, without relying on predefined schedules or external triggers.

This method is particularly useful for:

• Ad-hoc data processing
• Debugging and testing before deploying automated executions
• Running exploratory or one-time jobs

This method is especially valuable in development and troubleshooting scenarios, where users may need to modify parameters, validate logic, or rerun jobs multiple times before finalizing a production pipeline.

For manually deploying a pipeline simply click the three vertical dots beside the pipeline name and select “Run”.

Time-driven execution

The time-driven (cron-based) scheduling method, pipelines can be executed at fixed intervals (e.g., daily, hourly, weekly) based on a predefined schedule.

This is widely used for:

• Periodic ETL/ELT pipelines
• Batch data processing
• Scheduled reporting and analytics

By defining execution times in advance, cron-based scheduling ensures predictability, making it ideal for workloads that do not require real-time responsiveness but must be executed consistently.

For scheduling a pipeline execution simply click the three vertical dots beside the pipeline name and select “Schedule”.

External API trigger

In many modern architectures, processes need to be triggered dynamically from external systems. The External API Trigger execution mode enables0 third-party applications, microservices, or other workflows to remotely initiate pipelines via API calls.

This approach is beneficial for:

• Integrating pipelines within external applications
• Triggering jobs dynamically based on external events

This method allows Fyrefuse to be tightly integrated into larger enterprise ecosystems, making it possible to dynamically trigger pipelines based on real-time business needs.

Event-driven conditions

In highly dynamic environments, real-time or near-real-time processing is essential. Event-based execution allows Spark applications to be automatically triggered in response to specific system events happening within the platform.

This method is commonly used for:

• Triggering action based on user interactions
• Automating workflows based on system state changes

Unlike scheduled executions, event-based processing eliminates unnecessary runs and ensures jobs execute only when relevant conditions are met, leading to better resource efficiency.

Conclusion

By supporting multiple execution methods, the platform provides a versatile and scalable solution for managing pipelines. Whether applications need manual control, scheduled automation, remote triggering, or event-driven execution, this flexibility ensures that Fyrefuse workloads align with diverse business needs and operational strategies.