Pipeline Design

Introduction

Fyrefuse is primarily a Data Engineering tool designed to simplify the development, maintenance and deployment of an Apache Spark Application via an intuitive and pixel-perfect user interface.

It provides a comprehensive suite of tools—referred to as jobs—that support the creation of complex Data Pipelines.

At runtime, each job within the pipeline corresponds logically to a Spark DataFrame, ensuring seamless execution and transformation of data.

Moreover, the adoption of the Dataframes enables Fyrefuse to use SQL as its default query language.

Supported Jobs

Here the detailed list of the Fyrefuse supported tools:

1. Source job
2. Target job
3. SQL job
4. Script job
5. Variable job
6. Validation job
7. Anonymization job
8. UDF job
9. Custom job

They are grouped with the help of a visual divisor ( | ), this is to help users differentiate them according to their logic.

The first group (1, 2) includes the jobs for connecting the pipeline to sources and targets.

The second group (3, 4, 5) includes the low-code jobs.

The third group (6, 7) includes the no-code jobs.

The fourth group (8, 9) includes the custom-code jobs.