On this page
This page is a complete listing of configuration parameters for the Fyrefuse's docker compose file.
The same options are included in the custom installation but for an easier reference we also provide them here in a more structured format.
This Docker Compose file defines a multi-container setup for Fyrefuse, a distributed application consisting of a frontend, multiple backend instances, a database, and a worker service.
Below is a breakdown of the components.
The reusable configuration block (&default-backend) used by every Fyrefuse's backend instances includes the following elements.
| Name | Description | Default |
|---|---|---|
image |
The Backend image registry path | |
restart |
The Frontend version to be deployed | always |
environment |
Common backends' instances env variables | |
volumes |
Volumes to be mounted on every backend instance | spark_shared |
networks |
Accessible networks from every backend instance | fyrefuse-bridge |
The actual list of deployed services along with their specific parameters.
| Service | Optional | Role | Ports | Depends On | Notes |
|---|---|---|---|---|---|
| db | yes | PostgreSQL database | 5432:5432 | None | Stores application data |
| frontend | no | Web UI | 8001:80 | backend-1, backend-2, backend-3, backend-4 | Serves the frontend app |
| backend-1 | no | Main backend | 8002:6000 | None | Runs initial migrations and other preliminary tasks |
| backend-2 | yes | Additional backend | 8003:6000 | backend-1 | Handles API requests |
| backend-3 | yes | Additional backend | 8004:6000 | backend-1 | Handles API requests |
| backend-4 | yes | Additional backend | 8005:6000 | backend-1 | Handles API requests |
| worker | no | Background worker | None | backend-1, backend-2, backend-3, backend-4 | Runs background tasks |
Each container has its own set of parameters. Below is a breakdown of the parameters for each container.
| Variable Name | Description | Default |
|---|---|---|
| BE_VERSION | The Backend version to be deployed | latest |
| FE_VERSION | The Frontend version to be deployed | latest |
| DATABASE_USER | Specify the Fyrefuse database username | fyrefuse_user |
| DATABASE_PASSWORD | Specify the Fyrefuse database password | |
| DATABASE_HOST | Specify the Fyrefuse database hostname or ip | localhost |
| DATABASE_PORT | Specify the Fyrefuse database port | 5432 |
| DATABASE_NAME | Specify the Fyrefuse database name | fyrefuse_database |
| DATABASE_SCHEMA | Specify the Fyrefuse database schema | fyrefuse_schema |
| PIPELINE_LOGGER_LOGGER_URL | The api url to be used by the FEM to send pipelines’ logs | |
| PIPELINE_LOGGER_REPORT_URL | The api url to be used by the FEM to send pipelines’ reports | |
| GITLAB_PRIVATE_ACCESS_TOKEN | Default access token for retrieving the FEM from gitlab (in case of spark standalone deployment engine) | |
| FEM_BUILD_NAME | FEM version that will be used for running the Fyrefuse’s pipelines | |
| FEM_BUILD_LINK | FEM build link | |
| METASTORE_DB_NAME | Specify the Metastore database name | metastore |
| METASTORE_DB_SCHEMA | Specify the Metastore database schema | public |
| METASTORE_URI | Specify the Metastore connection uri | thrift://<metastore_hostname> |
| METASTORE_WAREHOUSE_DIR | Specify the Metastore warehouse dir | s3a://datalake/warehouse/ |
| METASTORE_DB_CONNECTION_STRING | Specify the Metastore database connection string | jdbc:postgresql://localhost:5432/metastore |
| METASTORE_DB_DRIVER | Specify the Metastore database driver | org.postgresql.Driver |
| METASTORE_DB_USERNAME | Specify the Metastore database username | fyrefuse_user |
| METASTORE_DB_PASSWORD | Specify the Metastore database password | |
| DATALAYER_HTTP_MODE | Specify the S3 object storage http mode | http |
| DATALAYER_ENDPOINT | Specify the S3 object storage endpoint | |
| DATALAYER_ACCESS_KEY | Specify the S3 object storage access key | |
| DATALAYER_SECRET_KEY | Specify the S3 object storage secret key |
| Variable Name | Description | Default |
|---|---|---|
| FE_VERSION | The Frontend version to be deployed | latest |
| API_URL | The base api url to be used by Fyrefuse frontend | |
| API_LOGGER | The endpoint to be used by Fyrefuse frontend to enable notifications | |
| WS_URL | The endpoint to be used by Fyrefuse frontend to enable Websockets |
| Variable Name | Description | Default |
|---|---|---|
| POSTGRES_USER | Specify the Fyrefuse database username | fyrefuse_user |
| POSTGRES_PASSWORD | Specify the Fyrefuse database password | |
| FYREFUSE_DB | Specify the Fyrefuse database name | fyrefuse_database |
| FYREFUSE_SCHEMA | Specify the Fyrefuse database schema | fyrefuse_schema |
The docker compose also contains volumes and networks items.
Mount the spark cluster shared NFS directory to share pipeline dependencies.
volumes:
spark-shared:
driver_opts:
type: "nfs"
o: "addr=spark-cluster.local,nolock,soft,nfsvers=4"
device: ":/mnt/volume-workers/nfs_volume"
Create a dedicate containers network
networks:
fyrefuse-bridge:
external: false
name: fyrefuse-bridge