Custom resource

AirflowCluster

An Airflow cluster stacklet. This resource is managed by the Stackable operator for Apache Airflow.

View SDP dev release Download CRD (YAML)

API group: airflow.stackable.tech
API versions in dev: v1alpha2v1alpha1
Scope: Namespaced
Operator: Apache Airflow
Roles: 5 (celeryExecutors, dagProcessors, schedulers, triggerers, webservers)

Schema

AirflowCluster.airflow.stackable.tech .spec fields for SDP dev.

Showing the v1alpha2 schema. v1alpha1 is also served in SDP dev.

Schema

clusterConfig # object required *
Configuration that applies to all roles and role groups. This includes settings for authentication, git sync, service exposition and volumes, among other things.

Configuration that applies to all roles and role groups. This includes settings for authentication, git sync, service exposition and volumes, among other things.
image # object required *
Specify which image to use, the easiest way is to only configure the productVersion. You can also configure a custom image registry to pull from, as well as completely custom images. Consult the Product image selection documentation for details.

Specify which image to use, the easiest way is to only configure the productVersion. You can also configure a custom image registry to pull from, as well as completely custom images. Consult the Product image selection documentation for details.
celeryExecutors # role object The celery executor. Deployed with an explicit number of replicas.

The celery executor. Deployed with an explicit number of replicas.
dagProcessors # role object
The dagProcessors role runs the DAG processor routine for DAG preparation.

The dagProcessors role runs the DAG processor routine for DAG preparation.
schedulers # role object
The schedulers is responsible for triggering jobs and persisting their metadata to the backend database. Jobs are scheduled on the workers/executors.

The schedulers is responsible for triggering jobs and persisting their metadata to the backend database. Jobs are scheduled on the workers/executors.
triggerers # role object
The triggerers role runs the triggerer process for use with deferrable DAG operators.

The triggerers role runs the triggerer process for use with deferrable DAG operators.
webservers # role object The webservers role provides the main UI for user interaction.

The webservers role provides the main UI for user interaction.
clusterOperation # object = {"stopped":false,"reconciliationPaused":false}
Cluster operations properties, allow stopping the product instance as well as pausing reconciliation.

Cluster operations properties, allow stopping the product instance as well as pausing reconciliation.
kubernetesExecutors # object With the Kubernetes executor, executor Pods are created on demand.

With the Kubernetes executor, executor Pods are created on demand.
objectOverrides # array = []
A list of generic Kubernetes objects, which are merged into the objects that the operator creates. List entries are arbitrary YAML objects, which need to be valid Kubernetes objects. Read the Object overrides documentation for more information.

A list of generic Kubernetes objects, which are merged into the objects that the operator creates. List entries are arbitrary YAML objects, which need to be valid Kubernetes objects. Read the Object overrides documentation for more information.