SDP 24.11 is not the latest You're viewing SDP 24.11, not the current release (SDP 26.3).

View SparkApplication in SDP 26.3 View SDP 26.3 →

Custom resource

SparkApplication

Auto-generated derived type for SparkApplicationSpec via `CustomResource`

View SDP 24.11 release Download CRD (YAML)

API group: spark.stackable.tech
API versions in 24.11: v1alpha1
Scope: Namespaced
Operator: Apache Spark
Roles: none

Schema

SparkApplication.spark.stackable.tech .spec fields for SDP 24.11.

Schema

mainApplicationFile # string required * The actual application file that will be called by spark-submit.

The actual application file that will be called by spark-submit.
mode # string · enum required * Mode: cluster or client. Currently only cluster is supported.

Mode: cluster or client. Currently only cluster is supported.

allowed: clusterclient
sparkImage # object required *
Specify which image to use, the easiest way is to only configure the productVersion. You can also configure a custom image registry to pull from, as well as completely custom images. Consult the Product image selection documentation for details.

Specify which image to use, the easiest way is to only configure the productVersion. You can also configure a custom image registry to pull from, as well as completely custom images. Consult the Product image selection documentation for details.
args # array = [] Arguments passed directly to the job artifact.

Arguments passed directly to the job artifact.
deps # object = {"packages":[],"repositories":[],"requirements":[],"excludePackages":[]}
Job dependencies: a list of python packages that will be installed via pip, a list of packages or repositories that is passed directly to spark-submit, or a list of excluded packages (also passed directly to spark-submit).

Job dependencies: a list of python packages that will be installed via pip, a list of packages or repositories that is passed directly to spark-submit, or a list of excluded packages (also passed directly to spark-submit).
driver # object
The driver role specifies the configuration that, together with the driver pod template, is used by Spark to create driver pods.

The driver role specifies the configuration that, together with the driver pod template, is used by Spark to create driver pods.
env # array = []
A list of environment variables that will be set in the job pod and the driver and executor pod templates.

A list of environment variables that will be set in the job pod and the driver and executor pod templates.
executor # object
The executor role specifies the configuration that, together with the driver pod template, is used by Spark to create the executor pods. This is RoleGroup instead of plain CommonConfiguration because it needs to allows for the number of replicas. to be specified.

The executor role specifies the configuration that, together with the driver pod template, is used by Spark to create the executor pods. This is RoleGroup instead of plain CommonConfiguration because it needs to allows for the number of replicas. to be specified.
image # string
User-supplied image containing spark-job dependencies that will be copied to the specified volume mount. See the examples.

User-supplied image containing spark-job dependencies that will be copied to the specified volume mount. See the examples.
job # object
The job builds a spark-submit command, complete with arguments and referenced dependencies such as templates, and passes it on to Spark. The reason this property uses its own type (SubmitConfigFragment) is because logging is not supported for spark-submit processes.

The job builds a spark-submit command, complete with arguments and referenced dependencies such as templates, and passes it on to Spark. The reason this property uses its own type (SubmitConfigFragment) is because logging is not supported for spark-submit processes.
logFileDirectory # object The log file directory definition used by the Spark history server.

The log file directory definition used by the Spark history server.
mainClass # string The main class - i.e. entry point - for JVM artifacts.

The main class - i.e. entry point - for JVM artifacts.
s3connection # object
Configure an S3 connection that the SparkApplication has access to. Read more in the Spark S3 usage guide.

Configure an S3 connection that the SparkApplication has access to. Read more in the Spark S3 usage guide.
sparkConf # object = {}
A map of key/value strings that will be passed directly to spark-submit.

A map of key/value strings that will be passed directly to spark-submit.
vectorAggregatorConfigMapName # string
Name of the Vector aggregator discovery ConfigMap. It must contain the key ADDRESS with the address of the Vector aggregator. Follow the logging tutorial to learn how to configure log aggregation with Vector.

Name of the Vector aggregator discovery ConfigMap. It must contain the key ADDRESS with the address of the Vector aggregator. Follow the logging tutorial to learn how to configure log aggregation with Vector.
volumes # array = []
A list of volumes that can be made available to the job, driver or executors via their volume mounts.

A list of volumes that can be made available to the job, driver or executors via their volume mounts.