Spark Package Management in SQL Server 2019 Big Data Clusters

This guide covers installing packages and submitting jobs to a SQL Server 2019 Big Data Cluster using Spark.

Built-In Tools
Install Packages from a Maven Repository onto the Spark Cluster at Runtime
Import .jar from HDFS for use at runtime
Import .jar at runtime through Azure Data Studio notebook cell configuration
Install Python Packages at Runtime for use with PySpark
Submit local .jar or python file

For more information on package managament, refer to Spark library management

Built-in Tools

Spark and Hadoop base packages
Python 3.8 with PySpark and Pandas, Sklearn, Numpy, and other data processing libraries.
R 3.5 with Spark.R, sparklyr and MRO packages

Install Packages from a Maven Repository onto the Spark Cluster at Runtime

Maven packages can be installed onto your Spark cluster using notebook cell configuration at the start of your spark session. Before starting a spark session in Azure Data Studio, run the following code:

%%configure -f \
{"conf": {"spark.jars.packages": "com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.1"}}

Import .jar from HDFS for use at runtime

Import jar at runtime through Azure Data Studio notebook cell configuration.

%%configure -f
{"conf": {"spark.jars": "/jar/mycodeJar.jar"}}

Import .jar at runtime through Azure Data Studio notebook cell configuration

%%configure -f
{"conf": {"spark.jars": "/jar/mycodeJar.jar"}}

Install Python Packages at Runtime for use with PySpark

This capability changed significantly after SQL Server Big Data Clusters CU10.

For more information on this scenario, refer to Spark library management

Submit local .jar or python file

One of the key scenarios for big data clusters is the ability to submit Spark jobs for SQL Server. The Spark job submission feature allows you to submit a local Jar or Py files with references to SQL Server 2019 big data cluster. It also enables you to execute a Jar or Py files, which are already located in the HDFS file system.

6.0 KiB Raw Blame History