sql-server-samples/samples/features/sql-big-data-cluster/spark/config-install/installpackage_Spark.ipynb

Packaging in Spark

Use Case 1: I can have key packages in boxed

• All pacakges that come with spark and hadoop distribution
• Python3.5 and Python 2.7
• Pandas, Sklearn and several other supporting ml packages
• R and supporting pacakges as part of MRO
• sparklyr

Use Case 2: I can install pacakges from maven repo to my spark cluster

Maven central is a source of lot of packages. A lot of spark ecosystem pacakges are availble there. These pacakages can be installed to your spark cluster using notebook cell configuration at the start of your spark session.

In [3]:

%%configure -f
{"conf": {"spark.jars.packages": "com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.1"}}

Current session configs: {'conf': {'spark.jars.packages': 'com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.50'}, 'kind': 'spark'}

No active sessions.

In [5]:

import com.microsoft.azure.eventhubs._

import com.microsoft.azure.eventhubs._

Use Case 3: I have a local jar that i want to run in the spark cluster

As a user you may build your own customer pacakges that want to run as part of your spark jobs. These pacakges can be uploaded as HDFS and using a notebook configuration spark can consume these pacakges in a jar.

In [0]:

%%configure -f
    {"conf": {"spark.jars": "/jar/mycodeJar.jar"}}

In [0]:

import com.my.mycodeJar._