sql-server-samples/samples/features/sql-big-data-cluster/spark/config-install/configure_spark_session.ipynb

Configuring a Spark session using configure-f

Refer to Spark Configurations for specific parameters

In [3]:

%%configure -f
{"conf": {
    "spark.executor.memory": "4g",
    "spark.driver.memory": "4g",
    "spark.executor.cores": 2,
    "spark.driver.cores": 1,
    "spark.executor.instances": 4
        }
}

Current session configs: {'conf': {'spark.executor.memory': '4g', 'spark.driver.memory': '4g', 'spark.executor.cores': 2, 'spark.driver.cores': 1, 'spark.executor.instances': 4}, 'kind': 'pyspark3'}

ID	YARN Application ID	Kind	State	Spark UI	Driver log	Current session?
93	application_1558765999724_0190	pyspark	idle	Link	Link

In [4]:

datafile = "/spark_data/AdultCensusIncome.csv"
df = spark.read.format('csv').options(header='true', inferSchema='true').load(datafile)

df.show(5)

Starting Spark application

ID	YARN Application ID	Kind	State	Spark UI	Driver log	Current session?
96	application_1558765999724_0193	pyspark3	idle	Link	Link	✔

SparkSession available as 'spark'.

+---+-----------------+--------+----------+--------------+-------------------+------------------+--------------+------+-------+-------------+-------------+---------------+---------------+-------+
|age|        workclass|  fnlwgt| education| education-num|     marital-status|        occupation|  relationship|  race|    sex| capital-gain| capital-loss| hours-per-week| native-country| income|
+---+-----------------+--------+----------+--------------+-------------------+------------------+--------------+------+-------+-------------+-------------+---------------+---------------+-------+
| 39|        State-gov| 77516.0| Bachelors|          13.0|      Never-married|      Adm-clerical| Not-in-family| White|   Male|       2174.0|          0.0|           40.0|  United-States|  <=50K|
| 50| Self-emp-not-inc| 83311.0| Bachelors|          13.0| Married-civ-spouse|   Exec-managerial|       Husband| White|   Male|          0.0|          0.0|           13.0|  United-States|  <=50K|
| 38|          Private|215646.0|   HS-grad|           9.0|           Divorced| Handlers-cleaners| Not-in-family| White|   Male|          0.0|          0.0|           40.0|  United-States|  <=50K|
| 53|          Private|234721.0|      11th|           7.0| Married-civ-spouse| Handlers-cleaners|       Husband| Black|   Male|          0.0|          0.0|           40.0|  United-States|  <=50K|
| 28|          Private|338409.0| Bachelors|          13.0| Married-civ-spouse|    Prof-specialty|          Wife| Black| Female|          0.0|          0.0|           40.0|           Cuba|  <=50K|
+---+-----------------+--------+----------+--------------+-------------------+------------------+--------------+------+-------+-------------+-------------+---------------+---------------+-------+
only showing top 5 rows

In [21]:

from pyspark import SparkConf
from pyspark.sql import SparkSession

def isConfiguredItem(cfg_items):
    if(cfg_items == 'spark.executor.instances' or cfg_items == 'spark.executor.memory' or \
       cfg_items == 'spark.executor.cores' or cfg_items == 'spark.driver.memory' or \
       cfg_items == 'spark.driver.cores'):
        return True

spark = SparkSession.builder.getOrCreate()
conf = SparkConf().getAll()

for cfg_items in conf:
    if(isConfiguredItem(cfg_items[0])):
        print(cfg_items)

('spark.executor.instances', '4')
('spark.driver.memory', '4g')
('spark.driver.cores', '1')
('spark.executor.memory', '4g')
('spark.executor.cores', '2')