Read and write from Spark to SQL

A typical big data scenario is large scale ETL in Spark and post processing the data is written out to SQLServer for access to LOB applications. This sample shows how to write to SQLServer from Spark. The main steps in the sample are

Reading a HDFS file,
Basic processing on it and
Then writing processed data to SQL Server table using JDBC

PreReq :

The sample uses a SQL database named "MyTestDatabase". Create this before you run this sample. The database can be created as follows
```
Create DATABASE MyTestDatabase
GO 
```
Download AdultCensusIncome.csv to your local machine. Create a hdfs folder named spark_data and upload the file there.

In [3]:


#Read a file and then write it to the SQL table
datafile = "/spark_data/AdultCensusIncome.csv"
df = spark.read.format('csv').options(header='true', inferSchema='true', ignoreLeadingWhiteSpace='true', ignoreTrailingWhiteSpace='true').load(datafile)
df.show(5)

Starting Spark application

ID	YARN Application ID	Kind	State	Spark UI	Driver log	Current session?
2	application_1554755839506_0003	pyspark3	idle	Link	Link	✔

SparkSession available as 'spark'.

+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
|age|       workclass|fnlwgt|education|education-num|    marital-status|       occupation| relationship| race|   sex|capital-gain|capital-loss|hours-per-week|native-country|income|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|
| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|
| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|
| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|
| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
only showing top 5 rows

In [4]:


#Process this data. Very simple data cleanup steps. Replacing "-" with "_" in column names
columns_new = [col.replace("-", "_") for col in df.columns]
df = df.toDF(*columns_new)
df.show(5)

+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|
| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|
| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|
| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|
| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
only showing top 5 rows

In [9]:

#Write from Spark to SQL table using JDBC
print("Use build in JDBC connector to write to SQLServer master instance in Big data ")

servername = "jdbc:sqlserver://master-0.master-svc"
dbname = "MyTestDatabase"
url = servername + ";" + "databaseName=" + dbname + ";"

dbtable = "dbo.AdultCensus"
user = "***"
password = "****"

print("url is ", url)

try:
  df.write \
    .format("jdbc") \
    .mode("overwrite") \
    .option("url", url) \
    .option("dbtable", dbtable) \
    .option("user", user) \
    .option("password", password)\
    .save()
except ValueError as error :
    print("JDBC Write failed", error)

print("JDBC Write done  ")

Use build in JDBC connector to write to SQLServer master instance in Big data 
url is  jdbc:sqlserver://master-0.master-svc;databaseName=MyTestDatabase;
JDBC Write done

In [11]:

#Read to Spark from SQL table using JDBC
print("read data from SQL server table  ")
jdbcDF = spark.read \
        .format("jdbc") \
        .option("url", url
        ) \
        .option("dbtable", dbtable) \
        .option("user", user) \
        .option("password", password) \
        .load()

jdbcDF.show(5)

read data from SQL server table  
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|
| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|
| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|
| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|
| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
only showing top 5 rows

10 KiB Raw Blame History

Read and write from Spark to SQL

10 KiB

Raw Blame History