mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Merge pull request #676 from shivsood/connector_GDR1
MSSQL Spark connector sample updated
This commit is contained in:
shivsood
-150
@@ -1,150 +0,0 @@
|
||||
{
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "pyspark3kernel",
|
||||
"display_name": "PySpark3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "pyspark3",
|
||||
"mimetype": "text/x-python",
|
||||
"codemirror_mode": {
|
||||
"name": "python",
|
||||
"version": 3
|
||||
},
|
||||
"pygments_lexer": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat_minor": 2,
|
||||
"nbformat": 4,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Read and write from Spark to SQL using the MSSQL Spark Connector\r\nA typical big data scenario a key usage pattern is high volume, velocity and variety data processing in Spark followed with batch or streaming writes to SQL for access to LOB applications. These usage patterns greatly benefit from a connector that utilizes key SQL optimizations and provides an efficient write to SQLServer master instance and SQL Server data pool in Big Data Clusters.\r\n\r\nMSSQL Spark connector provides an efficient write SQLServer master instance and SQL Server data pool in Big Data Clusters.\r\n\r\nUsage\r\n----\r\n- Familiar Spark DataSource V1 interface\r\n- Referenced by the name \"mssql\" or fully qualified \"com.microsoft.sqlserver.jdbc.spark\"\r\n- Use from supported Spark language bindings ( Python, Scala, Java, R)\r\n- Optionally pass Bulk Copy parameters \r\n\r\n** Note : The image here may not be visible dues to markdown bug. Please change path here to full path to view the image.\r\n<img src =\r\n\"../data-virtualization/MSSQL_Spark_Connector2.jpg\" style=\"float: center;\" alt=\"drawing\" width=\"900\">\r\n\r\nMore details\r\n-----------\r\n\r\nMSSQL Spark connector, uses [SQL Server Bulk copy APIS](https://docs.microsoft.com/en-us/sql/connect/jdbc/using-bulk-copy-with-the-jdbc-driver?view=sql-server-2017#sqlserverbulkcopyoptions) to implement an efficient write to SQL Server. The connector is based on Spark Data source APIs and provides a familiar JDBC interface for access\r\n\r\nThe Sample\r\n---------\r\n\r\nThe following sample shows MSSQL JDBC Connector for read/write QLServer master instance and SQL Server data pool in Big Data Clusters. The sample is divided into 2 parts. The first part shows read/write to SQL Master instance and Part 2 shows read/write to Data Pools in Big Data Cluster. \r\n\r\nIn the sample we' ll \r\n- Read a file from HDFS and do some basic processing \r\n- In Part 1, we'll write the dataframe to SQL server table and then read the table to a dataframe .\r\n- In Part 2, we'll write the dataframe to SQL Server data pool external table and then read it back to a spark data frame. \r\n\r\nPreReq: \r\n- The sample uses a SQL database named \"MyTestDatabase\". Create this before you run this sample. The database can be created as follows\r\n ``` sql\r\n Create DATABASE MyTestDatabase\r\n GO \r\n ``` \r\n- Download [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine. Create a hdfs folder named spark_data and upload the file there. \r\n- [For CTP2.5] Configure the spark session to use the MSSQL Connector jar. The jar can be found at /jar/spark-mssql-connector-assembly-1.0.0.jar post deployment of Big Data Cluster.\r\n\r\n``` sh\r\n %%configure -f\r\n {\"conf\": {\"spark.jars\": \"/jar/spark-mssql-connector-assembly-1.0.0.jar\"}}\r\n```\r\n\r\n\r\n ",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Read CSV into a data frame\r\nIn this step we read the CSV into a data frame and do some basic cleanup steps. \r\n\r\n\r\n",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#spark = SparkSession.builder.getOrCreate()\r\nsc.setLogLevel(\"INFO\")\r\n\r\n#Read a file and then write it to the SQL table\r\ndatafile = \"/spark_data/AdultCensusIncome.csv\"\r\ndf = spark.read.format('csv').options(header='true', inferSchema='true', ignoreLeadingWhiteSpace='true', ignoreTrailingWhiteSpace='true').load(datafile)\r\ndf.show(5)\r\n\r\n\r\n#Process this data. Very simple data cleanup steps. Replacing \"-\" with \"_\" in column names\r\ncolumns_new = [col.replace(\"-\", \"_\") for col in df.columns]\r\ndf = df.toDF(*columns_new)\r\ndf.show(5)\r\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Starting Spark application\n"
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/plain": "<IPython.core.display.HTML object>",
|
||||
"text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>0</td><td>application_1561072870175_0001</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://40.78.87.105:30443/gateway/default/yarn/proxy/application_1561072870175_0001/\">Link</a></td><td><a target=\"_blank\" href=\"http://storage-0-0.storage-0-svc.testbdc.svc.cluster.local:8042/node/containerlogs/container_1561072870175_0001_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "SparkSession available as 'spark'.\n"
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education-num| marital-status| occupation| relationship| race| sex|capital-gain|capital-loss|hours-per-week|native-country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows\n\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows"
|
||||
}
|
||||
],
|
||||
"execution_count": 3
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# (PART 1) Write and READ to SQL Table\r\n- Write dataframe to SQL table to SQL Master\r\n- Read SQL Table to Spark dataframe",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Write from Spark to SQL table using MSSQL Spark Connector\r\nprint(\"Use MSSQL connector to write to master SQL instance \")\r\n\r\nservername = \"jdbc:sqlserver://master-0.master-svc\"\r\ndbname = \"MyTestDatabase\"\r\nurl = servername + \";\" + \"databaseName=\" + dbname + \";\"\r\n\r\ndbtable = \"AdultCensus_test\"\r\nuser = \"*****\"\r\npassword = \"*****\" # Please specify password here\r\n\r\n#com.microsoft.sqlserver.jdbc.spark\r\n\r\ntry:\r\n df.write \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .mode(\"overwrite\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", dbtable) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password) \\\r\n .save()\r\nexcept ValueError as error :\r\n print(\"MSSQL Connector write failed\", error)\r\n\r\nprint(\"MSSQL Connector write(overwrite) succeeded \")\r\n\r\n\r\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Use MSSQL connector to write to master SQL instance \nMSSQL Connector write(overwrite) succeeded"
|
||||
}
|
||||
],
|
||||
"execution_count": 5
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Use mode as append\r\ntry:\r\n df.write \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .mode(\"append\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", dbtable) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password) \\\r\n .save()\r\nexcept ValueError as error :\r\n print(\"MSSQL Connector write failed\", error)\r\n\r\nprint(\"MSSQL Connector write(append) succeeded \")",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "MSSQL Connector write(append) succeeded"
|
||||
}
|
||||
],
|
||||
"execution_count": 6
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Read from SQL table using MSSQ Connector\r\nprint(\"read data from SQL server table \")\r\njdbcDF = spark.read \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", dbtable) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password).load()\r\n\r\njdbcDF.show(5)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "read data from SQL server table \n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows"
|
||||
}
|
||||
],
|
||||
"execution_count": 7
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# (PART 2) Write and READ to Data Pool external Tables in Big Data Cluster\r\n- Write dataframe to SQL external table in Data Pools in Big Data Cluste\r\n- Read SQL external Table to Spark dataframe",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Write from Spark to SQL table using MSSQL Spark Connector\r\nprint(\"Use MSSQL connector to write to master SQL instance \")\r\n\r\ndatapool_table = \"AdultCensus_DataPoolTable\"\r\ndatasource_name = \"test_data_src\"\r\n\r\n\r\ntry:\r\n df.write \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .mode(\"overwrite\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", datapool_table) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password) \\\r\n .option(\"dataPoolDataSource\",datasource_name)\\\r\n .save()\r\nexcept ValueError as error :\r\n print(\"MSSQL Connector write failed\", error)\r\n\r\nprint(\"MSSQL Connector write(overwrite) to data pool external table succeeded\")\r\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Use MSSQL connector to write to master SQL instance \nMSSQL Connector write(overwrite) to data pool external table succeeded"
|
||||
}
|
||||
],
|
||||
"execution_count": 8
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "try:\r\n df.write \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .mode(\"append\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", datapool_table) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password) \\\r\n .option(\"dataPoolDataSource\",datasource_name)\\\r\n .save()\r\nexcept ValueError as error :\r\n print(\"MSSQL Connector write failed\", error)\r\n\r\nprint(\"MSSQL Connector write(append) to data pool external table succeeded\")",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "MSSQL Connector write(append) to data pool external table succeeded"
|
||||
}
|
||||
],
|
||||
"execution_count": 9
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Read from SQL table using MSSQ Connector\r\nprint(\"read data from SQL server table \")\r\njdbcDF = spark.read \\\r\n .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n .option(\"url\", url) \\\r\n .option(\"dbtable\", datapool_table) \\\r\n .option(\"user\", user) \\\r\n .option(\"password\", password)\\\r\n .load()\r\n\r\njdbcDF.show(5)\r\n\r\nprint(\"MSSQL Connector read from data pool external table succeeded\")",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "read data from SQL server table \n+---+----------------+------+----------+-------------+------------------+---------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt| education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+----------+-------------+------------------+---------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 46|Self-emp-not-inc|277946|Assoc-acdm| 12| Separated| Craft-repair|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 38| Private| 91039| Bachelors| 13|Married-civ-spouse| Sales| Husband|White| Male| 15024| 0| 60| United-States| >50K|\n| 18| Private|156764| 11th| 7| Never-married| Other-service| Own-child|White| Male| 0| 0| 40| United-States| <=50K|\n| 34| Private|136721| HS-grad| 9| Divorced|Exec-managerial|Not-in-family|White|Female| 0| 0| 40| United-States| <=50K|\n| 39| State-gov| 77516| Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n+---+----------------+------+----------+-------------+------------------+---------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows\n\nMSSQL Connector read from data pool external table succeeded"
|
||||
}
|
||||
],
|
||||
"execution_count": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
+341
@@ -0,0 +1,341 @@
|
||||
{
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "pyspark3kernel",
|
||||
"display_name": "PySpark3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "pyspark3",
|
||||
"mimetype": "text/x-python",
|
||||
"codemirror_mode": {
|
||||
"name": "python",
|
||||
"version": 3
|
||||
},
|
||||
"pygments_lexer": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat_minor": 2,
|
||||
"nbformat": 4,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Read and Write from Spark to SQL using the MSSQL Spark Connector\r\n",
|
||||
"A typical big data scenario a key usage pattern is high volume, velocity and variety data processing in Spark followed with batch or streaming writes to SQL for access to LOB applications. These usage patterns greatly benefit from a connector that utilizes key SQL optimizations and provides an efficient write to SQLServer master instance and SQL Server data pool in Big Data Clusters.\r\n",
|
||||
"\r\n",
|
||||
"MSSQL Spark connector provides an efficient write SQLServer master instance and SQL Server data pool in Big Data Clusters.\r\n",
|
||||
"\r\n",
|
||||
"Usage\r\n",
|
||||
"----\r\n",
|
||||
"- Familiar Spark DataSource V1 interface\r\n",
|
||||
"- Referenced by fully qualified name \"com.microsoft.sqlserver.jdbc.spark\"\r\n",
|
||||
"- Use from supported Spark language bindings ( Python, Scala, Java, R)\r\n",
|
||||
"- Optionally pass Bulk Copy parameters \r\n",
|
||||
"\r\n",
|
||||
"** Note : The image here may not be visible dues to markdown bug. Please change path here to full path to view the image.\r\n",
|
||||
"<img src =\r\n",
|
||||
"\"../data-virtualization/MSSQL_Spark_Connector2.jpg\" style=\"float: center;\" alt=\"drawing\" width=\"900\">\r\n",
|
||||
"\r\n",
|
||||
"More details\r\n",
|
||||
"-----------\r\n",
|
||||
"\r\n",
|
||||
"MSSQL Spark connector, uses [SQL Server Bulk copy APIS](https://docs.microsoft.com/en-us/sql/connect/jdbc/using-bulk-copy-with-the-jdbc-driver?view=sql-server-2017#sqlserverbulkcopyoptions) to implement an efficient write to SQL Server. The connector is based on Spark Data source APIs and provides a familiar JDBC interface for access\r\n",
|
||||
"\r\n",
|
||||
"The Sample\r\n",
|
||||
"---------\r\n",
|
||||
"\r\n",
|
||||
"The following sample shows MSSQL JDBC Connector for read/write SQLServer master instance and SQL Server data pool in Big Data Clusters. The sample is divided into 2 parts. \r\n",
|
||||
"- Part 1 shows read/write to SQL Master instance and \r\n",
|
||||
"- Part 2 shows read/write to Data Pools in Big Data Cluster. \r\n",
|
||||
"\r\n",
|
||||
"In the sample we' ll \r\n",
|
||||
"- Read a file from HDFS and do some basic processing \r\n",
|
||||
"- In Part 1, we'll write the dataframe to SQL server table and then read the table to a dataframe .\r\n",
|
||||
"- In Part 2, we'll write the dataframe to SQL Server data pool external table and then read it back to a spark data frame. \r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
" "
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "2757df21-8174-4bb1-a52c-66eaf94f6b96"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## PreReq\r\n",
|
||||
"-------\r\n",
|
||||
"- Download [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine. Upload this file to hdfs folder named *spark_data*. \r\n",
|
||||
"- The sample uses a SQL database *connector_test_db*, user *connector_user* with password *password123!#* and datasource *connector_ds*. The database, user/password and datasource need to be created before running the full sample. Refer **data-virtualization/mssql_spark_connector_user_creation.ipynb** on steps to create this user."
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "48874729-541f-4b01-888f-3fdd2aeb59da"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Read CSV into a data frame\r\n",
|
||||
"In this step we read the CSV into a data frame and do some basic cleanup steps. \r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "ed8b58e0-3607-4a71-8dc8-034bc0180ee4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#spark = SparkSession.builder.getOrCreate()\r\n",
|
||||
"sc.setLogLevel(\"INFO\")\r\n",
|
||||
"\r\n",
|
||||
"#Read a file and then write it to the SQL table\r\n",
|
||||
"datafile = \"/spark_data/AdultCensusIncome.csv\"\r\n",
|
||||
"df = spark.read.format('csv').options(header='true', inferSchema='true', ignoreLeadingWhiteSpace='true', ignoreTrailingWhiteSpace='true').load(datafile)\r\n",
|
||||
"df.show(5)\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"#Process this data. Very simple data cleanup steps. Replacing \"-\" with \"_\" in column names\r\n",
|
||||
"columns_new = [col.replace(\"-\", \"_\") for col in df.columns]\r\n",
|
||||
"df = df.toDF(*columns_new)\r\n",
|
||||
"df.show(5)\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "813bbfa3-2613-45dd-9556-94faba602977"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education-num| marital-status| occupation| relationship| race| sex|capital-gain|capital-loss|hours-per-week|native-country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows\n\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 4
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# (PART 1) Write and READ to SQL Table\r\n",
|
||||
"- Write dataframe to SQL table to SQL Master\r\n",
|
||||
"- Read SQL Table to Spark dataframe"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "a6afceb2-6fbc-435b-af88-e9f5cc784f5d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Write from Spark to SQL table using MSSQL Spark Connector\r\n",
|
||||
"print(\"Use MSSQL connector to write to master SQL instance \")\r\n",
|
||||
"\r\n",
|
||||
"servername = \"jdbc:sqlserver://master-0.master-svc\"\r\n",
|
||||
"dbname = \"connector_test_db\"\r\n",
|
||||
"url = servername + \";\" + \"databaseName=\" + dbname + \";\"\r\n",
|
||||
"\r\n",
|
||||
"dbtable = \"AdultCensus_test\"\r\n",
|
||||
"user = \"connector_user\"\r\n",
|
||||
"password = \"password123!#\" # Please specify password here\r\n",
|
||||
"\r\n",
|
||||
"#com.microsoft.sqlserver.jdbc.spark\r\n",
|
||||
"\r\n",
|
||||
"try:\r\n",
|
||||
" df.write \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .mode(\"overwrite\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", dbtable) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password) \\\r\n",
|
||||
" .save()\r\n",
|
||||
"except ValueError as error :\r\n",
|
||||
" print(\"MSSQL Connector write failed\", error)\r\n",
|
||||
"\r\n",
|
||||
"print(\"MSSQL Connector write(overwrite) succeeded \")\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "b851fe61-6e85-4e46-a20f-4063fc6586e0"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "Use MSSQL connector to write to master SQL instance \nMSSQL Connector write(overwrite) succeeded",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 5
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Use mode as append\r\n",
|
||||
"try:\r\n",
|
||||
" df.write \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .mode(\"append\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", dbtable) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password) \\\r\n",
|
||||
" .save()\r\n",
|
||||
"except ValueError as error :\r\n",
|
||||
" print(\"MSSQL Connector write failed\", error)\r\n",
|
||||
"\r\n",
|
||||
"print(\"MSSQL Connector write(append) succeeded \")"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "fbfa1938-59a3-4424-acf7-844c3ef0984a"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "MSSQL Connector write(append) succeeded",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 6
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Read from SQL table using MSSQ Connector\r\n",
|
||||
"print(\"read data from SQL server table \")\r\n",
|
||||
"jdbcDF = spark.read \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", dbtable) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password).load()\r\n",
|
||||
"\r\n",
|
||||
"jdbcDF.show(5)"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "e3e19e1f-1325-47ea-87d1-1170f316d2d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "read data from SQL server table \n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt|education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39| State-gov| 77516|Bachelors| 13| Never-married| Adm-clerical|Not-in-family|White| Male| 2174| 0| 40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors| 13|Married-civ-spouse| Exec-managerial| Husband|White| Male| 0| 0| 13| United-States| <=50K|\n| 38| Private|215646| HS-grad| 9| Divorced|Handlers-cleaners|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 53| Private|234721| 11th| 7|Married-civ-spouse|Handlers-cleaners| Husband|Black| Male| 0| 0| 40| United-States| <=50K|\n| 28| Private|338409|Bachelors| 13|Married-civ-spouse| Prof-specialty| Wife|Black|Female| 0| 0| 40| Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 7
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# (PART 2) Write and READ to Data Pool external Tables in Big Data Cluster\r\n",
|
||||
"- Write dataframe to SQL external table in Data Pools in Big Data Cluste\r\n",
|
||||
"- Read SQL external Table to Spark dataframe"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "375fa271-fd2a-4a89-a782-0e34ba9baf3c"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Write from Spark to SQL table using MSSQL Spark Connector\r\n",
|
||||
"print(\"Use MSSQL connector to write to master SQL instance \")\r\n",
|
||||
"\r\n",
|
||||
"datapool_table = \"AdultCensus_DataPoolTable\"\r\n",
|
||||
"datasource_name = \"connector_ds\"\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"try:\r\n",
|
||||
" df.write \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .mode(\"overwrite\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", datapool_table) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password) \\\r\n",
|
||||
" .option(\"dataPoolDataSource\",datasource_name)\\\r\n",
|
||||
" .save()\r\n",
|
||||
"except ValueError as error :\r\n",
|
||||
" print(\"MSSQL Connector write failed\", error)\r\n",
|
||||
"\r\n",
|
||||
"print(\"MSSQL Connector write(overwrite) to data pool external table succeeded\")\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "f12a886e-6423-40aa-83ab-e8578b56a10c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "Use MSSQL connector to write to master SQL instance \nMSSQL Connector write(overwrite) to data pool external table succeeded",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 8
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"try:\r\n",
|
||||
" df.write \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .mode(\"append\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", datapool_table) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password) \\\r\n",
|
||||
" .option(\"dataPoolDataSource\",datasource_name)\\\r\n",
|
||||
" .save()\r\n",
|
||||
"except ValueError as error :\r\n",
|
||||
" print(\"MSSQL Connector write failed\", error)\r\n",
|
||||
"\r\n",
|
||||
"print(\"MSSQL Connector write(append) to data pool external table succeeded\")"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "a9340485-9bfe-414a-a938-09d71976a5b3"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "MSSQL Connector write(append) to data pool external table succeeded",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 9
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Read from SQL table using MSSQ Connector\r\n",
|
||||
"print(\"read data from SQL server table \")\r\n",
|
||||
"jdbcDF = spark.read \\\r\n",
|
||||
" .format(\"com.microsoft.sqlserver.jdbc.spark\") \\\r\n",
|
||||
" .option(\"url\", url) \\\r\n",
|
||||
" .option(\"dbtable\", datapool_table) \\\r\n",
|
||||
" .option(\"user\", user) \\\r\n",
|
||||
" .option(\"password\", password)\\\r\n",
|
||||
" .load()\r\n",
|
||||
"\r\n",
|
||||
"jdbcDF.show(5)\r\n",
|
||||
"\r\n",
|
||||
"print(\"MSSQL Connector read from data pool external table succeeded\")"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "e120a1ab-72e5-4ae7-8bd4-b8d45c1e2267"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "read data from SQL server table \n+---+----------------+------+------------+-------------+------------------+-------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age| workclass|fnlwgt| education|education_num| marital_status| occupation| relationship| race| sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+------------+-------------+------------------+-------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 36| Private| 99374|Some-college| 10| Divorced| Craft-repair|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 27| Private|248402| Bachelors| 13| Never-married| Tech-support| Unmarried|Black|Female| 0| 0| 40| United-States| <=50K|\n| 46|Self-emp-not-inc|277946| Assoc-acdm| 12| Separated| Craft-repair|Not-in-family|White| Male| 0| 0| 40| United-States| <=50K|\n| 38| Private| 91039| Bachelors| 13|Married-civ-spouse| Sales| Husband|White| Male| 15024| 0| 60| United-States| >50K|\n| 18| Private|156764| 11th| 7| Never-married|Other-service| Own-child|White| Male| 0| 0| 40| United-States| <=50K|\n+---+----------------+------+------------+-------------+------------------+-------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows\n\nMSSQL Connector read from data pool external table succeeded",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "sparkrkernel",
|
||||
"display_name": "Spark | R"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "sparkR",
|
||||
"mimetype": "text/x-rsrc",
|
||||
"codemirror_mode": "text/x-rsrc",
|
||||
"pygments_lexer": "r"
|
||||
}
|
||||
},
|
||||
"nbformat_minor": 2,
|
||||
"nbformat": 4,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Using MsSQLSpark Conenctor from SparkR\r\n",
|
||||
"\r\n",
|
||||
"The following notebook demostrates usage of MSSQLSpark connector in sparkR. For full set of capability supported by MSSQLSpark Connector refer **mssql_spark_connector_non_ad_pyspark.ipynb** "
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "f4f5a3ce-b06b-4bcd-be5d-ae7e2fcd6108"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## PreReq\r\n",
|
||||
"-------\r\n",
|
||||
"- Download [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine. Upload this file to hdfs folder named *spark_data*. \r\n",
|
||||
"- The sample uses a SQL database *connector_test_db*, user *connector_user* with password *password123!#* and datasource *connector_ds*. The database, user/password and datasource need to be created before running the full sample. Refer **data-virtualization/mssql_spark_connector_user_creation.ipynb** on steps to create this user."
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "820bc224-56e1-4867-9df8-ddd9c3aa3a93"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Load a CSV file to a dataframe"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "29e3c7f0-b2d0-4456-949d-b80dc6857486"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"people <- read.df(\"/spark_data/AdultCensusIncome.csv\", \"csv\")\r\n",
|
||||
"head(people)"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "6be47cc0-0a41-4fbe-b43d-bf363e3c4c0a"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "Starting Spark application\n",
|
||||
"output_type": "stream"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": "<IPython.core.display.HTML object>",
|
||||
"text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>5</td><td>application_1572997761827_0009</td><td>sparkr</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.91.90.171:30443/gateway/default/yarn/proxy/application_1572997761827_0009/\">Link</a></td><td><a target=\"_blank\" href=\"https://10.91.90.171:30443/gateway/default/yarn/container/container_1572997761827_0009_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "SparkSession available as 'spark'.\n",
|
||||
"output_type": "stream"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": " _c0 _c1 _c2 _c3 _c4 _c5\n1 age workclass fnlwgt education education-num marital-status\n2 39 State-gov 77516 Bachelors 13 Never-married\n3 50 Self-emp-not-inc 83311 Bachelors 13 Married-civ-spouse\n4 38 Private 215646 HS-grad 9 Divorced\n5 53 Private 234721 11th 7 Married-civ-spouse\n6 28 Private 338409 Bachelors 13 Married-civ-spouse\n _c6 _c7 _c8 _c9 _c10 _c11\n1 occupation relationship race sex capital-gain capital-loss\n2 Adm-clerical Not-in-family White Male 2174 0\n3 Exec-managerial Husband White Male 0 0\n4 Handlers-cleaners Not-in-family White Male 0 0\n5 Handlers-cleaners Husband Black Male 0 0\n6 Prof-specialty Wife Black Female 0 0\n _c12 _c13 _c14\n1 hours-per-week native-country income\n2 40 United-States <=50K\n3 13 United-States <=50K\n4 40 United-States <=50K\n5 40 United-States <=50K\n6 40 Cuba <=50K",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 3
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## User MSQL Spark connector to save the dataframe as a table in SQL Server"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "f8304a0e-e3ef-4666-8f15-c1abddac9b25"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Using deafault JDBC connector\r\n",
|
||||
"#Using MSSQLSpark connector\r\n",
|
||||
"dbname = \"connector_test_db\"\r\n",
|
||||
"url = paste(\"jdbc:sqlserver://master-0.master-svc;databaseName=\", \"connector_test_db\", sep=\"\")\r\n",
|
||||
"print(url)\r\n",
|
||||
"\r\n",
|
||||
"dbtable = \"AdultCensus_test_sparkr\"\r\n",
|
||||
"user = \"connector_user\"\r\n",
|
||||
"password = \"password123!#\" # Please specify password here\r\n",
|
||||
"\r\n",
|
||||
"saveDF(people,\r\n",
|
||||
" dbtable,\r\n",
|
||||
" source = \"com.microsoft.sqlserver.jdbc.spark\", \r\n",
|
||||
" url = url, \r\n",
|
||||
" dbtable=dbtable, \r\n",
|
||||
" mode = \"overwrite\",\r\n",
|
||||
" user=user, \r\n",
|
||||
" password=password)\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "7216f3a5-0439-44a8-a726-f393db2d612b"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "[1] \"jdbc:sqlserver://master-0.master-svc;databaseName=connector_test_db\"",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
{
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "SQL",
|
||||
"display_name": "SQL",
|
||||
"language": "sql"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "sql",
|
||||
"version": ""
|
||||
}
|
||||
},
|
||||
"nbformat_minor": 2,
|
||||
"nbformat": 4,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# User Creation for MSQL Spark Connector\r\n",
|
||||
"MSSQL Spark connector provides an efficient write SQLServer master instance and SQL Server data pool in Big Data Clusters. To transfer data to SQL Server the MSSQL Spark Connector requires the right credentials. This notebook explains creation of the user account the permissions for use in MSSQL Spark Connector.\r\n",
|
||||
"\r\n",
|
||||
"## What permissions does MSSQL Spark connector requires.\r\n",
|
||||
"The answer depends on the capability that's used. MSSQL Spark connector can read, write, create tables in SQL Server Master instance and Data Pools in SQL Server 2019 Big Data Clusters. For using the full set of functionality the user account needs to have permissions to read, write, create tables and external tables in data pools.\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "4d6539fc-aa99-4cd8-a9de-357e2d6424e2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"-- Start with creating a testdatabase. \r\n",
|
||||
"Create DATABASE connector_test_db"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "8fdf2f56-f89d-4bf9-81be-1dfb92631251"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Commands completed successfully."
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Total execution time: 00:00:01.345"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"Use connector_test_db;\n",
|
||||
"CREATE LOGIN connector_user WITH PASSWORD ='password123!#' \n",
|
||||
"CREATE USER connector_user FROM LOGIN connector_user\n",
|
||||
"\n",
|
||||
"-- To create external tables in data pools\n",
|
||||
"grant alter any external data source to connector_user;\n",
|
||||
"\n",
|
||||
"-- To create external table\n",
|
||||
"grant create table to connector_user;\n",
|
||||
"grant alter any schema to connector_user;\n",
|
||||
"\n",
|
||||
"ALTER ROLE [db_datareader] ADD MEMBER connector_user\n",
|
||||
"ALTER ROLE [db_datawriter] ADD MEMBER connector_user"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "4dd976a3-1ed7-43d3-ad29-657fe1046fb3"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Commands completed successfully."
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Total execution time: 00:00:00.087"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"execution_count": 2
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"-- Add this user to a data pool user.\r\n",
|
||||
"Use connector_test_db;\r\n",
|
||||
"CREATE EXTERNAL DATA SOURCE connector_ds WITH (LOCATION = 'sqldatapool://controller-svc/default');\r\n",
|
||||
"EXECUTE('USE connector_test_db; CREATE EXTERNAL TABLE [dummy3] ([number] int, [word] nvarchar(2048)) WITH (DATA_SOURCE = connector_ds, DISTRIBUTION = ROUND_ROBIN)')\r\n",
|
||||
"\r\n",
|
||||
""
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "0b06540d-0787-493d-aba6-83420200da14"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Commands completed successfully."
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Total execution time: 00:00:04.496"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"execution_count": 3
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"-- Create a login in data pools and Provide right permissions to this user\r\n",
|
||||
"EXECUTE( ' Use connector_test_db; CREATE LOGIN connector_user WITH PASSWORD = ''password123!#'' ; ' ) AT DATA_SOURCE connector_ds;\r\n",
|
||||
"EXECUTE( ' Use connector_test_db; CREATE USER connector_user ; ALTER ROLE [db_datareader] ADD MEMBER connector_user; ALTER ROLE [db_datawriter] ADD MEMBER connector_user ;') AT DATA_SOURCE connector_ds;"
|
||||
],
|
||||
"metadata": {
|
||||
"azdata_cell_guid": "5079dfe9-22e3-4a8f-919b-c6219f5a901a"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Commands completed successfully."
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Total execution time: 00:00:01.174"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"execution_count": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user