github/sql-server-samples
1
0
Fork 0
mirror of https://github.com/Microsoft/sql-server-samples.git synced 2025-12-08 14:58:54 +00:00
Code Issues Releases Wiki Activity

Re-organized spark samples

Browse Source
This commit is contained in:
Umachandar Jayachandran
2019-04-18 14:26:45 -07:00
parent 954fb3cd58
commit 92ae53e05b
7 changed files with 169 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/features/sql-big-data-cluster/spark/README.md
+16 -10
View File
@@ -1,23 +1,29 @@
# SQL Server big data clusters
The new built-in notebooks in Azure Data Studio enables data scientists and data engineers to run Python, R, or Scala code against the cluster.
The new built-in notebooks in Azure Data Studio enables data scientists and data engineers to run Python, R, Scala, or Spark SQL code against the cluster.
## Instructions to open a notebook from Azure Data Studio
## Instructions to open a notebook from Azure Data Studio and execute the commands
1. Connect to the SQL Server Master instance in a big data cluster
1. Right-click on the server name, select **Manage**, switch to **SQL Server Big Data Cluster** tab, and use open Notebook
1. Right-click on the server name, select **Manage**, switch to **SQL Server Big Data Cluster** tab, and use open Notebook.
## __[dataloading](dataloading/)__
1. Open the notebook in Azure Data Studio, wait for the “Kernel” and the target context (“Attach to”) to be populated.
This folder contains samples that show how to load data using Spark.
1. Run each cell in the Notebook sequentially.
[dataloading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
## __[data-loading](data-loading/)__
## Instructions
This folder contains samples that show how to load data using Spark and query them using SQL statements.
1. Download and save the notebook file [dataloading/transnform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/) locally.
[data-loading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
1. Open the notebook in Azure Data Studio, wait for the “Kernel” and the target context (“Attach to”) to be populated. Set the “Kernel” to **PySpark3** and **Attach to** needs to be the IP address of your big data cluster endpoint.
This samnple notebook shows how to transform CSV files in HDFS to parquet files.
1. Run each cell in the Notebook sequentially.
[dataloading/spark-sql.ipynb](dataloading/spark-sql.ipynb/)
This samnple notebook shows how to query hive tables created from Spark.
## __[data-virtualization](data-virtualization/)__
This folder contains samples that show how to integrate Spark with other data sources.