Initial samples for SQL Server 2019 big data cluster

Demonstrates various functionality in big data cluster.
2025-12-08 14:58:54 +00:00 · 2018-10-11 14:48:44 -07:00
parent 5083b5c707
commit 09b207f69a
20 changed files with 862 additions and 0 deletions
@@ -0,0 +1,48 @@
+# Data ingestion using Spark streaming
+
+SQL Server Big Data clusters provide scale-out compute and storage to improve the performance of analyzing any data. Data from a variety of sources can be ingested and distributed across data pool instances for analysis. In this example, you are going to use Spark to read and transform data from HDFS and cache it in a data pool. Querying the external table created over this aggregated data stored in data pools will be much more efficient than going to the raw data always.  
+
+### Instructions
+
+1. Using Azure Data Studio, connect to the HDFS/Spark gateway (SQL Server big data cluster connection type).
+
+1. Connect to SQL Server Master instance using Azure Data Studio.
+
+1. Execute the SQL script [data-ingestion-spark.sql](data-ingestion-spark.sql).
+
+1. Create and submit a Spark job that ingests data from HDFS into the external table.
+
+Submitting a Spark job will start a Spark streaming session using spark-submit.
+    
+    The arguments to the jar file are:
+
+    1. server name - sql server to connect to read the table schema
+    2. port number 
+    3. username - sql server username for master instance
+    4. password - sql server password for master instance
+    5. database name
+    6. external table name
+    7. Source directory for streaming. This must be a full URI - such as "hdfs:///clickstream_data"
+    8. Input format. This can be "csv", "parquet", "json".
+    9. enable checkpoint: true or false
+
+  Submit a Spark job with the below parameters. You can use the Spark submit experience from Azure Data Studio (right click on big data cluster endpoint -> Submit Spark Job):
+
+    ARGUMENTS:
+    
+    **job name:** yourJobName
+
+    **switch** from "Local" to "HDFS"
+    
+    **Path to jar** (copy/paste this):
+
+    /jar/mssql-spark-lib-assembly-1.0.jar
+
+    **Main class:**
+    FileStreaming
+
+    **Parameters (copy/paste this; make sure you replace the password!):**
+    
+    mssql-master-pool-0.service-master-pool 1433 sa passwordHere sales web_clickstreams_spark_results hdfs:///clickstream_data csv false
+
+6. Query the external table we created earlier using the SELECT queries in the script to see data coming from the streaming job and landing in the table.
@@ -0,0 +1,54 @@
+USE sales
+GO
+
+-- Create external table in a data pool in SQL Server 2019 big data cluster.
+-- The SqlDataPool data source is a special data source that is available in 
+-- any new database in SQL Master instance. This is used to reference the
+-- data pool in a SQL Server 2019 big data cluster.
+--
+CREATE EXTERNAL TABLE [web_clickstreams_spark_results]
+("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
+WITH
+(
+    DATA_SOURCE = SqlDataPool,
+	DISTRIBUTION = ROUND_ROBIN
+);
+
+-- Data can be ingested into the external table from a spark job.
+--
+-- Submit spark job with below parameters. You can use the Spark submit experience from Azure Data Studio.
+-- Right click on server name in a SQL Server big data cluster connection and click "Submit Spark Job".
+--
+-- Specify following values in the Job submission dialog box:
+---- job name: <yourJobName>
+---- switch from "Local" to "HDFS"
+---- Main class: "FileStreaming"
+---- Path to jar: /jar/mssql-spark-lib-assembly-1.0.jar
+---- Arguments:
+---- mssql-master-pool-0.service-master-pool 1433 sa %PASSWORD% sales web_clickstreams_spark_results hdfs:///clickstream_data csv false 
+
+-- The arguments to jar file are
+-- 1: server name - sql server to connect to read the table schema
+-- 2: port number
+-- 3: username - sql server username for master instance
+-- 4: password - sql server password for master instance
+-- 5: database name
+-- 6: external table name
+-- 7: Source directory for streaming. This must be a full URI - such as "hdfs:///clickstream_data"
+-- 8: Input format. This can be "csv", "parquet", "json".
+-- 9: enable checkpoint: true or false
+--
+
+-- After the Spark streaming job has been sucessfully submitted, you can run below query to view the results.
+--
+-- Wait until some rows are available.
+WHILE (1=1)
+    IF EXISTS(SELECT * FROM [web_clickstreams_spark_results])
+        BREAK;
+
+SELECT count(*) FROM [web_clickstreams_spark_results];
+SELECT TOP 10 * FROM [web_clickstreams_spark_results];
+GO
+
+DROP EXTERNAL TABLE [dbo].[web_clickstreams_spark_results];
+GO
@@ -0,0 +1,9 @@
+# Data ingestion using SQL stored procedure
+
+SQL Server Big Data clusters provide scale-out compute and storage to improve the performance of analyzing any data. Data from a variety of sources can be ingested and distributed across data pool instances for analysis. In this example, we will insert data from a SQL query into an external table stored in a data pool and query it.
+
+## Instructions
+
+1. Connect to SQL Server Master instance.
+
+1. Execute the .sql script [data-ingestion-sql.sql](data-ingestion-sql.sql).
@@ -0,0 +1,58 @@
+USE sales
+GO
+
+-- Create external table in a data pool in SQL Server 2019 big data cluster.
+-- The SqlDataPool data source is a special data source that is available in 
+-- any new database in SQL Master instance. This is used to reference the
+-- data pool in a SQL Server 2019 big data cluster.
+--
+CREATE EXTERNAL TABLE [web_clickstreams_dp]
+("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
+WITH
+(
+    DATA_SOURCE = SqlDataPool,
+	DISTRIBUTION = ROUND_ROBIN
+);
+GO
+-- Currently the create external table operation is asynchronous and there is no
+-- way to determine completion of the operation. To prevent failures of the insert
+-- into the external table, wait for few minutes.
+WAITFOR DELAY '00:02:00';
+GO
+-- Insert results of a SELECT statement into the external table created on the data pool
+--
+DECLARE @db_name SYSNAME = 'sales'
+DECLARE @schema_name SYSNAME = 'dbo'
+DECLARE @table_name SYSNAME = 'web_clickstreams_dp'
+DECLARE @query SYSNAME = 'SELECT TOP(1000) * FROM sales.dbo.web_clickstreams WHERE wcs_user_sk IS NOT NULL'
+
+EXEC model..sp_data_pool_table_insert_data @db_name, @schema_name, @table_name, @query
+GO
+
+-- Query data inserted from sp_data_pool_table_insert_data
+--
+SELECT count(*) FROM [dbo].[web_clickstreams_dp]
+SELECT TOP 10 * FROM [dbo].[web_clickstreams_dp]
+
+-- Join external table with local tables
+--
+SELECT TOP (100)
+    wcs_user_sk,
+    SUM( CASE WHEN i_category = 'Books' THEN 1 ELSE 0 END) AS book_category_clicks,
+    SUM( CASE WHEN i_category_id = 1 THEN 1 ELSE 0 END) AS [Home & Kitchen],
+    SUM( CASE WHEN i_category_id = 2 THEN 1 ELSE 0 END) AS [Music],
+    SUM( CASE WHEN i_category_id = 3 THEN 1 ELSE 0 END) AS [Books],
+    SUM( CASE WHEN i_category_id = 4 THEN 1 ELSE 0 END) AS [Clothing & Accessories],
+    SUM( CASE WHEN i_category_id = 5 THEN 1 ELSE 0 END) AS [Electronics],
+    SUM( CASE WHEN i_category_id = 6 THEN 1 ELSE 0 END) AS [Tools & Home Improvement],
+    SUM( CASE WHEN i_category_id = 7 THEN 1 ELSE 0 END) AS [Toys & Games],
+    SUM( CASE WHEN i_category_id = 8 THEN 1 ELSE 0 END) AS [Movies & TV],
+    SUM( CASE WHEN i_category_id = 9 THEN 1 ELSE 0 END) AS [Sports & Outdoors]
+  FROM [dbo].[web_clickstreams_dp]
+  INNER JOIN item it ON (wcs_item_sk = i_item_sk
+                        AND wcs_user_sk IS NOT NULL)
+GROUP BY  wcs_user_sk;
+GO
+
+DROP EXTERNAL TABLE [dbo].[web_clickstreams_dp];
+GO