diff --git a/samples/features/sql-big-data-cluster/README.md b/samples/features/sql-big-data-cluster/README.md index a370a240..0b949e01 100644 --- a/samples/features/sql-big-data-cluster/README.md +++ b/samples/features/sql-big-data-cluster/README.md @@ -11,33 +11,46 @@ Installation instructions for SQL Server 2019 big data cluster can be found [her ## Samples Setup -**Before you begin**, download the sample database [backup file](https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak) and save it locally. Run the CMD script called *bootstrap-sample-db.cmd* or the shell script *bootstrap-sample-db.sh* depending on your platform. This script will restore the database on the SQL Master instance, execute the *bootstrap-sample-db.sql* script, create the database objects needed, export the web_clickstreams & inventory tables to CSV file, and upload the web_clickstreams CSV file to HDFS inside the SQL Server 2019 big data cluster. +**Before you begin**, run the CMD script called [bootstrap-sample-db.cmd](bootstrap-sample-db.cmd) or the shell script [bootstrap-sample-db.sh](bootstrap-sample-db.sh) depending on your platform. This script does the following operations: + +1. Downloads the tpcx-bb 1GB sample database +1. Restores the database on the SQL Master instance +1. Executes the bootstrap-sample-db.SQL script +1. Exports the web_clickstreams, inventory, customer & product_reviews tables to files +1. Uploads the web_clickstreams CSV file to the HDFS inside the SQL Server 2019 big data cluster __[data-pool](data-pool/)__ +SQL Server 2019 big data cluster contains a data pool which consists of many SQL Server instances to store data & query in a scale-out manner. + ### Data ingestion using Spark -Connect to the master instance in your SQL Server big data cluster and the SQL Server big data cluster endpoint, and follow the steps in *data-pool/data-ingestion-spark.sql*. +The sample script [data-pool/data-ingestion-spark.sql](data-pool/data-ingestion-spark.sql) shows how to perform data ingestion from Spark into data pool table(s). ### Data ingestion using sql -Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-pool/data-ingestion-sql.sql*. +The sample script [data-pool/data-ingestion-sql.sql](data-pool/data-ingestion-sql.sql) shows how to perform data ingestion from T-SQL into data pool table(s). __[data-virtualization](data-virtualization/)__ -### External table over HDFS -Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-hdfs.sql*. +SQL Server 2019 or SQL Server 2019 big data cluster can use PolyBase external tables to connect to other data sources. + +### External table over Storage Pool +SQL Server 2019 big data cluster contains a storage pool consisting of HDFS, Spark and SQL Server instances. The [data-virtualization/storage-pool](data-virtualization/storage-pool) folder contains samples that demonstrate how to query data in HDFS inside SQL Server 2019 big data cluster. ### External table over Oracle -To execute this sample script, you will need following: -1. Oracle instance and credentials -1. Create inventory table in Oracle using [data-virtualization/inventory-oracle.sql](data-virtualization/inventory-oracle.sql/) script -1. Import the inventory.csv file generated by the bootstrap-sample-db script to a table in Oracle +SQL Server 2019 uses new ODBC connectors to enable connectivity to SQL Server, Oracle, Teradata, MongoDB and generic ODBC data sources. -Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-oracle.sql*. +The [data-virtualization/oracle](data-virtualization/oracle) folder contains samples that demonstrate how to query data in Oracle using external tables. + +__[deployment](deployment/)__ + +The [deployment](deployment) folder contains the scripts for deploying a Kubernetes cluster for SQL Server 2019 big data cluster. __[machine-learning](machine-learning/)__ -### SQL Server ML Services on master instance -Connect to the master instance in your SQL Server big data cluster and execute the steps in *machine-learning/sql/book-category-r-ml.sql*. +SQL Server 2016 added support executing R scripts from T-SQL. SQL Server 2017 added support for executing Python scripts from T-SQL. SQL Server 2019 adds support for executing Java code from T-SQL. SQL Server 2019 big data cluster adds support for executing Spark code inside the big data cluster. -### Spark ML -Connect to the SQL Server big data cluster endpoint, and run the notebook files *machine-learning/spark/1-data-prep.ipynb* and *machine-learning/spark/2-build-ml-model.ipynb* cell by cell. +### SQL Server Machine Learning Services +The [machine-learning\sql](machine-learning\sql) folder contains the sample SQL scripts that show how to invoke R, Python, and Java code from T-SQL. + +### Spark Machine Learning +The [machine-learning\spark](machine-learning\spark) folder contains the Spark samples. diff --git a/samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd b/samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd index ff589d1d..ffd01897 100644 --- a/samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd +++ b/samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd @@ -1,17 +1,18 @@ @echo off REM bootstrap sample database CMD script setlocal enableextensions +setlocal enabledelayedexpansion set CLUSTER_NAMESPACE=%1 set SQL_MASTER_IP=%2 set SQL_MASTER_SA_PASSWORD=%3 -set KNOX_IP=%3 -set KNOX_PASSWORD=%4 +set KNOX_IP=%4 +set KNOX_PASSWORD=%5 set STARTUP_PATH=%~dp0 +set TMP_DIR_NAME=%~nx0 if NOT DEFINED CLUSTER_NAMESPACE goto :usage if NOT DEFINED SQL_MASTER_IP goto :usage if NOT DEFINED SQL_MASTER_SA_PASSWORD goto :usage -if NOT DEFINED BACKUP_FILE_PATH goto :usage if NOT DEFINED KNOX_IP goto :usage if NOT DEFINED KNOX_PASSWORD set KNOX_PASSWORD=%SQL_MASTER_SA_PASSWORD% @@ -23,49 +24,52 @@ for %%F in (sqlcmd.exe bcp.exe kubectl.exe curl.exe) do ( ) pushd "%tmp%" +md %TMP_DIR_NAME% echo Downloading sample database backup file... -curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak +%DEBUG% curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak REM Copy the backup file, restore the database, create necessary objects and data file echo Copying database backup file... %DEBUG% kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit del tpcxbb_1gb.bak >NUL -popd echo Configuring sample database... -%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "%STARTUP_PATH%bootstrap.out" -I -b || goto exit +%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || goto exit for %%F in (web_clickstreams inventory customer) do ( echo Exporting %%F data... if /i %%F EQU web_clickstreams (set DELIMITER=,) else (SET DELIMITER=^|) - %DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"%DELIMITER%" -o "%STARTUP_PATH%%%F.out" -e "%STARTUP_PATH%%%F.err" || goto exit + %DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"!DELIMITER!" -o "%%F.out" -e "%%F.err" || goto exit ) echo Exporting product_reviews data... -%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "%STARTUP_PATH%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "%STARTUP_PATH%product_reviews.out" -e "%STARTUP_PATH%product_reviews.err" || goto exit +%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "%TMP_DIR_NAME%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit REM Copy the data file to HDFS -pushd "%STARTUP_PATH%" echo Uploading web_clickstreams data to HDFS... %DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || goto exit -%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit +%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit +echo. echo Uploading product_reviews data to HDFS... %DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || goto exit -%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit -:: del /q *.out *.err *.csv -popd +%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit +%DEBUG% del /q *.out *.err *.csv + +popd +%DEBUG% rd /q "%tmp%\%TMP_DIR_NAME%" endlocal exit /b 0 goto :eof :exit echo Bootstrap of the sample database failed. + echo Output and error files are in directory [%TMP%\%TMP_DIR_NAME%]. exit /b 1 :usage - echo USAGE: %0 ^ ^ ^ ^ ^ [^] + echo USAGE: %0 ^ ^ ^ ^ [^] echo Default ports are assumed for SQL Master instance ^& Knox gateway. exit /b 0 \ No newline at end of file diff --git a/samples/features/sql-big-data-cluster/bootstrap-sample-db.sh b/samples/features/sql-big-data-cluster/bootstrap-sample-db.sh index 33cf879c..b88d7794 100644 --- a/samples/features/sql-big-data-cluster/bootstrap-sample-db.sh +++ b/samples/features/sql-big-data-cluster/bootstrap-sample-db.sh @@ -1,8 +1,10 @@ #!/bin/bash set -e set -o pipefail +STARTUP_PATH=$(dirname $0) +TMP_DIR_NAME=$(basename $0) USAGE_MESSAGE="USAGE: $0 []" -ERROR_MESSAGE="Bootstrap of the sample database failed." +ERROR_MESSAGE="Bootstrap of the sample database failed. Output and error files are in directory [/tmp/$TMP_DIR_NAME]." # Print usage if mandatory parameters are missing : "${1:?$USAGE_MESSAGE}" @@ -26,41 +28,51 @@ KNOX_ENDPOINT=$KNOX_IP:30443 for util in sqlcmd.exe bcp.exe kubectl.exe curl.exe do - echo Verifying $util is in path & which $util 1>NUL 2>NUL || (echo Unable to locate $util && exit 1) + echo Verifying $util is in path & which $util 1>/dev/nul 2>/dev/nul || (echo Unable to locate $util && exit 1) done # Copy the backup file, restore the database, create necessary objects and data file pushd "/tmp" +$DEBUG mkdir "$TMP_DIR_NAME" echo Downloading sample database backup file... $DEBUG curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak echo Copying database backup file... $DEBUG kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1) -rm tpcxbb_1gb.bak -popd +$DEBUG rm tpcxbb_1gb.bak echo Configuring sample database... # WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/SQLCMD.EXE" -$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2) +$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "$STARTUP_PATH\bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2) -for table in web_clickstreams inventory +for table in web_clickstreams inventory customer do echo Exporting $table data... + if [ $table == web_clickstreams ] + then + DELIMITER=, + else + DELIMITER="|" + fi # WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/bcp.exe" - $DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "$table.err" || (echo $ERROR_MESSAGE && exit 3) + $DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t"$DELIMITER" -e "$table.err" || (echo $ERROR_MESSAGE && exit 3) done echo Exporting product_reviews data... -$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3) +$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3) # Copy the data file to HDFS echo Uploading web_clickstreams data to HDFS... $DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 4) -$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5) +$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5) +echo echo Uploading product_reviews data to HDFS... $DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 6) -$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7) +$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7) -# rm -f *.out *.err *.csv +$DEBUG rm -f *.out *.err *.csv +popd + +$DEBUG rmdir "/tmp/$TMP_DIR_NAME" exit \ No newline at end of file diff --git a/samples/features/sql-big-data-cluster/bootstrap-sample-db.sql b/samples/features/sql-big-data-cluster/bootstrap-sample-db.sql index b792e82c..8963d777 100644 --- a/samples/features/sql-big-data-cluster/bootstrap-sample-db.sql +++ b/samples/features/sql-big-data-cluster/bootstrap-sample-db.sql @@ -25,11 +25,14 @@ IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePo WITH (LOCATION = 'sqlhdfs://service-mssql-controller:8080'); GO --- Create view used for ML services training stored procedure -CREATE OR ALTER VIEW [dbo].[web_clickstreams_book_clicks] +-- Create view used for ML services training and scoring stored procedures +CREATE OR ALTER VIEW [dbo].[web_clickstreams_book_clicks] AS SELECT - q.clicks_in_category, + /* There is a bug in TPCx-BB data generator which results in data where all users have purchased books. + As a result, we cannot use the data as is for ML training purposes. So we will treat users with 1-5 clicks + in the book category as not interested in books. */ + CASE WHEN q.clicks_in_category < 6 THEN 0 ELSE q.clicks_in_category END AS clicks_in_category, CASE WHEN cd.cd_education_status IN ('Advanced Degree', 'College', '4 yr Degree', '2 yr Degree') THEN 1 ELSE 0 END AS college_education, CASE WHEN cd.cd_gender = 'M' THEN 1 ELSE 0 END AS male, COALESCE(cd.cd_credit_rating, 'Unknown') as cd_credit_rating, @@ -62,15 +65,4 @@ AS ) AS q INNER JOIN customer as c ON q.wcs_user_sk = c.c_customer_sk INNER JOIN customer_demographics as cd ON c.c_current_cdemo_sk = cd.cd_demo_sk; -GO - --- Create table for storing the machine learning models -IF NOT EXISTS(SELECT * FROM sys.tables WHERE name = 'sales_models') - CREATE TABLE sales_models ( - model_name varchar(100) NOT NULL PRIMARY KEY, - model varbinary(max) NOT NULL, - model_native varbinary(max) NULL, - created_by nvarchar(300) NOT NULL DEFAULT(SYSTEM_USER), - create_time datetime2 NOT NULL DEFAULT(SYSDATETIME()) - ); -GO +GO \ No newline at end of file diff --git a/samples/features/sql-big-data-cluster/data-virtualization/oracle/README.md b/samples/features/sql-big-data-cluster/data-virtualization/oracle/README.md index 0c54d140..0a473bb7 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/oracle/README.md +++ b/samples/features/sql-big-data-cluster/data-virtualization/oracle/README.md @@ -1,6 +1,6 @@ # Data virtualization in SQL Server 2019 -**Applies to: SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster** +***Applies to:*** SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server, Oracle, MongoDB and Teradata. These connectors can be used from stand-alone SQL Server 2019 on Windows or Linux or SQL Server 2019 big data cluster. diff --git a/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/README.md b/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/README.md index b15d7012..66688751 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/README.md +++ b/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/README.md @@ -1,11 +1,9 @@ # Oracle setup -This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019 big data cluster. +This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019+ or SQL Server 2019+ big data cluster. # Instructions -1. Connect to Oracle instance. +***Before you begin***, you need the Oracle instance name and credentials. -1. Execute the [sales-user.sql](sales-user.sql). This script creates the sample user. If there is name conflict please change the script user/credentials. - -1. Execute the [inventory.sql](inventory.sql). This script creates the inventory table. \ No newline at end of file +1. Execute the [bootstrap-oracle.cmd](bootstrap-oracle.cmd) to create the necessary objects in Oracle diff --git a/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/bootstrap-oracle.cmd b/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/bootstrap-oracle.cmd index 4887ee43..f3231982 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/bootstrap-oracle.cmd +++ b/samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/bootstrap-oracle.cmd @@ -9,19 +9,21 @@ if NOT DEFINED ORACLE_SERVER goto :usage if NOT DEFINED ORACLE_USER goto :usage if NOT DEFINED ORACLE_PASSWORD goto :usage -echo Verifying sqlplus.exe is in path & CALL WHERE /Q sqlplus.exe || GOTO exit -echo Verifying sqlldr.exe is in path & CALL WHERE /Q sqlldr.exe || GOTO exit +for %F in (sqlplus.exe sqlldr.exe) do ( + echo Verifying %%F is in path & CALL WHERE /Q %%F || GOTO exit +) -echo Creating user & tables... -echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit -echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @inventory.sql || GOTO exit -echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @customer.sql || GOTO exit -echo Loading tables data... -sqlldr CONTROL=inventory.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit -sqlldr CONTROL=customer.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit +for %%F in (sales-user.sql inventory.sql customer.sql) do ( + echo Executing [%%F]... + echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit +) + +for %%F in (inventory.ctl customer.ctl) do ( + echo Loading [%%F]... + sqlldr CONTROL=%%F userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit +) -:: del /q *.out *.err *.csv endlocal exit /b 0 goto :eof diff --git a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/README.md b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/README.md index ab41027a..29c9971a 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/README.md +++ b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/README.md @@ -1,10 +1,10 @@ # Data virtualization in SQL Server 2019 big data cluster -In SQL Server 2019 big data clusters, the SQL Server engine has gained the ability to natively read HDFS files, such as CSV and parquet files, by using SQL Server instances collocated on each of the HDFS data nodes to filter and aggregate data locally in parallel across all of the HDFS data nodes. SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server, Oracle, MongoDB and Teradata. +In SQL Server 2019 big data clusters, the SQL Server engine has gained the ability to natively read HDFS files, such as CSV and parquet files, by using SQL Server instances collocated on each of the HDFS data nodes to filter and aggregate data locally in parallel across all of the HDFS data nodes. ## Query data in HDFS from SQL Server master -**Applies to: SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2019 big data cluster In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node with SQL Server & Spark endpoints. In this example, you are going to create an external table in the SQL Server Master instance that points to data in HDFS within the SQL Server Big data cluster. You will then join the data in the external table with high value data in SQL Master instance. @@ -21,3 +21,5 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node 1. Execute the [web-clickstreams-parquet.sql](web-clickstreams-hdfs-parquet.sql). This script demonstrates how to read parquet file(s) stored in HDFS. 1. Execute the [product-reviews-hdfs-csv.sql](product-reviews-hdfs-csv.sql). This script demonstrates how to read CSV file(s) stored in HDFS. + +1. Execute the [product-reviews-hdfs-parquet.sql](product-reviews-hdfs-parquet.sql). This script demonstrates how to read parquet file(s) stored in HDFS. diff --git a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-csv.sql b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-csv.sql index 0df6094f..5221de48 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-csv.sql +++ b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-csv.sql @@ -19,14 +19,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file') -- is a special data source that is available in any new database in -- SQL Master instance. -- -CREATE EXTERNAL TABLE [product_reviews_hdfs_csv] -("pr_review_sk" BIGINT , "pr_review_content" varchar(8000)) -WITH -( - DATA_SOURCE = SqlStoragePool, - LOCATION = '/product_review_data', - FILE_FORMAT = csv_file -); +IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'product_reviews_hdfs_csv') + CREATE EXTERNAL TABLE [product_reviews_hdfs_csv] + ("pr_review_sk" BIGINT , "pr_review_content" varchar(8000)) + WITH + ( + DATA_SOURCE = SqlStoragePool, + LOCATION = '/product_review_data', + FILE_FORMAT = csv_file + ); GO -- Join external table with local tables @@ -38,5 +39,8 @@ SELECT ON pc.pr_review_sk = p.pr_review_sk; GO +-- Cleanup +/* DROP EXTERNAL TABLE [dbo].[product_reviews_hdfs_csv]; -GO \ No newline at end of file +GO +*/ diff --git a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-parquet.sql b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-parquet.sql index 9c5595e2..1d2e318f 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-parquet.sql +++ b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-parquet.sql @@ -15,14 +15,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file -- is a special data source that is available in any new database in -- SQL Master instance. -- -CREATE EXTERNAL TABLE [product_reviews_hdfs_parquet] -("pr_review_sk" BIGINT , "pr_review_content" varchar(8000)) -WITH -( - DATA_SOURCE = SqlStoragePool, - LOCATION = '/user/hive/warehouse/product_review_data', - FILE_FORMAT = parquet_file -); +IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'product_reviews_hdfs_parquet') + CREATE EXTERNAL TABLE [product_reviews_hdfs_parquet] + ("pr_review_sk" BIGINT , "pr_review_content" varchar(8000)) + WITH + ( + DATA_SOURCE = SqlStoragePool, + LOCATION = '/user/hive/warehouse/product_reviews', + FILE_FORMAT = parquet_file + ); GO -- Join external table with local tables @@ -34,5 +35,8 @@ SELECT ON pc.pr_review_sk = p.pr_review_sk; GO +-- Cleanup +/* DROP EXTERNAL TABLE [dbo].[product_reviews_hdfs_parquet]; -GO \ No newline at end of file +GO +*/ diff --git a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-csv.sql b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-csv.sql index 09da5050..a36319f2 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-csv.sql +++ b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-csv.sql @@ -18,14 +18,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file') -- is a special data source that is available in any new database in -- SQL Master instance. -- -CREATE EXTERNAL TABLE [web_clickstreams_hdfs_csv] -("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT) -WITH -( - DATA_SOURCE = SqlStoragePool, - LOCATION = '/clickstream_data', - FILE_FORMAT = csv_file -); +IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_hdfs_csv') + CREATE EXTERNAL TABLE [web_clickstreams_hdfs_csv] + ("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT) + WITH + ( + DATA_SOURCE = SqlStoragePool, + LOCATION = '/clickstream_data', + FILE_FORMAT = csv_file + ); GO -- Join external table with local tables @@ -48,5 +49,8 @@ SELECT GROUP BY wcs_user_sk; GO +-- Cleanup +/* DROP EXTERNAL TABLE [dbo].[web_clickstreams_hdfs_csv]; -GO \ No newline at end of file +GO +*/ diff --git a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-parquet.sql b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-parquet.sql index 73d67ee6..ec9ee8a6 100644 --- a/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-parquet.sql +++ b/samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-parquet.sql @@ -14,14 +14,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file -- is a special data source that is available in any new database in -- SQL Master instance. -- -CREATE EXTERNAL TABLE [web_clickstreams_hdfs_parquet] -("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT) -WITH -( - DATA_SOURCE = SqlStoragePool, - LOCATION = '/user/hive/warehouse/web_clickstreams', - FILE_FORMAT = parquet_file -); +IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_hdfs_parquet') + CREATE EXTERNAL TABLE [web_clickstreams_hdfs_parquet] + ("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT) + WITH + ( + DATA_SOURCE = SqlStoragePool, + LOCATION = '/user/hive/warehouse/web_clickstreams', + FILE_FORMAT = parquet_file + ); GO -- Join external table with local tables @@ -44,5 +45,8 @@ SELECT GROUP BY wcs_user_sk; GO +-- Cleanup +/* DROP EXTERNAL TABLE [dbo].[web_clickstreams_hdfs_parquet]; -GO \ No newline at end of file +GO +*/ diff --git a/samples/features/sql-big-data-cluster/machine-learning/README.md b/samples/features/sql-big-data-cluster/machine-learning/README.md index d34971cd..6e9c7b6f 100644 --- a/samples/features/sql-big-data-cluster/machine-learning/README.md +++ b/samples/features/sql-big-data-cluster/machine-learning/README.md @@ -14,6 +14,7 @@ The new built-in notebooks in Azure Data Studio enables data scientists and data This sample builds a machine learning model using AdultCensusIncome.csv available [here](https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv). +[spark](spark) ### Instructions diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/README.md b/samples/features/sql-big-data-cluster/machine-learning/sql/README.md index 4aa5c22b..d36e1e27 100644 --- a/samples/features/sql-big-data-cluster/machine-learning/sql/README.md +++ b/samples/features/sql-big-data-cluster/machine-learning/sql/README.md @@ -1,8 +1,8 @@ # SQL Server Machine Learning Services -SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL. +SQL Server 2016 added support for running R scripts from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL. -**Applies to: SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster [r](r) diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/python/README.md b/samples/features/sql-big-data-cluster/machine-learning/sql/python/README.md index f3030554..25b6af05 100644 --- a/samples/features/sql-big-data-cluster/machine-learning/sql/python/README.md +++ b/samples/features/sql-big-data-cluster/machine-learning/sql/python/README.md @@ -4,13 +4,13 @@ SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 add [book-click-prediction-py.sql](book-click-prediction-py.sql/) -**Applies to: SQL Server 2017+, SQL Server 2019, SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2017+, SQL Server 2019, SQL Server 2019 big data cluster In this example, we are building a machine learning model using Python and a logistic regression algorithm for a recommendation engine on an online store. Based on existing users' click pattern online and their interest in other categories and demographics, we are training a machine learning model. This model will then be used to predict if the visitor is interested in a given item category using the T-SQL PREDICT function. [book-click-prediction-partitioned-py.sql](book-click-prediction-partitioned-py.sql/) -**Applies to: SQL Server 2019, SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2019, SQL Server 2019 big data cluster In this example, we are leveraging the new partitioning support (SQL Server 2019) in sp_execute_external_script to partition the input data and run the Python script per partition. So we will modify the training script to train model per group of users based on credit rating. The Python script will produce N models for the same input data set. diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-partitioned-py.sql b/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-partitioned-py.sql index b59ebd6d..e9fbea7f 100644 Binary files a/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-partitioned-py.sql and b/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-partitioned-py.sql differ diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-py.sql b/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-py.sql index 3d1f6b74..723820fb 100644 Binary files a/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-py.sql and b/samples/features/sql-big-data-cluster/machine-learning/sql/python/book-click-prediction-py.sql differ diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/r/README.md b/samples/features/sql-big-data-cluster/machine-learning/sql/r/README.md index 09c9ba03..d0cce408 100644 --- a/samples/features/sql-big-data-cluster/machine-learning/sql/r/README.md +++ b/samples/features/sql-big-data-cluster/machine-learning/sql/r/README.md @@ -1,16 +1,16 @@ # SQL Server Machine Learning Services -SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL. +SQL Server 2016 added support for running R scripts from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL. [book-click-prediction-r.sql](book-click-prediction-r.sql/) -**Applies to: SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster In this example, we are building a machine learning model using R and a logistic regression algorithm for a recommendation engine on an online store. Based on existing users' click pattern online and their interest in other categories and demographics, we are training a machine learning model. This model will then be used to predict if the visitor is interested in a given item category using the T-SQL PREDICT function. [book-click-prediction-partitioned-r.sql](book-click-prediction-partitioned-r.sql/) -**Applies to: SQL Server 2019, SQL Server 2019 big data cluster** +**Applies to:** SQL Server 2019, SQL Server 2019 big data cluster In this example, we are leveraging the new partitioning support (SQL Server 2019) in sp_execute_external_script to partition the input data and run the R script per partition. So we will modify the training script to train model per group of users based on credit rating. The R script will produce N models for the same input data set. diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-partitioned-r.sql b/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-partitioned-r.sql index 65b247ac..d71d3746 100644 Binary files a/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-partitioned-r.sql and b/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-partitioned-r.sql differ diff --git a/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-r.sql b/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-r.sql index 298438da..a1f015cb 100644 Binary files a/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-r.sql and b/samples/features/sql-big-data-cluster/machine-learning/sql/r/book-click-prediction-r.sql differ diff --git a/samples/features/sql-big-data-cluster/spark/spark-sql.ipynb b/samples/features/sql-big-data-cluster/spark/spark-sql.ipynb index cfc9422d..1a87b8c9 100644 --- a/samples/features/sql-big-data-cluster/spark/spark-sql.ipynb +++ b/samples/features/sql-big-data-cluster/spark/spark-sql.ipynb @@ -39,7 +39,7 @@ }, { "cell_type": "code", - "source": "# Disable saving SUCCESS file\r\nsc._jsc.hadoopConfiguration().set(\"mapreduce.fileoutputcommitter.marksuccessfuljobs\", \"false\") \r\n\r\n# Print the current warehouse directory\r\nprint(spark.conf.get(\"spark.sql.warehouse.dir\"))\r\n\r\n# Save results as parquet file and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"web_clickstreams\")\r\n", + "source": "# Disable saving SUCCESS file\r\nsc._jsc.hadoopConfiguration().set(\"mapreduce.fileoutputcommitter.marksuccessfuljobs\", \"false\") \r\n\r\n# Print the current warehouse directory where the parquet files will be stored\r\nprint(spark.conf.get(\"spark.sql.warehouse.dir\"))\r\n\r\n# Save results as parquet file and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"web_clickstreams\")\r\n", "metadata": { "language": "python" }, @@ -61,7 +61,7 @@ "outputs": [ { "name": "stdout", - "text": "+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 38250| 4172| 67067| 11504| 40| 16819|\n| 38251| 28919| 67090| 13782| 40| 11283|\n| 38251| 77021| 67096| 4330| 40| 60107|\n| 38251| 29023| 67109| 15796| 40| 31730|\n| 38251| 54047| 67110| 9739| 40| 50449|\n| 38251| 85733| 67117| 6843| 40| 39327|\n| 38252| 53176| 67141| 12525| 40| 37913|\n| 38252| 15873| 67153| 13008| 40| 92546|\n| 38252| 39147| 67167| 5208| 40| 74534|\n| 38252| 79540| 67171| 11552| 40| 94065|\n| 38252| 35200| 67175| 9622| 40| 80502|\n| 38253| 26068| 67191| 8585| 40| 43314|\n| 38253| 63065| 67195| 17486| 40| 63793|\n| 38253| 9687| 67214| 9856| 40| 92780|\n| 38253| 18373| 67219| 406| 40| 38319|\n| 38254| 80201| 67229| 13610| 40| 15342|\n| 38254| 40058| 67239| 13594| 40| 41879|\n| 38254| 79136| 67243| 1933| 40| 42095|\n| 38254| 14684| 67244| 14267| 40| 39119|\n| 38254| 36369| 67248| 641| 40| 82237|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows\n\n+-----------+--------+\n|wcs_user_sk|count(1)|\n+-----------+--------+\n| 65042| 832|\n| 55928| 821|\n| 15570| 791|\n| 31138| 788|\n| 68188| 784|\n| 88205| 760|\n| 15678| 757|\n| 48063| 741|\n| 77518| 741|\n| 92978| 728|\n| 82129| 727|\n| 21700| 725|\n| 69707| 724|\n| 38895| 719|\n| 97643| 716|\n| 74426| 707|\n| 7813| 704|\n| 49528| 700|\n| 55766| 698|\n| 54355| 697|\n+-----------+--------+\nonly showing top 20 rows", + "text": "+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 36890| 40052| null| 4379| 34| null|\n| 36890| 41285| null| 6245| 34| null|\n| 36890| 23115| null| 13852| 34| null|\n| 36890| 17702| null| 15975| 34| null|\n| 36890| 62676| null| 2119| 34| null|\n| 36890| 34267| null| 10273| 34| null|\n| 36890| 8502| null| 17790| 34| null|\n| 36890| 54340| null| 3453| 34| null|\n| 36890| 54370| null| 6372| 34| null|\n| 36890| 6578| null| 17203| 34| null|\n| 36890| 75088| null| 4891| 34| null|\n| 36890| 23922| null| 11332| 34| null|\n| 36890| 28761| null| 4484| 34| null|\n| 36890| 21444| null| 5582| 34| null|\n| 36890| 58917| null| 8833| 34| null|\n| 36890| 27578| null| 8599| 34| null|\n| 36890| 8059| null| 6720| 34| null|\n| 36890| 43008| null| 17175| 34| null|\n| 36890| 4378| null| 10644| 34| null|\n| 36890| 55403| null| 8139| 34| null|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows\n\n+-----------+--------+\n|wcs_user_sk|count(1)|\n+-----------+--------+\n| 65042| 832|\n| 55928| 821|\n| 15570| 791|\n| 31138| 788|\n| 68188| 784|\n| 88205| 760|\n| 15678| 757|\n| 48063| 741|\n| 77518| 741|\n| 92978| 728|\n| 82129| 727|\n| 21700| 725|\n| 69707| 724|\n| 38895| 719|\n| 97643| 716|\n| 74426| 707|\n| 7813| 704|\n| 49528| 700|\n| 55766| 698|\n| 54355| 697|\n+-----------+--------+\nonly showing top 20 rows", "output_type": "stream" } ],