mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Bug fixes. Refactored of scripts & README content.
This commit is contained in:
Umachandar Jayachandran
@@ -11,33 +11,46 @@ Installation instructions for SQL Server 2019 big data cluster can be found [her
|
||||
|
||||
## Samples Setup
|
||||
|
||||
**Before you begin**, download the sample database [backup file](https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak) and save it locally. Run the CMD script called *bootstrap-sample-db.cmd* or the shell script *bootstrap-sample-db.sh* depending on your platform. This script will restore the database on the SQL Master instance, execute the *bootstrap-sample-db.sql* script, create the database objects needed, export the web_clickstreams & inventory tables to CSV file, and upload the web_clickstreams CSV file to HDFS inside the SQL Server 2019 big data cluster.
|
||||
**Before you begin**, run the CMD script called [bootstrap-sample-db.cmd](bootstrap-sample-db.cmd) or the shell script [bootstrap-sample-db.sh](bootstrap-sample-db.sh) depending on your platform. This script does the following operations:
|
||||
|
||||
1. Downloads the tpcx-bb 1GB sample database
|
||||
1. Restores the database on the SQL Master instance
|
||||
1. Executes the bootstrap-sample-db.SQL script
|
||||
1. Exports the web_clickstreams, inventory, customer & product_reviews tables to files
|
||||
1. Uploads the web_clickstreams CSV file to the HDFS inside the SQL Server 2019 big data cluster
|
||||
|
||||
__[data-pool](data-pool/)__
|
||||
|
||||
SQL Server 2019 big data cluster contains a data pool which consists of many SQL Server instances to store data & query in a scale-out manner.
|
||||
|
||||
### Data ingestion using Spark
|
||||
Connect to the master instance in your SQL Server big data cluster and the SQL Server big data cluster endpoint, and follow the steps in *data-pool/data-ingestion-spark.sql*.
|
||||
The sample script [data-pool/data-ingestion-spark.sql](data-pool/data-ingestion-spark.sql) shows how to perform data ingestion from Spark into data pool table(s).
|
||||
|
||||
### Data ingestion using sql
|
||||
Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-pool/data-ingestion-sql.sql*.
|
||||
The sample script [data-pool/data-ingestion-sql.sql](data-pool/data-ingestion-sql.sql) shows how to perform data ingestion from T-SQL into data pool table(s).
|
||||
|
||||
__[data-virtualization](data-virtualization/)__
|
||||
|
||||
### External table over HDFS
|
||||
Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-hdfs.sql*.
|
||||
SQL Server 2019 or SQL Server 2019 big data cluster can use PolyBase external tables to connect to other data sources.
|
||||
|
||||
### External table over Storage Pool
|
||||
SQL Server 2019 big data cluster contains a storage pool consisting of HDFS, Spark and SQL Server instances. The [data-virtualization/storage-pool](data-virtualization/storage-pool) folder contains samples that demonstrate how to query data in HDFS inside SQL Server 2019 big data cluster.
|
||||
|
||||
### External table over Oracle
|
||||
To execute this sample script, you will need following:
|
||||
1. Oracle instance and credentials
|
||||
1. Create inventory table in Oracle using [data-virtualization/inventory-oracle.sql](data-virtualization/inventory-oracle.sql/) script
|
||||
1. Import the inventory.csv file generated by the bootstrap-sample-db script to a table in Oracle
|
||||
SQL Server 2019 uses new ODBC connectors to enable connectivity to SQL Server, Oracle, Teradata, MongoDB and generic ODBC data sources.
|
||||
|
||||
Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-oracle.sql*.
|
||||
The [data-virtualization/oracle](data-virtualization/oracle) folder contains samples that demonstrate how to query data in Oracle using external tables.
|
||||
|
||||
__[deployment](deployment/)__
|
||||
|
||||
The [deployment](deployment) folder contains the scripts for deploying a Kubernetes cluster for SQL Server 2019 big data cluster.
|
||||
|
||||
__[machine-learning](machine-learning/)__
|
||||
|
||||
### SQL Server ML Services on master instance
|
||||
Connect to the master instance in your SQL Server big data cluster and execute the steps in *machine-learning/sql/book-category-r-ml.sql*.
|
||||
SQL Server 2016 added support executing R scripts from T-SQL. SQL Server 2017 added support for executing Python scripts from T-SQL. SQL Server 2019 adds support for executing Java code from T-SQL. SQL Server 2019 big data cluster adds support for executing Spark code inside the big data cluster.
|
||||
|
||||
### Spark ML
|
||||
Connect to the SQL Server big data cluster endpoint, and run the notebook files *machine-learning/spark/1-data-prep.ipynb* and *machine-learning/spark/2-build-ml-model.ipynb* cell by cell.
|
||||
### SQL Server Machine Learning Services
|
||||
The [machine-learning\sql](machine-learning\sql) folder contains the sample SQL scripts that show how to invoke R, Python, and Java code from T-SQL.
|
||||
|
||||
### Spark Machine Learning
|
||||
The [machine-learning\spark](machine-learning\spark) folder contains the Spark samples.
|
||||
|
||||
@@ -1,17 +1,18 @@
|
||||
@echo off
|
||||
REM bootstrap sample database CMD script
|
||||
setlocal enableextensions
|
||||
setlocal enabledelayedexpansion
|
||||
set CLUSTER_NAMESPACE=%1
|
||||
set SQL_MASTER_IP=%2
|
||||
set SQL_MASTER_SA_PASSWORD=%3
|
||||
set KNOX_IP=%3
|
||||
set KNOX_PASSWORD=%4
|
||||
set KNOX_IP=%4
|
||||
set KNOX_PASSWORD=%5
|
||||
set STARTUP_PATH=%~dp0
|
||||
set TMP_DIR_NAME=%~nx0
|
||||
|
||||
if NOT DEFINED CLUSTER_NAMESPACE goto :usage
|
||||
if NOT DEFINED SQL_MASTER_IP goto :usage
|
||||
if NOT DEFINED SQL_MASTER_SA_PASSWORD goto :usage
|
||||
if NOT DEFINED BACKUP_FILE_PATH goto :usage
|
||||
if NOT DEFINED KNOX_IP goto :usage
|
||||
if NOT DEFINED KNOX_PASSWORD set KNOX_PASSWORD=%SQL_MASTER_SA_PASSWORD%
|
||||
|
||||
@@ -23,49 +24,52 @@ for %%F in (sqlcmd.exe bcp.exe kubectl.exe curl.exe) do (
|
||||
)
|
||||
|
||||
pushd "%tmp%"
|
||||
md %TMP_DIR_NAME%
|
||||
echo Downloading sample database backup file...
|
||||
curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
|
||||
%DEBUG% curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
|
||||
|
||||
REM Copy the backup file, restore the database, create necessary objects and data file
|
||||
echo Copying database backup file...
|
||||
%DEBUG% kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
|
||||
del tpcxbb_1gb.bak >NUL
|
||||
popd
|
||||
|
||||
echo Configuring sample database...
|
||||
%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "%STARTUP_PATH%bootstrap.out" -I -b || goto exit
|
||||
%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || goto exit
|
||||
|
||||
for %%F in (web_clickstreams inventory customer) do (
|
||||
echo Exporting %%F data...
|
||||
if /i %%F EQU web_clickstreams (set DELIMITER=,) else (SET DELIMITER=^|)
|
||||
%DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"%DELIMITER%" -o "%STARTUP_PATH%%%F.out" -e "%STARTUP_PATH%%%F.err" || goto exit
|
||||
%DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"!DELIMITER!" -o "%%F.out" -e "%%F.err" || goto exit
|
||||
)
|
||||
|
||||
echo Exporting product_reviews data...
|
||||
%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "%STARTUP_PATH%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "%STARTUP_PATH%product_reviews.out" -e "%STARTUP_PATH%product_reviews.err" || goto exit
|
||||
%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "%TMP_DIR_NAME%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit
|
||||
|
||||
REM Copy the data file to HDFS
|
||||
pushd "%STARTUP_PATH%"
|
||||
echo Uploading web_clickstreams data to HDFS...
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
|
||||
|
||||
echo.
|
||||
echo Uploading product_reviews data to HDFS...
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
|
||||
:: del /q *.out *.err *.csv
|
||||
popd
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
|
||||
|
||||
%DEBUG% del /q *.out *.err *.csv
|
||||
|
||||
popd
|
||||
%DEBUG% rd /q "%tmp%\%TMP_DIR_NAME%"
|
||||
endlocal
|
||||
exit /b 0
|
||||
goto :eof
|
||||
|
||||
:exit
|
||||
echo Bootstrap of the sample database failed.
|
||||
echo Output and error files are in directory [%TMP%\%TMP_DIR_NAME%].
|
||||
exit /b 1
|
||||
|
||||
:usage
|
||||
echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<BACKUP_FILE_PATH^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>]
|
||||
echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>]
|
||||
echo Default ports are assumed for SQL Master instance ^& Knox gateway.
|
||||
exit /b 0
|
||||
@@ -1,8 +1,10 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
set -o pipefail
|
||||
STARTUP_PATH=$(dirname $0)
|
||||
TMP_DIR_NAME=$(basename $0)
|
||||
USAGE_MESSAGE="USAGE: $0 <CLUSTER_NAMESPACE> <SQL_MASTER_IP> <SQL_MASTER_SA_PASSWORD> <KNOX_IP> [<KNOX_PASSWORD>]"
|
||||
ERROR_MESSAGE="Bootstrap of the sample database failed."
|
||||
ERROR_MESSAGE="Bootstrap of the sample database failed. Output and error files are in directory [/tmp/$TMP_DIR_NAME]."
|
||||
|
||||
# Print usage if mandatory parameters are missing
|
||||
: "${1:?$USAGE_MESSAGE}"
|
||||
@@ -26,41 +28,51 @@ KNOX_ENDPOINT=$KNOX_IP:30443
|
||||
|
||||
for util in sqlcmd.exe bcp.exe kubectl.exe curl.exe
|
||||
do
|
||||
echo Verifying $util is in path & which $util 1>NUL 2>NUL || (echo Unable to locate $util && exit 1)
|
||||
echo Verifying $util is in path & which $util 1>/dev/nul 2>/dev/nul || (echo Unable to locate $util && exit 1)
|
||||
done
|
||||
|
||||
# Copy the backup file, restore the database, create necessary objects and data file
|
||||
pushd "/tmp"
|
||||
$DEBUG mkdir "$TMP_DIR_NAME"
|
||||
echo Downloading sample database backup file...
|
||||
$DEBUG curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
|
||||
|
||||
echo Copying database backup file...
|
||||
$DEBUG kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1)
|
||||
rm tpcxbb_1gb.bak
|
||||
popd
|
||||
$DEBUG rm tpcxbb_1gb.bak
|
||||
|
||||
echo Configuring sample database...
|
||||
# WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/SQLCMD.EXE"
|
||||
$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2)
|
||||
$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "$STARTUP_PATH\bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2)
|
||||
|
||||
for table in web_clickstreams inventory
|
||||
for table in web_clickstreams inventory customer
|
||||
do
|
||||
echo Exporting $table data...
|
||||
if [ $table == web_clickstreams ]
|
||||
then
|
||||
DELIMITER=,
|
||||
else
|
||||
DELIMITER="|"
|
||||
fi
|
||||
# WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/bcp.exe"
|
||||
$DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "$table.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
$DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t"$DELIMITER" -e "$table.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
done
|
||||
|
||||
echo Exporting product_reviews data...
|
||||
$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
|
||||
# Copy the data file to HDFS
|
||||
echo Uploading web_clickstreams data to HDFS...
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 4)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
|
||||
|
||||
echo
|
||||
echo Uploading product_reviews data to HDFS...
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 6)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
|
||||
|
||||
# rm -f *.out *.err *.csv
|
||||
$DEBUG rm -f *.out *.err *.csv
|
||||
popd
|
||||
|
||||
$DEBUG rmdir "/tmp/$TMP_DIR_NAME"
|
||||
exit
|
||||
@@ -25,11 +25,14 @@ IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePo
|
||||
WITH (LOCATION = 'sqlhdfs://service-mssql-controller:8080');
|
||||
GO
|
||||
|
||||
-- Create view used for ML services training stored procedure
|
||||
CREATE OR ALTER VIEW [dbo].[web_clickstreams_book_clicks]
|
||||
-- Create view used for ML services training and scoring stored procedures
|
||||
CREATE OR ALTER VIEW [dbo].[web_clickstreams_book_clicks]
|
||||
AS
|
||||
SELECT
|
||||
q.clicks_in_category,
|
||||
/* There is a bug in TPCx-BB data generator which results in data where all users have purchased books.
|
||||
As a result, we cannot use the data as is for ML training purposes. So we will treat users with 1-5 clicks
|
||||
in the book category as not interested in books. */
|
||||
CASE WHEN q.clicks_in_category < 6 THEN 0 ELSE q.clicks_in_category END AS clicks_in_category,
|
||||
CASE WHEN cd.cd_education_status IN ('Advanced Degree', 'College', '4 yr Degree', '2 yr Degree') THEN 1 ELSE 0 END AS college_education,
|
||||
CASE WHEN cd.cd_gender = 'M' THEN 1 ELSE 0 END AS male,
|
||||
COALESCE(cd.cd_credit_rating, 'Unknown') as cd_credit_rating,
|
||||
@@ -62,15 +65,4 @@ AS
|
||||
) AS q
|
||||
INNER JOIN customer as c ON q.wcs_user_sk = c.c_customer_sk
|
||||
INNER JOIN customer_demographics as cd ON c.c_current_cdemo_sk = cd.cd_demo_sk;
|
||||
GO
|
||||
|
||||
-- Create table for storing the machine learning models
|
||||
IF NOT EXISTS(SELECT * FROM sys.tables WHERE name = 'sales_models')
|
||||
CREATE TABLE sales_models (
|
||||
model_name varchar(100) NOT NULL PRIMARY KEY,
|
||||
model varbinary(max) NOT NULL,
|
||||
model_native varbinary(max) NULL,
|
||||
created_by nvarchar(300) NOT NULL DEFAULT(SYSTEM_USER),
|
||||
create_time datetime2 NOT NULL DEFAULT(SYSDATETIME())
|
||||
);
|
||||
GO
|
||||
GO
|
||||
@@ -1,6 +1,6 @@
|
||||
# Data virtualization in SQL Server 2019
|
||||
|
||||
**Applies to: SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster**
|
||||
***Applies to:*** SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster
|
||||
|
||||
SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server, Oracle, MongoDB and Teradata. These connectors can be used from stand-alone SQL Server 2019 on Windows or Linux or SQL Server 2019 big data cluster.
|
||||
|
||||
|
||||
@@ -1,11 +1,9 @@
|
||||
# Oracle setup
|
||||
|
||||
This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019 big data cluster.
|
||||
This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019+ or SQL Server 2019+ big data cluster.
|
||||
|
||||
# Instructions
|
||||
|
||||
1. Connect to Oracle instance.
|
||||
***Before you begin***, you need the Oracle instance name and credentials.
|
||||
|
||||
1. Execute the [sales-user.sql](sales-user.sql). This script creates the sample user. If there is name conflict please change the script user/credentials.
|
||||
|
||||
1. Execute the [inventory.sql](inventory.sql). This script creates the inventory table.
|
||||
1. Execute the [bootstrap-oracle.cmd](bootstrap-oracle.cmd) to create the necessary objects in Oracle
|
||||
|
||||
+12
-10
@@ -9,19 +9,21 @@ if NOT DEFINED ORACLE_SERVER goto :usage
|
||||
if NOT DEFINED ORACLE_USER goto :usage
|
||||
if NOT DEFINED ORACLE_PASSWORD goto :usage
|
||||
|
||||
echo Verifying sqlplus.exe is in path & CALL WHERE /Q sqlplus.exe || GOTO exit
|
||||
echo Verifying sqlldr.exe is in path & CALL WHERE /Q sqlldr.exe || GOTO exit
|
||||
for %F in (sqlplus.exe sqlldr.exe) do (
|
||||
echo Verifying %%F is in path & CALL WHERE /Q %%F || GOTO exit
|
||||
)
|
||||
|
||||
echo Creating user & tables...
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @inventory.sql || GOTO exit
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @customer.sql || GOTO exit
|
||||
|
||||
echo Loading tables data...
|
||||
sqlldr CONTROL=inventory.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
|
||||
sqlldr CONTROL=customer.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
|
||||
for %%F in (sales-user.sql inventory.sql customer.sql) do (
|
||||
echo Executing [%%F]...
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit
|
||||
)
|
||||
|
||||
for %%F in (inventory.ctl customer.ctl) do (
|
||||
echo Loading [%%F]...
|
||||
sqlldr CONTROL=%%F userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
|
||||
)
|
||||
|
||||
:: del /q *.out *.err *.csv
|
||||
endlocal
|
||||
exit /b 0
|
||||
goto :eof
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Data virtualization in SQL Server 2019 big data cluster
|
||||
|
||||
In SQL Server 2019 big data clusters, the SQL Server engine has gained the ability to natively read HDFS files, such as CSV and parquet files, by using SQL Server instances collocated on each of the HDFS data nodes to filter and aggregate data locally in parallel across all of the HDFS data nodes. SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server, Oracle, MongoDB and Teradata.
|
||||
In SQL Server 2019 big data clusters, the SQL Server engine has gained the ability to natively read HDFS files, such as CSV and parquet files, by using SQL Server instances collocated on each of the HDFS data nodes to filter and aggregate data locally in parallel across all of the HDFS data nodes.
|
||||
|
||||
## Query data in HDFS from SQL Server master
|
||||
|
||||
**Applies to: SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2019 big data cluster
|
||||
|
||||
In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node with SQL Server & Spark endpoints. In this example, you are going to create an external table in the SQL Server Master instance that points to data in HDFS within the SQL Server Big data cluster. You will then join the data in the external table with high value data in SQL Master instance.
|
||||
|
||||
@@ -21,3 +21,5 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node
|
||||
1. Execute the [web-clickstreams-parquet.sql](web-clickstreams-hdfs-parquet.sql). This script demonstrates how to read parquet file(s) stored in HDFS.
|
||||
|
||||
1. Execute the [product-reviews-hdfs-csv.sql](product-reviews-hdfs-csv.sql). This script demonstrates how to read CSV file(s) stored in HDFS.
|
||||
|
||||
1. Execute the [product-reviews-hdfs-parquet.sql](product-reviews-hdfs-parquet.sql). This script demonstrates how to read parquet file(s) stored in HDFS.
|
||||
|
||||
+13
-9
@@ -19,14 +19,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file')
|
||||
-- is a special data source that is available in any new database in
|
||||
-- SQL Master instance.
|
||||
--
|
||||
CREATE EXTERNAL TABLE [product_reviews_hdfs_csv]
|
||||
("pr_review_sk" BIGINT , "pr_review_content" varchar(8000))
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/product_review_data',
|
||||
FILE_FORMAT = csv_file
|
||||
);
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'product_reviews_hdfs_csv')
|
||||
CREATE EXTERNAL TABLE [product_reviews_hdfs_csv]
|
||||
("pr_review_sk" BIGINT , "pr_review_content" varchar(8000))
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/product_review_data',
|
||||
FILE_FORMAT = csv_file
|
||||
);
|
||||
GO
|
||||
|
||||
-- Join external table with local tables
|
||||
@@ -38,5 +39,8 @@ SELECT
|
||||
ON pc.pr_review_sk = p.pr_review_sk;
|
||||
GO
|
||||
|
||||
-- Cleanup
|
||||
/*
|
||||
DROP EXTERNAL TABLE [dbo].[product_reviews_hdfs_csv];
|
||||
GO
|
||||
GO
|
||||
*/
|
||||
|
||||
+13
-9
@@ -15,14 +15,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file
|
||||
-- is a special data source that is available in any new database in
|
||||
-- SQL Master instance.
|
||||
--
|
||||
CREATE EXTERNAL TABLE [product_reviews_hdfs_parquet]
|
||||
("pr_review_sk" BIGINT , "pr_review_content" varchar(8000))
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/user/hive/warehouse/product_review_data',
|
||||
FILE_FORMAT = parquet_file
|
||||
);
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'product_reviews_hdfs_parquet')
|
||||
CREATE EXTERNAL TABLE [product_reviews_hdfs_parquet]
|
||||
("pr_review_sk" BIGINT , "pr_review_content" varchar(8000))
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/user/hive/warehouse/product_reviews',
|
||||
FILE_FORMAT = parquet_file
|
||||
);
|
||||
GO
|
||||
|
||||
-- Join external table with local tables
|
||||
@@ -34,5 +35,8 @@ SELECT
|
||||
ON pc.pr_review_sk = p.pr_review_sk;
|
||||
GO
|
||||
|
||||
-- Cleanup
|
||||
/*
|
||||
DROP EXTERNAL TABLE [dbo].[product_reviews_hdfs_parquet];
|
||||
GO
|
||||
GO
|
||||
*/
|
||||
|
||||
+13
-9
@@ -18,14 +18,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file')
|
||||
-- is a special data source that is available in any new database in
|
||||
-- SQL Master instance.
|
||||
--
|
||||
CREATE EXTERNAL TABLE [web_clickstreams_hdfs_csv]
|
||||
("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/clickstream_data',
|
||||
FILE_FORMAT = csv_file
|
||||
);
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_hdfs_csv')
|
||||
CREATE EXTERNAL TABLE [web_clickstreams_hdfs_csv]
|
||||
("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/clickstream_data',
|
||||
FILE_FORMAT = csv_file
|
||||
);
|
||||
GO
|
||||
|
||||
-- Join external table with local tables
|
||||
@@ -48,5 +49,8 @@ SELECT
|
||||
GROUP BY wcs_user_sk;
|
||||
GO
|
||||
|
||||
-- Cleanup
|
||||
/*
|
||||
DROP EXTERNAL TABLE [dbo].[web_clickstreams_hdfs_csv];
|
||||
GO
|
||||
GO
|
||||
*/
|
||||
|
||||
+13
-9
@@ -14,14 +14,15 @@ IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file
|
||||
-- is a special data source that is available in any new database in
|
||||
-- SQL Master instance.
|
||||
--
|
||||
CREATE EXTERNAL TABLE [web_clickstreams_hdfs_parquet]
|
||||
("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/user/hive/warehouse/web_clickstreams',
|
||||
FILE_FORMAT = parquet_file
|
||||
);
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_tables WHERE name = 'web_clickstreams_hdfs_parquet')
|
||||
CREATE EXTERNAL TABLE [web_clickstreams_hdfs_parquet]
|
||||
("wcs_click_date_sk" BIGINT , "wcs_click_time_sk" BIGINT , "wcs_sales_sk" BIGINT , "wcs_item_sk" BIGINT , "wcs_web_page_sk" BIGINT , "wcs_user_sk" BIGINT)
|
||||
WITH
|
||||
(
|
||||
DATA_SOURCE = SqlStoragePool,
|
||||
LOCATION = '/user/hive/warehouse/web_clickstreams',
|
||||
FILE_FORMAT = parquet_file
|
||||
);
|
||||
GO
|
||||
|
||||
-- Join external table with local tables
|
||||
@@ -44,5 +45,8 @@ SELECT
|
||||
GROUP BY wcs_user_sk;
|
||||
GO
|
||||
|
||||
-- Cleanup
|
||||
/*
|
||||
DROP EXTERNAL TABLE [dbo].[web_clickstreams_hdfs_parquet];
|
||||
GO
|
||||
GO
|
||||
*/
|
||||
|
||||
@@ -14,6 +14,7 @@ The new built-in notebooks in Azure Data Studio enables data scientists and data
|
||||
|
||||
This sample builds a machine learning model using AdultCensusIncome.csv available [here](https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv).
|
||||
|
||||
[spark](spark)
|
||||
|
||||
### Instructions
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# SQL Server Machine Learning Services
|
||||
|
||||
SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL.
|
||||
SQL Server 2016 added support for running R scripts from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL.
|
||||
|
||||
**Applies to: SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster
|
||||
|
||||
[r](r)
|
||||
|
||||
|
||||
@@ -4,13 +4,13 @@ SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 add
|
||||
|
||||
[book-click-prediction-py.sql](book-click-prediction-py.sql/)
|
||||
|
||||
**Applies to: SQL Server 2017+, SQL Server 2019, SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2017+, SQL Server 2019, SQL Server 2019 big data cluster
|
||||
|
||||
In this example, we are building a machine learning model using Python and a logistic regression algorithm for a recommendation engine on an online store. Based on existing users' click pattern online and their interest in other categories and demographics, we are training a machine learning model. This model will then be used to predict if the visitor is interested in a given item category using the T-SQL PREDICT function.
|
||||
|
||||
[book-click-prediction-partitioned-py.sql](book-click-prediction-partitioned-py.sql/)
|
||||
|
||||
**Applies to: SQL Server 2019, SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2019, SQL Server 2019 big data cluster
|
||||
|
||||
In this example, we are leveraging the new partitioning support (SQL Server 2019) in sp_execute_external_script to partition the input data and run the Python script per partition. So we will modify the training script to train model per group of users based on credit rating. The Python script will produce N models for the same input data set.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -1,16 +1,16 @@
|
||||
# SQL Server Machine Learning Services
|
||||
|
||||
SQL Server 2016 added capability to run R script from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL.
|
||||
SQL Server 2016 added support for running R scripts from T-SQL. SQL Server 2017 added support for running Python scripts from T-SQL. SQL Server 2019 adds support for running Java code from T-SQL.
|
||||
|
||||
[book-click-prediction-r.sql](book-click-prediction-r.sql/)
|
||||
|
||||
**Applies to: SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2016+, SQL Server 2019, SQL Server 2019 big data cluster
|
||||
|
||||
In this example, we are building a machine learning model using R and a logistic regression algorithm for a recommendation engine on an online store. Based on existing users' click pattern online and their interest in other categories and demographics, we are training a machine learning model. This model will then be used to predict if the visitor is interested in a given item category using the T-SQL PREDICT function.
|
||||
|
||||
[book-click-prediction-partitioned-r.sql](book-click-prediction-partitioned-r.sql/)
|
||||
|
||||
**Applies to: SQL Server 2019, SQL Server 2019 big data cluster**
|
||||
**Applies to:** SQL Server 2019, SQL Server 2019 big data cluster
|
||||
|
||||
In this example, we are leveraging the new partitioning support (SQL Server 2019) in sp_execute_external_script to partition the input data and run the R script per partition. So we will modify the training script to train model per group of users based on credit rating. The R script will produce N models for the same input data set.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -39,7 +39,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "# Disable saving SUCCESS file\r\nsc._jsc.hadoopConfiguration().set(\"mapreduce.fileoutputcommitter.marksuccessfuljobs\", \"false\") \r\n\r\n# Print the current warehouse directory\r\nprint(spark.conf.get(\"spark.sql.warehouse.dir\"))\r\n\r\n# Save results as parquet file and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"web_clickstreams\")\r\n",
|
||||
"source": "# Disable saving SUCCESS file\r\nsc._jsc.hadoopConfiguration().set(\"mapreduce.fileoutputcommitter.marksuccessfuljobs\", \"false\") \r\n\r\n# Print the current warehouse directory where the parquet files will be stored\r\nprint(spark.conf.get(\"spark.sql.warehouse.dir\"))\r\n\r\n# Save results as parquet file and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"web_clickstreams\")\r\n",
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
@@ -61,7 +61,7 @@
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 38250| 4172| 67067| 11504| 40| 16819|\n| 38251| 28919| 67090| 13782| 40| 11283|\n| 38251| 77021| 67096| 4330| 40| 60107|\n| 38251| 29023| 67109| 15796| 40| 31730|\n| 38251| 54047| 67110| 9739| 40| 50449|\n| 38251| 85733| 67117| 6843| 40| 39327|\n| 38252| 53176| 67141| 12525| 40| 37913|\n| 38252| 15873| 67153| 13008| 40| 92546|\n| 38252| 39147| 67167| 5208| 40| 74534|\n| 38252| 79540| 67171| 11552| 40| 94065|\n| 38252| 35200| 67175| 9622| 40| 80502|\n| 38253| 26068| 67191| 8585| 40| 43314|\n| 38253| 63065| 67195| 17486| 40| 63793|\n| 38253| 9687| 67214| 9856| 40| 92780|\n| 38253| 18373| 67219| 406| 40| 38319|\n| 38254| 80201| 67229| 13610| 40| 15342|\n| 38254| 40058| 67239| 13594| 40| 41879|\n| 38254| 79136| 67243| 1933| 40| 42095|\n| 38254| 14684| 67244| 14267| 40| 39119|\n| 38254| 36369| 67248| 641| 40| 82237|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows\n\n+-----------+--------+\n|wcs_user_sk|count(1)|\n+-----------+--------+\n| 65042| 832|\n| 55928| 821|\n| 15570| 791|\n| 31138| 788|\n| 68188| 784|\n| 88205| 760|\n| 15678| 757|\n| 48063| 741|\n| 77518| 741|\n| 92978| 728|\n| 82129| 727|\n| 21700| 725|\n| 69707| 724|\n| 38895| 719|\n| 97643| 716|\n| 74426| 707|\n| 7813| 704|\n| 49528| 700|\n| 55766| 698|\n| 54355| 697|\n+-----------+--------+\nonly showing top 20 rows",
|
||||
"text": "+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 36890| 40052| null| 4379| 34| null|\n| 36890| 41285| null| 6245| 34| null|\n| 36890| 23115| null| 13852| 34| null|\n| 36890| 17702| null| 15975| 34| null|\n| 36890| 62676| null| 2119| 34| null|\n| 36890| 34267| null| 10273| 34| null|\n| 36890| 8502| null| 17790| 34| null|\n| 36890| 54340| null| 3453| 34| null|\n| 36890| 54370| null| 6372| 34| null|\n| 36890| 6578| null| 17203| 34| null|\n| 36890| 75088| null| 4891| 34| null|\n| 36890| 23922| null| 11332| 34| null|\n| 36890| 28761| null| 4484| 34| null|\n| 36890| 21444| null| 5582| 34| null|\n| 36890| 58917| null| 8833| 34| null|\n| 36890| 27578| null| 8599| 34| null|\n| 36890| 8059| null| 6720| 34| null|\n| 36890| 43008| null| 17175| 34| null|\n| 36890| 4378| null| 10644| 34| null|\n| 36890| 55403| null| 8139| 34| null|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows\n\n+-----------+--------+\n|wcs_user_sk|count(1)|\n+-----------+--------+\n| 65042| 832|\n| 55928| 821|\n| 15570| 791|\n| 31138| 788|\n| 68188| 784|\n| 88205| 760|\n| 15678| 757|\n| 48063| 741|\n| 77518| 741|\n| 92978| 728|\n| 82129| 727|\n| 21700| 725|\n| 69707| 724|\n| 38895| 719|\n| 97643| 716|\n| 74426| 707|\n| 7813| 704|\n| 49528| 700|\n| 55766| 698|\n| 54355| 697|\n+-----------+--------+\nonly showing top 20 rows",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user