github/sql-server-samples
1
0
Fork 0
mirror of https://github.com/Microsoft/sql-server-samples.git synced 2025-12-08 14:58:54 +00:00
Code Issues Releases Wiki Activity

mssql spark connector sample

Browse Source
This commit is contained in:
shivsood
2019-04-23 12:37:31 -07:00
parent e67a4c87fd d6b9c3e373
commit babf4bc4c5
31 changed files with 775 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/features/query-store/Auto-Param Analysis.sql
+61
View File
@@ -0,0 +1,61 @@
USE [AdventureWorks2016_EXT]
GO
/* (1) Do cardinality analysis when suspect on ad-hoc workloads*/
SELECT COUNT(*) AS CountQueryTextRows FROM sys.query_store_query_text;
SELECT COUNT(*) AS CountQueryRows FROM sys.query_store_query;
SELECT COUNT(DISTINCT query_hash) AS CountDifferentQueryRows FROM sys.query_store_query;
SELECT COUNT(*) AS CountPlanRows FROM sys.query_store_plan;
SELECT COUNT(DISTINCT query_plan_hash) AS CountDifferentPlanRows FROM sys.query_store_plan;
/* (2) Get Compile Vs Execution times: ad-hoc workloads tend to spend lot of time in compilation*/
EXEC sp_GetCompilAndExecutionTotalTime
/* (3) See query pattern*/
SELECT TOP 10 * FROM sys.query_store_query_text
/* (4) I'm not getting new queries?
Look at Query Store parameters - is Query Store in READ_ONLY mode?
*/
SELECT current_storage_size_mb, max_storage_size_mb, desired_state, desired_state_desc, actual_state, actual_state_desc, readonly_reason, flush_interval_seconds,
interval_length_minutes, stale_query_threshold_days, max_plans_per_query, query_capture_mode, query_capture_mode_desc, size_based_cleanup_mode,
size_based_cleanup_mode_desc, actual_state_additional_info
FROM sys.database_query_store_options
ALTER DATABASE [AdventureWorks2016_EXT] SET QUERY_STORE CLEAR;
ALTER DATABASE [AdventureWorks2016_EXT] SET QUERY_STORE = ON (OPERATION_MODE = READ_WRITE);
GO
/* (5) How do we fix the Auto-Param problem?*/
/* At the query level: apply the plan guide for selected query template */
DECLARE @stmt nvarchar(max);
DECLARE @params nvarchar(max);
EXEC sp_get_query_template
N'select * from part p join partdetails pp on p.partid = pp.partid where p.partid = 46911',
@stmt OUTPUT,
@params OUTPUT;
EXEC sp_create_plan_guide
N'TemplateGuide1',
@stmt,
N'TEMPLATE',
NULL,
@params,
N'OPTION(PARAMETERIZATION FORCED)';
/*(6) Alternative (at the database level): force parametrization for all queries*/
ALTER DATABASE [AdventureWorks2016_EXT] SET PARAMETERIZATION FORCED;
/* Run analysis query (1), (2) again to see results of parametrization */
/*(7) Reset the DB state*/
ALTER DATABASE [AdventureWorks2016_EXT] SET PARAMETERIZATION SIMPLE;
GO
EXEC sp_control_plan_guide N'DROP', N'TemplateGuide1';
GO
ALTER DATABASE [AdventureWorks2016_EXT] SET QUERY_STORE CLEAR;
ALTER DATABASE [AdventureWorks2016_EXT] SET QUERY_STORE = ON (OPERATION_MODE = READ_WRITE);
SELECT * FROM sys.database_query_store_options
GO
samples/features/query-store/NewDatabaseSettings.sql
BIN
View File
Binary file not shown.
samples/features/query-store/QS_SSMS.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 390 KiB

samples/features/query-store/QueryStoreSimpleDemo.exe
BIN
View File
Binary file not shown.
samples/features/query-store/README.md
+46
View File
@@ -0,0 +1,46 @@
## Query Store demo
This demo shows capabilities of Query Store. Usually we demo 3-4 scenarios:
1. How to turn on and initially configure Query Store
2. How Query Store collects & exposed data
3. Detecting and fixing query with plan choice regression
4. Detecting and fixing workload that is candidate for auto-parametrization
## Prerequisites
Restore AdventureWorks2016_EXT database from the provided BAK at https://github.com/Microsoft/sql-server-samples/releases/tag/adventureworks.
After restoring the database AdventureWorks2016_EXT, go to Properties / Query Store tab, turn ON Query Store, and configure it according to best practices in https://docs.microsoft.com/sql/relational-databases/performance/best-practice-with-the-query-store.
![Query Store in SSMS](./QS_SSMS.png)
Use docs content to walk through main config settings: https://docs.microsoft.com//sql/relational-databases/performance/best-practice-with-the-query-store#Configure
### How Query Store Works
Open ShowBasics.sql script and execute queries individually:
- Run simple `SELECT * FROM` Part
- Show where query ends in sys.query_store_query_text, sys.query_store_query, sys.query_store_plan, sys.query_store_runtime_stats
- Use custom view `vw_QueryStoreCompileInfo` to get info more easily. The main point here is: people can write their own scripts combining Query Store views
- Execute the same user query from the proc, using sp_executesql, trigger and show that containing object defines query identity in QDS (each instance of the same query text becomes separate query that can be monitored and tuned independently)
- Show what happens with query that gets auto-parametrized. It cannot be searched using the original query text because QDS stores query as parametrized. Hopefully, sys.fn_stmt_sql_handle_from_sql_stmt can be used to track down query using original query text
- Run `vw_QueryStoreRuntimeInfo` (again custom view) to show main runtime stats combined with query/plan info
### Query with plan regression
1. Run QueryStoreSimpleDemo.exe with option R or option S
2. Open SSMS, analyze and explain - two execution plans that SQL Server use alternately (switches between 2 plan almost randomly). This is known as Parameter Sniffing problem - plan gets generated based on parameter available at the compilation time. When compilation happens frequently and randomly and data is skewed (not all parameter values are uniformly distributed PSP is likely to occur and degradations are common)
3. Force better plan, explain what happens (SSMS)
4. Summarize benefits for DBA – fixing performance quickly without knowing details about the query. Fully transparent to running apps
### Detect and fix ad hoc workload that is candidate for parametrization
1. Run QueryStoreSimpleDemo.exe with option P and let it work for some time (15-20 sec)
2. Open “Auto-Param Analysis.sql” and run queries from groups (1) and (2). What we see is:
a. Large number of queries / plan entries, small number of different query/plan hashes indicates queries that are not parametrized although they are good candidates
b. Relatively big compile time shows that system wastes resources on compilation instead of execution
3. Open SSMS Top Resource Consuming queries – if you increase number of presented queries to 50 you’ll see that majority of queries has similar /negligible consumption – there’s nothing user can optimize/tune. This is what we call “death by a thousand of cuts”
4. Run (3) query to see query text pattern – it becomes obvious that queries differ only by provided literal value
5. If you run (1) you’ll notice that numbers do not change although workload is running. (4) gives us the answer – Query Store went to READ_ONLY due to large number of queries / plans. This is another point you should make: ad-hoc queries are not bad for SQL Server & execution but also for Query Store as it goes to RO mode which means we do not operate with latest facts!
6. Run (5) to parametrize query and clear Query Store. Workload is still running!
7. Run (1) again to see numbers now: ration between count(queries) and count(distinct query_hash) is now near to 1.
8. Open Open SSMS Top Resource Consuming queries: you’ll see dozen of different queries to tune
9. (6) show alternative solution – applying forced parametrization for the entire DB. Just mention, as a possible solution
10. Run (7) to reset DB to initial state.
samples/features/query-store/ShowBasics.sql
+96
View File
@@ -0,0 +1,96 @@
/*Clear Query Store and procedure cache*/
ALTER DATABASE AdventureWorks2016_EXT SET QUERY_STORE CLEAR;
ALTER DATABASE AdventureWorks2016_EXT SET QUERY_STORE = ON (QUERY_CAPTURE_MODE = ALL);
DBCC FREEPROCCACHE
GO
USE AdventureWorks2016_EXT;
GO
/*Run simple query - what data is collected and where does it go to?*/
SELECT * FROM Part;
SELECT * FROM sys.query_store_query_text;
SELECT * FROM sys.query_store_query;
SELECT * FROM sys.query_store_plan;
SELECT * FROM sys.query_store_runtime_stats;
/*
Combine all info
vw_QueryStoreCompileInfo is custom view (created for presentation)
*/
SELECT * FROM vw_QueryStoreCompileInfo
WHERE query_sql_text = 'SELECT * FROM Part'
/*The same query from the proc*/
DROP PROCEDURE IF EXISTS sp_GetParts
GO
CREATE PROCEDURE sp_GetParts
AS
SELECT * FROM Part;
GO
EXEC sp_GetParts;
/*Again the same query, from sp_executesql*/
EXEC sp_executesql N'SELECT * FROM Part'
SELECT * FROM vw_QueryStoreCompileInfo
WHERE query_sql_text = 'SELECT * FROM Part'
/*Finally trigger*/
DROP TRIGGER IF EXISTS dbo.OnPartInsert
GO
CREATE TRIGGER dbo.OnPartInsert
ON dbo.Part
AFTER INSERT
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
SELECT * FROM Part;
END
GO
INSERT INTO Part VALUES (3000020, 'Part_300020');
SELECT * FROM vw_QueryStoreCompileInfo
WHERE query_sql_text = 'SELECT * FROM Part'
/*What happens with parametrized query?*/
SELECT * FROM Part WHERE PartId = 5;
SELECT * FROM vw_QueryStoreCompileInfo
WHERE query_sql_text = 'SELECT * FROM Part = 5'
/* Check sys.query_store_query_text */
SELECT * FROM sys.query_store_query_text;
/*Try sys.fn_stmt_sql_handle_from_sql_stmt this instead*/
SELECT * FROM sys.fn_stmt_sql_handle_from_sql_stmt
('SELECT * FROM Part WHERE PartId = 5', NULL)
/*Changed searched criteria*/
SELECT V.* FROM vw_QueryStoreCompileInfo V
JOIN sys.fn_stmt_sql_handle_from_sql_stmt
('SELECT * FROM Part WHERE PartId = 5', NULL) F
ON V.statement_sql_handle = F.statement_sql_handle
/*Get runtime info for the queries*/
SELECT * FROM vw_QueryStoreRuntimeInfo
WHERE query_sql_text = 'SELECT * FROM Part'
ORDER BY start_time DESC
SELECT * FROM vw_QueryStoreRuntimeInfo V
JOIN sys.fn_stmt_sql_handle_from_sql_stmt
('SELECT * FROM Part WHERE PartId = 5', NULL) F
ON V.statement_sql_handle = F.statement_sql_handle
ORDER BY start_time DESC
samples/features/query-store/sp_GetCompilAndExecutionTotalTime.sql
+25
View File
@@ -0,0 +1,25 @@
USE [AdventureWorks2016_EXT]
GO
DROP PROCEDURE IF EXISTS sp_GetCompilAndExecutionTotalTime
GO
CREATE PROCEDURE sp_GetCompilAndExecutionTotalTime
AS
DECLARE @totalCompiles int
DECLARE @totalExecutions int
DECLARE @totalCompileTime decimal(18,4)
DECLARE @totalExecutionTime decimal(18,4)
SELECT @totalCompiles = SUM(count_compiles),
@totalCompileTime = SUM(count_compiles * avg_compile_duration / 1000.)
FROM sys.query_store_plan;
SELECT @totalExecutions = SUM(count_executions),
@totalExecutionTime = SUM(count_executions * avg_duration / 1000.)
FROM sys.query_store_runtime_stats
SELECT @totalCompiles AS TotalCompiles, @totalExecutions AS TotalExecutions,
@totalCompileTime AS TotalCompileTime, @totalExecutionTime AS TotalDurationTime
GO
samples/features/query-store/vw_QueryStoreCompileInfo.sql
+19
View File
@@ -0,0 +1,19 @@
USE [AdventureWorks2016_EXT]
GO
DROP VIEW IF EXISTS [vw_QueryStoreCompileInfo];
GO
CREATE VIEW [dbo].[vw_QueryStoreCompileInfo]
AS
SELECT qt.query_text_id, q.query_id, p.plan_id, qt.query_sql_text, s.name AS ContainingSchema, o.name AS ContainingObject, q.query_hash, qt.statement_sql_handle, q.is_internal_query,
q.query_parameterization_type_desc, q.count_compiles AS query_count_compiles, p.query_plan_hash, p.count_compiles AS plan_count_compiles, p.last_compile_start_time, p.engine_version,
p.compatibility_level, p.query_plan, p.is_trivial_plan, p.is_parallel_plan, p.is_forced_plan
FROM sys.query_store_query_text AS qt INNER JOIN
sys.query_store_query AS q ON qt.query_text_id = q.query_text_id INNER JOIN
sys.query_store_plan AS p ON q.query_id = p.query_id LEFT OUTER JOIN
sys.objects AS o ON q.object_id = o.object_id LEFT OUTER JOIN
sys.schemas AS s ON s.schema_id = o.schema_id
GO
samples/features/query-store/vw_QueryStoreRuntimeInfo.sql
+21
View File
@@ -0,0 +1,21 @@
USE [AdventureWorks2016_EXT]
GO
DROP VIEW IF EXISTS [dbo].[vw_QueryStoreRuntimeInfo]
GO
CREATE VIEW [dbo].[vw_QueryStoreRuntimeInfo]
AS
SELECT qt.query_text_id, q.query_id, p.plan_id, qt.query_sql_text, s.name AS ContainingSchema, o.name AS ContainingObject, qt.statement_sql_handle, rsi.start_time, rsi.end_time, rs.execution_type_desc,
rs.count_executions, rs.avg_duration, rs.max_duration, rs.avg_cpu_time, rs.max_cpu_time, rs.avg_logical_io_reads, rs.max_logical_io_reads, rs.avg_physical_io_reads, rs.max_physical_io_reads,
rs.avg_logical_io_writes, rs.max_logical_io_writes, rs.avg_query_max_used_memory, rs.max_query_max_used_memory, rs.avg_rowcount, rs.max_rowcount, rs.avg_dop, rs.max_dop
FROM sys.query_store_query_text AS qt INNER JOIN
sys.query_store_query AS q ON qt.query_text_id = q.query_text_id INNER JOIN
sys.query_store_plan AS p ON q.query_id = p.query_id LEFT OUTER JOIN
sys.objects AS o ON q.object_id = o.object_id LEFT OUTER JOIN
sys.schemas AS s ON s.schema_id = o.schema_id INNER JOIN
sys.query_store_runtime_stats AS rs ON rs.plan_id = p.plan_id INNER JOIN
sys.query_store_runtime_stats_interval AS rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id
GO
samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd
+36 -34
View File
@@ -8,6 +8,8 @@ set SQL_MASTER_SA_PASSWORD=%3
set KNOX_IP=%4
set KNOX_PASSWORD=%5
set AW_WWI_SAMPLES=%6
set SQL_MASTER_PORT=%7
set KNOX_PORT=%8
set STARTUP_PATH=%~dp0
set TMP_DIR_NAME=%~nx0
@@ -17,16 +19,18 @@ if NOT DEFINED SQL_MASTER_SA_PASSWORD goto :usage
if NOT DEFINED KNOX_IP goto :usage
if NOT DEFINED KNOX_PASSWORD set KNOX_PASSWORD=%SQL_MASTER_SA_PASSWORD%
if NOT DEFINED AW_WWI_SAMPLES set AW_WWI_SAMPLES=no
if NOT DEFINED SQL_MASTER_PORT set SQL_MASTER_PORT=31433
if NOT DEFINED KNOX_PORT set KNOX_PORT=30443
set SQL_MASTER_INSTANCE=%SQL_MASTER_IP%,31433
set KNOX_ENDPOINT=%KNOX_IP%:30443
set SQL_MASTER_INSTANCE=%SQL_MASTER_IP%,%SQL_MASTER_PORT%
set KNOX_ENDPOINT=%KNOX_IP%:%KNOX_PORT%
for %%F in (sqlcmd.exe bcp.exe kubectl.exe curl.exe) do (
echo Verifying %%F is in path & CALL WHERE /Q %%F || GOTO exit
)
pushd "%tmp%"
md %TMP_DIR_NAME%
md %TMP_DIR_NAME% >NUL
cd %TMP_DIR_NAME%
if NOT EXIST tpcxbb_1gb.bak (
@@ -34,47 +38,45 @@ if NOT EXIST tpcxbb_1gb.bak (
%DEBUG% curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
)
set SQLCMDSERVER=%SQL_MASTER_INSTANCE%
set SQLCMDUSER=sa
set SQLCMDPASSWORD=%SQL_MASTER_SA_PASSWORD%
for /F "usebackq" %%v in (`sqlcmd -I -b -h-1 -Q "print RTRIM((CAST(SERVERPROPERTY('ProductLevel') as nvarchar(128))));"`) do SET CTP_VERSION=%%v
if /i "%CTP_VERSION%" EQU "CTP2.4" (set MASTER_POD_NAME=mssql-master-pool-0) else (set MASTER_POD_NAME=master-0)
REM Copy the backup file, restore the database, create necessary objects and data file
echo Copying sales database backup file to SQL Master instance...
%DEBUG% kubectl cp tpcxbb_1gb.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
%DEBUG% kubectl cp tpcxbb_1gb.bak %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:/var/opt/mssql/data -c mssql-server || goto exit
REM Download and copy the sample backup files
if /i %AW_WWI_SAMPLES% EQU --install-extra-samples (
if NOT EXIST AdventureWorks2016_EXT.bak (
echo Downloading AdventureWorks2016_EXT sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorks2016_EXT.bak" -o AdventureWorks2016_EXT.bak
if /i "%AW_WWI_SAMPLES%" EQU "--install-extra-samples" (
set FILES=AdventureWorks2016_EXT.bak AdventureWorksDW2016_EXT.bak
for %%f in (!FILES!) do (
if NOT EXIST %%f (
echo Downloading %%f sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/%%f" -o %%f
)
echo Copying %%f database backup file to SQL Master instance...
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:/var/opt/mssql/data -c mssql-server || goto exit
)
echo Copying AdventureWorks2016_EXT database backup file to SQL Master instance...
%DEBUG% kubectl cp AdventureWorks2016_EXT.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
if NOT EXIST AdventureWorksDW2016_EXT.bak (
echo Downloading AdventureWorksDW2016_EXT sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorksDW2016_EXT.bak" -o AdventureWorksDW2016_EXT.bak
set FILES=WideWorldImporters-Full.bak WideWorldImportersDW-Full.bak
for %%f in (!FILES!) do (
if NOT EXIST %%f (
echo Downloading %%f sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/%%f" -o %%f
)
echo Copying %%f database backup file to SQL Master instance...
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:/var/opt/mssql/data -c mssql-server || goto exit
)
echo Copying AdventureWorksDW2016_EXT database backup file to SQL Master instance...
%DEBUG% kubectl cp AdventureWorksDW2016_EXT.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
if NOT EXIST WideWorldImporters-Full.bak (
echo Downloading WideWorldImporters sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak" -o WideWorldImporters-Full.bak
)
echo Copying WideWorldImporters-Full database backup file to SQL Master instance...
%DEBUG% kubectl cp WideWorldImporters-Full.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
if NOT EXIST WideWorldImportersDW-Full.bak (
echo Downloading WideWorldImportersDW sample database backup file...
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImportersDW-Full.bak" -o WideWorldImportersDW-Full.bak
)
echo Copying WideWorldImportersDW-Full database backup file to SQL Master instance...
%DEBUG% kubectl cp WideWorldImportersDW-Full.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
)
echo Configuring sample database(s)...
%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b -v SA_PASSWORD="%KNOX_PASSWORD%" || goto exit
%DEBUG% sqlcmd -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b -v SA_PASSWORD="%KNOX_PASSWORD%" || goto exit
REM remove files copied into the pod:
echo Removing database backup files...
kubectl exec mssql-master-pool-0 -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
%DEBUG% kubectl exec %MASTER_POD_NAME% -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
for %%F in (web_clickstreams inventory customer) do (
if NOT EXIST %%F.csv (
@@ -84,7 +86,6 @@ for %%F in (web_clickstreams inventory customer) do (
)
)
if NOT EXIST product_reviews.csv (
echo Exporting product_reviews data...
%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit
@@ -103,6 +104,7 @@ echo Uploading product_reviews data to HDFS...
:: del /q product_reviews.*
REM %DEBUG% del /q *.out *.err *.csv
echo .
echo Bootstrap of the sample database completed successfully.
echo You can now login using "root" and Knox password to get the unified experience in Azure Data Studio.
echo Data files for Oracle setup are located at [%TMP%\%TMP_DIR_NAME%].
@@ -118,6 +120,6 @@ goto :eof
exit /b 1
:usage
echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>] [--install-extra-samples]
echo Default ports are assumed for SQL Master instance ^& Knox gateway.
echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>] [--install-extra-samples] [SQL_MASTER_PORT] [KNOX_PORT]
echo Default ports are assumed for SQL Master instance ^& Knox gateway unless specified.
exit /b 0
samples/features/sql-big-data-cluster/bootstrap-sample-db.sh
+25 -8
View File
@@ -3,7 +3,7 @@ set -e
set -o pipefail
STARTUP_PATH=$(pwd)
TMP_DIR_NAME=$(basename $0)
USAGE_MESSAGE="USAGE: $0 <CLUSTER_NAMESPACE> <SQL_MASTER_IP> <SQL_MASTER_SA_PASSWORD> <KNOX_IP> [<KNOX_PASSWORD>] [--install-extra-samples]"
USAGE_MESSAGE="USAGE: $0 <CLUSTER_NAMESPACE> <SQL_MASTER_IP> <SQL_MASTER_SA_PASSWORD> <KNOX_IP> [<KNOX_PASSWORD>] [--install-extra-samples] [SQL_MASTER_PORT] [KNOX_PORT]"
ERROR_MESSAGE="Bootstrap of the sample database failed. Output and error files are in directory [/tmp/$TMP_DIR_NAME]."
# Print usage if mandatory parameters are missing
@@ -20,12 +20,20 @@ SQL_MASTER_SA_PASSWORD=$3
KNOX_IP=$4
KNOX_PASSWORD=$5
AW_WWI_SAMPLES=$6
SQL_MASTER_PORT=$7
KNOX_PORT=$8
# If Knox password is not supplied then default to SQL Master password
KNOX_PASSWORD=${KNOX_PASSWORD:=$SQL_MASTER_SA_PASSWORD}
SQL_MASTER_INSTANCE=$SQL_MASTER_IP,31433
KNOX_ENDPOINT=$KNOX_IP:30443
# Skip if extra samples doesn't need to be installed
AW_WWI_SAMPLES=${AW_WWI_SAMPLES:=no}
# Use default ports if not specified
SQL_MASTER_PORT=${SQL_MASTER_PORT:=31433}
KNOX_PORT=${KNOX_PORT:=30443}
SQL_MASTER_INSTANCE=$SQL_MASTER_IP,$SQL_MASTER_PORT
KNOX_ENDPOINT=$KNOX_IP:$KNOX_PORT
for util in sqlcmd bcp kubectl curl
do
@@ -43,11 +51,20 @@ then
$DEBUG curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
fi
CTP_VERSION=$(sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b -h-1 -Q "print RTRIM((CAST(SERVERPROPERTY('ProductLevel') as nvarchar(128))));")
if [ "$CTP_VERSION" == "CTP2.4" ]
then
MASTER_POD_NAME=mssql-master-pool-0
else
MASTER_POD_NAME=master-0
fi
echo Copying database backup file...
$DEBUG kubectl cp tpcxbb_1gb.bak $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
$DEBUG kubectl cp tpcxbb_1gb.bak $CLUSTER_NAMESPACE/$MASTER_POD_NAME:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
# $DEBUG rm tpcxbb_1gb.bak
if [ $AW_WWI_SAMPLES == --install-extra-samples ]
if [ "$AW_WWI_SAMPLES" == "--install-extra-samples" ]
then
for file in AdventureWorks2016_EXT.bak AdventureWorksDW2016_EXT.bak
do
@@ -57,7 +74,7 @@ then
$DEBUG curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/$file" -o $file
fi
echo Copying $file database backup file to SQL Master instance...
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/$MASTER_POD_NAME:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
done
@@ -69,7 +86,7 @@ then
$DEBUG curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/$file" -o $file
fi
echo Copying $file database backup file to SQL Master instance...
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/$MASTER_POD_NAME:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
done
fi
@@ -80,7 +97,7 @@ $DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b < "$S
# remove files copied into the pod:
echo Removing database backup files...
kubectl exec mssql-master-pool-0 -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
kubectl exec $MASTER_POD_NAME -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
for table in web_clickstreams inventory customer
do
samples/features/sql-big-data-cluster/bootstrap-sample-db.sql
+18 -6
View File
@@ -80,16 +80,28 @@ BEGIN
WITH (LOCATION = 'sqldatapool://service-mssql-controller:8080/datapools/default');
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://nmnode-0-0.nmnode-0-svc:9000/',
RESOURCE_MANAGER_LOCATION='master-0.master-svc:8032'
);
END;
GO
samples/features/sql-big-data-cluster/data-virtualization/generic-odbc/mysql/mysql_version.sql
+2 -1
View File
@@ -13,7 +13,7 @@ IF NOT EXISTS(SELECT * FROM sys.database_scoped_credentials WHERE name = 'MySQL8
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'MySQL80')
CREATE EXTERNAL DATA SOURCE MySQL80
WITH (LOCATION = 'odbc://uc-win19-vm.redmond.corp.microsoft.com'
, CONNECTION_OPTIONS = 'Driver={MySQL 8.0 ODBC Driver Unicode Driver};User name=%u;Passwword=%p'
, CONNECTION_OPTIONS = 'Driver={MySQL ODBC 8.0 Unicode Driver};User name=%u;Passwword=%p;IGNORE_SPACE=1'
, CREDENTIAL = [MySQL80-user]);
-- Create external table over inventory table on MySQL server
@@ -60,6 +60,7 @@ SELECT * FROM mysql_tables;
*/
-- Cleanup
/*
DROP EXTERNAL TABLE mysql_version
DROP EXTERNAL TABLE mysql_tables
DROP EXTERNAL DATA SOURCE MySQL80
DROP DATABASE SCOPED CREDENTIAL [MySQL80-user]
samples/features/sql-big-data-cluster/data-virtualization/hadoop/README.md
+2 -4
View File
@@ -10,12 +10,10 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node
### Instructions
1. Connect to HDFS/Knox gateway from Azure Data Studio using SQL Server big data cluster connection type.
1. Run the [../../spark/spark-sql.ipynb](../../spark/spark-sql.ipynb/) notebook to generate the sample parquet file(s).
1. Connect to SQL Server Master instance.
1. Run the [../../spark/dataloading/transform-csv-files.ipynb](../../spark/dataloading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
1. Execute the [web-clickstreams-hdfs-orc.sql](web-clickstreams-hdfs-orc.sql). This script demonstrates how to read ORC file(s) stored in HDFS.
1. Execute the [product-reviews-hdfs-orc.sql](product-reviews-hdfs-orc.sql). This script demonstrates how to read ORC file(s) stored in HDFS.
samples/features/sql-big-data-cluster/data-virtualization/hadoop/inventory-export-hdfs-rcfile.sql
+14 -6
View File
@@ -17,12 +17,20 @@ GO
-- execution.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://nmnode-0-0.nmnode-0-svc:9000/',
RESOURCE_MANAGER_LOCATION='master-0.master-svc:8032'
);
-- Create file format for RCFILE with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/hadoop/product-reviews-hdfs-orc.sql
+14 -6
View File
@@ -7,12 +7,20 @@ GO
-- execution.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://nmnode-0-0.nmnode-0-svc:9000/',
RESOURCE_MANAGER_LOCATION='master-0.master-svc:8032'
);
-- Create file format for orc file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/hadoop/web-clickstreams-hdfs-orc.sql
+14 -6
View File
@@ -7,12 +7,20 @@ GO
-- execution.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
);
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE HadoopData
WITH(
TYPE=HADOOP,
LOCATION='hdfs://nmnode-0-0.nmnode-0-svc:9000/',
RESOURCE_MANAGER_LOCATION='master-0.master-svc:8032'
);
-- Create file format for orc file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/README.md
+2 -4
View File
@@ -10,12 +10,10 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node
### Instructions
1. Connect to HDFS/Knox gateway from Azure Data Studio using SQL Server big data cluster connection type.
1. Run the [../../spark/spark-sql.ipynb](../../spark/spark-sql.ipynb/) notebook to generate the sample parquet file(s).
1. Connect to SQL Server Master instance.
1. Run the [../../spark/dataloading/transform-csv-files.ipynb](../../spark/dataloading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
1. Execute the [web-clickstreams-hdfs-csv.sql](web-clickstreams-hdfs-csv.sql). This script demonstrates how to read CSV file(s) stored in HDFS.
1. Execute the [web-clickstreams-parquet.sql](web-clickstreams-hdfs-parquet.sql). This script demonstrates how to read parquet file(s) stored in HDFS.
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-csv.sql
+7 -3
View File
@@ -1,11 +1,15 @@
USE sales
GO
-- Create external data source for HDFS inside SQ: big data cluster.
-- Create external data source for HDFS inside SQL big data cluster.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
-- Create file format for CSV separated file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-parquet.sql
+7 -3
View File
@@ -1,11 +1,15 @@
USE sales
GO
-- Create external data source for HDFS inside SQ: big data cluster.
-- Create external data source for HDFS inside SQL big data cluster.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
-- Create file format for parquet file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/product-reviews-hdfs-tsv.sql
+7 -3
View File
@@ -1,11 +1,15 @@
USE sales
GO
-- Create external data source for HDFS inside SQ: big data cluster.
-- Create external data source for HDFS inside SQL big data cluster.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
-- Create file format for tab separated file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-csv.sql
+7 -3
View File
@@ -1,11 +1,15 @@
USE sales
GO
-- Create external data source for HDFS inside SQ: big data cluster.
-- Create external data source for HDFS inside SQL big data cluster.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
-- Create file format for CSV file with appropriate properties.
--
samples/features/sql-big-data-cluster/data-virtualization/storage-pool/web-clickstreams-hdfs-parquet.sql
+7 -3
View File
@@ -1,11 +1,15 @@
USE sales
GO
-- Create external data source for HDFS inside SQ: big data cluster.
-- Create external data source for HDFS inside SQL big data cluster.
--
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.5'
CREATE EXTERNAL DATA SOURCE SqlStoragePool
WITH (LOCATION = 'sqlhdfs://nmnode-0-0.nmnode-0-svc:50070');
-- Create file format for parquet file with appropriate properties.
--
samples/features/sql-big-data-cluster/spark/README.md
+1 -1
View File
@@ -16,7 +16,7 @@ SQL Server Big Data cluster bundles Spark and HDFS together with SQL server. Azu
## Instructions on how to run in Azure Data Studio
1. Download and save the notebook file [dataloading/transnform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/) locally.
[data-loading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
2. From Azure Data Studio Connect to the SQL Server Master instance in a big data cluster.
samples/features/sql-big-data-cluster/spark/dataloading/hello_PySpark.ipynb → samples/features/sql-big-data-cluster/spark/data-loading/hello_PySpark.ipynb
View File
samples/features/sql-big-data-cluster/spark/dataloading/hello_Scala.ipynb → samples/features/sql-big-data-cluster/spark/data-loading/hello_Scala.ipynb
View File
samples/features/sql-big-data-cluster/spark/dataloading/hello_sparkR.ipynb → samples/features/sql-big-data-cluster/spark/data-loading/hello_sparkR.ipynb
View File
samples/features/sql-big-data-cluster/spark/data-loading/spark-sql.ipynb
+186
View File
File diff suppressed because one or more lines are too long
samples/features/sql-big-data-cluster/spark/dataloading/transform-csv-files.ipynb → samples/features/sql-big-data-cluster/spark/data-loading/transform-csv-files.ipynb
+31 -39
View File
@@ -19,7 +19,7 @@
"cells": [
{
"cell_type": "markdown",
"source": "# Spark sample showing read/write methods\nIn this sample notebook, we will read CSV file from HDFS, write it as parquet file and save a Hive table definition. We will also run some Spark SQL commands using the Hive table.\n",
"source": "# Spark sample showing read/write methods\nIn this sample notebook, we will read CSV file(s) from HDFS, write it as parquet & orc file(s) and save a Hive table definition.",
"metadata": {}
},
{
@@ -28,12 +28,30 @@
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "root\n |-- wcs_click_date_sk: integer (nullable = true)\n |-- wcs_click_time_sk: integer (nullable = true)\n |-- wcs_sales_sk: integer (nullable = true)\n |-- wcs_item_sk: integer (nullable = true)\n |-- wcs_web_page_sk: integer (nullable = true)\n |-- wcs_user_sk: integer (nullable = true)\n\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 36890| 40052| null| 4379| 34| null|\n| 36890| 41285| null| 6245| 34| null|\n| 36890| 23115| null| 13852| 34| null|\n| 36890| 17702| null| 15975| 34| null|\n| 36890| 62676| null| 2119| 34| null|\n| 36890| 34267| null| 10273| 34| null|\n| 36890| 8502| null| 17790| 34| null|\n| 36890| 54340| null| 3453| 34| null|\n| 36890| 54370| null| 6372| 34| null|\n| 36890| 6578| null| 17203| 34| null|\n| 36890| 75088| null| 4891| 34| null|\n| 36890| 23922| null| 11332| 34| null|\n| 36890| 28761| null| 4484| 34| null|\n| 36890| 21444| null| 5582| 34| null|\n| 36890| 58917| null| 8833| 34| null|\n| 36890| 27578| null| 8599| 34| null|\n| 36890| 8059| null| 6720| 34| null|\n| 36890| 43008| null| 17175| 34| null|\n| 36890| 4378| null| 10644| 34| null|\n| 36890| 55403| null| 8139| 34| null|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows",
"output_type": "stream"
"text": "Starting Spark application\n"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>0</td><td>application_1555189187089_0001</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"http://master-0.master-svc:8088/proxy/application_1555189187089_0001/\">Link</a></td><td><a target=\"_blank\" href=\"http://storage-0-1.storage-0-svc.demo-ctp25.svc.cluster.local:8042/node/containerlogs/container_1555189187089_0001_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": "SparkSession available as 'spark'.\n"
},
{
"output_type": "stream",
"name": "stdout",
"text": "root\n |-- wcs_click_date_sk: integer (nullable = true)\n |-- wcs_click_time_sk: integer (nullable = true)\n |-- wcs_sales_sk: integer (nullable = true)\n |-- wcs_item_sk: integer (nullable = true)\n |-- wcs_web_page_sk: integer (nullable = true)\n |-- wcs_user_sk: integer (nullable = true)\n\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 36890| 40052| null| 4379| 34| null|\n| 36890| 41285| null| 6245| 34| null|\n| 36890| 23115| null| 13852| 34| null|\n| 36890| 17702| null| 15975| 34| null|\n| 36890| 62676| null| 2119| 34| null|\n| 36890| 34267| null| 10273| 34| null|\n| 36890| 8502| null| 17790| 34| null|\n| 36890| 54340| null| 3453| 34| null|\n| 36890| 54370| null| 6372| 34| null|\n| 36890| 6578| null| 17203| 34| null|\n| 36890| 75088| null| 4891| 34| null|\n| 36890| 23922| null| 11332| 34| null|\n| 36890| 28761| null| 4484| 34| null|\n| 36890| 21444| null| 5582| 34| null|\n| 36890| 58917| null| 8833| 34| null|\n| 36890| 27578| null| 8599| 34| null|\n| 36890| 8059| null| 6720| 34| null|\n| 36890| 43008| null| 17175| 34| null|\n| 36890| 4378| null| 10644| 34| null|\n| 36890| 55403| null| 8139| 34| null|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows"
}
],
"execution_count": 3
"execution_count": 2
},
{
"cell_type": "code",
@@ -41,25 +59,12 @@
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "hdfs:///user/hive/warehouse",
"output_type": "stream"
"text": "hdfs:///user/hive/warehouse"
}
],
"execution_count": 4
},
{
"cell_type": "code",
"source": "# Execute Spark SQL commands\r\nsqlDF = spark.sql(\"SELECT * FROM web_clickstreams LIMIT 100\")\r\nsqlDF.show()\r\n\r\nsqlDF = spark.sql(\"SELECT wcs_user_sk, COUNT(*)\\\r\n FROM web_clickstreams\\\r\n WHERE wcs_user_sk IS NOT NULL\\\r\n GROUP BY wcs_user_sk\\\r\n ORDER BY COUNT(*) DESC LIMIT 100\")\r\nsqlDF.show()",
"metadata": {},
"outputs": [
{
"name": "stdout",
"text": "+-----------------+-----------------+------------+-----------+---------------+-----------+\n|wcs_click_date_sk|wcs_click_time_sk|wcs_sales_sk|wcs_item_sk|wcs_web_page_sk|wcs_user_sk|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\n| 37506| 7933| null| 1384| 2| 39437|\n| 37506| 56044| null| 14689| 2| 26419|\n| 37506| 52706| null| 8541| 2| 44016|\n| 37506| 67325| null| 16129| 2| 83371|\n| 37506| 84857| null| 1869| 2| 13090|\n| 37506| 49599| null| 2994| 2| 8940|\n| 37506| 78150| null| 11392| 2| 65633|\n| 37506| 38720| null| 14366| 2| 22281|\n| 37506| 79915| null| 11102| 2| 81755|\n| 37506| 67253| null| 5380| 2| 46868|\n| 37506| 6507| null| 6813| 2| 49363|\n| 37506| 18280| null| 1458| 2| 49363|\n| 37506| 72258| null| 2869| 2| 67756|\n| 37506| 8045| null| 615| 2| 86035|\n| 37506| 86164| null| 7000| 2| 94821|\n| 37506| 29724| null| 2767| 2| 94821|\n| 37506| 55471| null| 3584| 2| 62792|\n| 37506| 677| null| 1720| 2| 27212|\n| 37506| 66638| null| 9898| 2| 20370|\n| 37506| 48515| null| 9394| 2| 17157|\n+-----------------+-----------------+------------+-----------+---------------+-----------+\nonly showing top 20 rows\n\n+-----------+--------+\n|wcs_user_sk|count(1)|\n+-----------+--------+\n| 65042| 832|\n| 55928| 821|\n| 15570| 791|\n| 31138| 788|\n| 68188| 784|\n| 88205| 760|\n| 15678| 757|\n| 48063| 741|\n| 77518| 741|\n| 92978| 728|\n| 82129| 727|\n| 21700| 725|\n| 69707| 724|\n| 38895| 719|\n| 97643| 716|\n| 74426| 707|\n| 7813| 704|\n| 49528| 700|\n| 55766| 698|\n| 54355| 697|\n+-----------+--------+\nonly showing top 20 rows",
"output_type": "stream"
}
],
"execution_count": 5
"execution_count": 3
},
{
"cell_type": "code",
@@ -67,32 +72,19 @@
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "root\n |-- pr_review_sk: integer (nullable = true)\n |-- pr_review_content: string (nullable = true)\n\n+------------+--------------------+\n|pr_review_sk| pr_review_content|\n+------------+--------------------+\n| 72621|Works fine. Easy ...|\n| 89334|great product to ...|\n| 89335|Next time will go...|\n| 84259|Great Gift Great ...|\n| 84398|After trip to Par...|\n| 66434|Simply the best t...|\n| 66501|This is the exact...|\n| 66587|Not super magnet;...|\n| 66680|Installed as bath...|\n| 66694|Our home was buil...|\n| 84489|Hi ;We are runnin...|\n| 79052|Terra cotta is th...|\n| 73034|One of my fingern...|\n| 73298|We installed thes...|\n| 66810|needed silicone c...|\n| 66912|Great Gift Great ...|\n| 67028|Laguiole knives a...|\n| 89770|Good sound timers...|\n| 84679|AWESOME FEEDBACK ...|\n| 84953|love the retro gl...|\n+------------+--------------------+\nonly showing top 20 rows",
"output_type": "stream"
"text": "root\n |-- pr_review_sk: integer (nullable = true)\n |-- pr_review_content: string (nullable = true)\n\n+------------+--------------------+\n|pr_review_sk| pr_review_content|\n+------------+--------------------+\n| 72621|Works fine. Easy ...|\n| 89334|great product to ...|\n| 89335|Next time will go...|\n| 84259|Great Gift Great ...|\n| 84398|After trip to Par...|\n| 66434|Simply the best t...|\n| 66501|This is the exact...|\n| 66587|Not super magnet;...|\n| 66680|Installed as bath...|\n| 66694|Our home was buil...|\n| 84489|Hi ;We are runnin...|\n| 79052|Terra cotta is th...|\n| 73034|One of my fingern...|\n| 73298|We installed thes...|\n| 66810|needed silicone c...|\n| 66912|Great Gift Great ...|\n| 67028|Laguiole knives a...|\n| 89770|Good sound timers...|\n| 84679|AWESOME FEEDBACK ...|\n| 84953|love the retro gl...|\n+------------+--------------------+\nonly showing top 20 rows"
}
],
"execution_count": 6
"execution_count": 5
},
{
"cell_type": "code",
"source": "# Save results as parquet, and orc formats and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"product_reviews\")\r\nresults.write.format(\"orc\").mode(\"overwrite\").saveAsTable(\"product_reviews_orc\")\r\n",
"source": "# Save results as parquet, and orc formats and create hive table\r\nresults.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"product_reviews\")\r\nresults.write.format(\"orc\").mode(\"overwrite\").saveAsTable(\"product_reviews_orc\")",
"metadata": {},
"outputs": [],
"execution_count": 7
},
{
"cell_type": "code",
"source": "# Execute Spark SQL commands\r\nsqlDF = spark.sql(\"SELECT pr_review_sk, CHAR_LENGTH(pr_review_content) as len FROM product_reviews LIMIT 100\")\r\nsqlDF.show()",
"metadata": {},
"outputs": [
{
"name": "stdout",
"text": "+------------+----+\n|pr_review_sk| len|\n+------------+----+\n| 26035| 876|\n| 26037| 109|\n| 26038| 478|\n| 26041| 106|\n| 26043| 332|\n| 26044| 487|\n| 26045| 428|\n| 26048| 87|\n| 26049| 118|\n| 26051|2906|\n| 26053| 464|\n| 26054| 212|\n| 26059| 191|\n| 26060| 207|\n| 26061| 515|\n| 26063| 59|\n| 26069| 487|\n| 26070| 160|\n| 26071| 380|\n| 26072| 234|\n+------------+----+\nonly showing top 20 rows",
"output_type": "stream"
}
],
"execution_count": 8
"execution_count": 6
}
]
}}
}
samples/features/sql-big-data-cluster/spark/spark_to_sql/spark_to_sql_jdbc.ipynb → samples/features/sql-big-data-cluster/spark/data-virtualization/spark_to_sql_jdbc.ipynb
View File
samples/features/sql-big-data-cluster/spark/spark_to_sql/mssql_spark_connector.ipynb
+127
View File
File diff suppressed because one or more lines are too long