mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Alex Protsenko
@@ -0,0 +1,85 @@
|
||||
|
||||
|
||||
# OPTIMIZE_FOR_SEQUENTIAL_KEY
|
||||
|
||||
In SQL Server 2019, a new index option was added called [OPTIMIZE_FOR_SEQUENTIAL_KEY](https://docs.microsoft.com/sql/t-sql/statements/create-index-transact-sql#sequential-keys) that is intended to address an issue known as [last page insert contention](https://support.microsoft.com/kb/4460004). Most of the solutions to this problem that have been suggested in the past involve making changes to either the application or the structure of the contentious index, which can be costly and sometimes involve performance trade-offs. Rather than making major structural changes, OPTIMIZE_FOR_SEQUENTIAL_KEY addresses some of the SQL Server scheduling issues that can lead to severely reduced throughput when last page insert contention occurs. Using the OPTMIZE_FOR_SEQUENTIAL_KEY index option can help maintain consistent throughput in high-concurrency environments when the following conditions are true:
|
||||
|
||||
- The index has a sequential key
|
||||
- The number of concurrent insert threads to the index far exceeds the number of schedulers (in other words logical cores)
|
||||
- The index has a high rate of new page allocations (page splits), which is most often due to a large row size
|
||||
|
||||
This sample illustrates how OPTIMIZE_FOR_SEQUENTIAL_KEY can be used to improve throughput on workloads that are suffering from severe last page insert contention bottlenecks.
|
||||
|
||||
### Contents
|
||||
|
||||
[About this sample](#about-this-sample)<br/>
|
||||
[Before you begin](#before-you-begin)<br/>
|
||||
[Run this sample](#run-this-sample)<br/>
|
||||
[Disclaimers](#disclaimers)<br/>
|
||||
[Related links](#related-links)<br/>
|
||||
|
||||
|
||||
<a name=about-this-sample></a>
|
||||
|
||||
## About this sample
|
||||
|
||||
- **Applies to:** SQL Server 2019 (or higher)
|
||||
- **Workload:** High-concurrency OLTP
|
||||
- **Programming Language:** T-SQL
|
||||
- **Authors:** Pam Lahoud
|
||||
- **Update history:** Created August 15, 2019
|
||||
|
||||
<a name=before-you-begin></a>
|
||||
|
||||
## Before you begin
|
||||
|
||||
To run this sample, you need the following prerequisites.
|
||||
|
||||
1. SQL Server 2019 (or higher)
|
||||
2. A server (physical or virtual) with multiple cores
|
||||
3. The [AdventureWorks2016_EXT](https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorks2016_EXT.bak) sample database
|
||||
|
||||
[!NOTE]
|
||||
> This sample was designed for a server with 8 logical cores. If you run the sample on a server with more cores, you may need to increase the number of concurrent threads in order to observe the improvement.
|
||||
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Copy the files from the root folder to a folder on the SQL Server.
|
||||
|
||||
2. Download [AdventureWorks2016_EXT.bak](https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorks2016_EXT.bak) and restore it to your SQL Server 2019 instance.
|
||||
|
||||
3. From SQL Server Management Studio or Azure Data Studio, run the Setup.sql script.
|
||||
|
||||
4. Modify the SequentialInserts_Optimized.bat and SequentialInserts_Unoptimized.bat files and change the -S parameter to point to the server where the setup script was run. For example, `-S.\SQL2019` points to an instance named SQL2019 on the local server.
|
||||
|
||||
5. Open the SQL2019_LatchWaits.htm file to open a Performance Monitor session in your default browser.
|
||||
|
||||
6. Right-click anywhere in the browser window to clear the existing data from the session.
|
||||
|
||||
7. Click the play button to start the Performance Monitor session.
|
||||
|
||||
8. From a Command Prompt, browse to the folder that contains the demo files and run SequentialInserts_Unoptimized.bat, then return to the Performance Monitor window. You should see a high number of Page Latch waits as well as high average wait times. Note the time it takes for the script to complete.
|
||||
|
||||
9. Run the SequentialInserts_Optimized.bat script from the same Command Prompt window and again return to the Performance Monitor window. This time you should see much lower number and duration of Page Latch waits, along with higher Batch requests/sec. Note the time it takes for the script to complete, it should be significantly faster than the Unoptimized script.
|
||||
|
||||
10. **OPTIONAL** - Modify the `-n256` parameter in the Optimized and Unoptimized scripts to see the effect on performance. Generally, the larger the number of concurrent sessions, the greater the improvement will be with OPTIMIZE_FOR_SEQUENTIAL_KEY.
|
||||
|
||||
|
||||
|
||||
<a name=disclaimers></a>
|
||||
|
||||
## Disclaimers
|
||||
The code included in this sample is not intended to be a set of best practices on how to build scalable enterprise grade applications. This is beyond the scope of this quick start sample.
|
||||
|
||||
<a name=related-links></a>
|
||||
|
||||
## Related Links
|
||||
|
||||
For more information, see these articles:
|
||||
|
||||
[CREATE INDEX - Sequential Keys](https://docs.microsoft.com/sql/t-sql/statements/create-index-transact-sql#sequential-keys)
|
||||
|
||||
[Behind the Scenes on OPTIMIZE_FOR_SEQUENTIAL_KEY](https://techcommunity.microsoft.com/t5/SQL-Server/Behind-the-Scenes-on-OPTIMIZE-FOR-SEQUENTIAL-KEY/ba-p/806888)
|
||||
Binary file not shown.
@@ -0,0 +1,5 @@
|
||||
ECHO OFF
|
||||
rd /s /q %temp%\output
|
||||
"ostress.exe" -E -S.\SQL2019 -dAdventureWorks2016_EXT -Q"EXEC usp_InsertLogRecord @Optimized = 1" -mstress -quiet -n1 -r1 | FINDSTR "Cantfindthisstring"
|
||||
rd /s /q %temp%\output
|
||||
"ostress.exe" -E -S.\SQL2019 -dAdventureWorks2016_EXT -Q"EXEC usp_InsertLogRecord @Optimized = 1" -mstress -quiet -n256 -r250 | FINDSTR "QEXEC Starting Creating elapsed"
|
||||
@@ -0,0 +1,5 @@
|
||||
ECHO OFF
|
||||
rd /s /q %temp%\output
|
||||
"ostress.exe" -E -S.\SQL2019 -dAdventureWorks2016_EXT -Q"EXEC usp_InsertLogRecord" -mstress -quiet -n1 -r1 | FINDSTR "Cantfindthisstring"
|
||||
rd /s /q %temp%\output
|
||||
"ostress.exe" -E -S.\SQL2019 -dAdventureWorks2016_EXT -Q"EXEC usp_InsertLogRecord" -mstress -quiet -n256 -r250 | FINDSTR "QEXEC Starting Creating elapsed"
|
||||
@@ -0,0 +1,91 @@
|
||||
USE AdventureWorks2016_EXT;
|
||||
GO
|
||||
|
||||
-- Create regular table
|
||||
|
||||
DROP TABLE IF EXISTS [dbo].[TestSequentialKey];
|
||||
GO
|
||||
|
||||
CREATE TABLE [dbo].[TestSequentialKey](
|
||||
[DatabaseLogID] [bigint] IDENTITY(1,1) NOT NULL,
|
||||
[PostTime] [datetime2] NOT NULL,
|
||||
[DatabaseUser] [sysname] NOT NULL,
|
||||
[Event] [sysname] NOT NULL,
|
||||
[Schema] [sysname] NULL,
|
||||
[Object] [sysname] NULL,
|
||||
[TSQL] [nvarchar](max) NOT NULL
|
||||
CONSTRAINT [PK_TestSequentialKey_DatabaseLogID] PRIMARY KEY NONCLUSTERED
|
||||
(
|
||||
[DatabaseLogID] ASC
|
||||
));
|
||||
|
||||
CREATE CLUSTERED INDEX CIX_TestSequentialKey_PostTime ON TestSequentialKey (PostTime);
|
||||
GO
|
||||
|
||||
-- Create optimized table
|
||||
|
||||
DROP TABLE IF EXISTS [dbo].[TestSequentialKey_Optimized];
|
||||
GO
|
||||
|
||||
CREATE TABLE [dbo].[TestSequentialKey_Optimized](
|
||||
[DatabaseLogID] [bigint] IDENTITY(1,1) NOT NULL,
|
||||
[PostTime] [datetime2] NOT NULL,
|
||||
[DatabaseUser] [sysname] NOT NULL,
|
||||
[Event] [sysname] NOT NULL,
|
||||
[Schema] [sysname] NULL,
|
||||
[Object] [sysname] NULL,
|
||||
[TSQL] [nvarchar](max) NOT NULL
|
||||
CONSTRAINT [PK_TestSequentialKey_Optimized_DatabaseLogID] PRIMARY KEY NONCLUSTERED
|
||||
(
|
||||
[DatabaseLogID] ASC
|
||||
)
|
||||
WITH (OPTIMIZE_FOR_SEQUENTIAL_KEY=ON));
|
||||
|
||||
CREATE CLUSTERED INDEX CIX_TestSequentialKey_Optimized_PostTime ON TestSequentialKey_Optimized (PostTime) WITH (OPTIMIZE_FOR_SEQUENTIAL_KEY=ON);
|
||||
GO
|
||||
|
||||
-- Create INSERT stored procedure
|
||||
|
||||
CREATE OR ALTER PROCEDURE usp_InsertLogRecord @Optimized bit = 0 AS
|
||||
|
||||
DECLARE @PostTime datetime2 = SYSDATETIME(), @User sysname, @Event sysname, @Schema sysname, @Object sysname, @TSQL nvarchar(max)
|
||||
|
||||
SELECT @User = name
|
||||
FROM sys.sysusers
|
||||
WHERE issqlrole = 0 and hasdbaccess = 1 and status = 0
|
||||
ORDER BY NEWID();
|
||||
|
||||
SELECT @Object = t.name, @Schema = s.name
|
||||
FROM sys.tables t
|
||||
INNER JOIN sys.schemas s ON s.schema_id = t.schema_id
|
||||
ORDER BY NEWID();
|
||||
|
||||
IF DATEPART(ms, @PostTime) % 4 = 0
|
||||
BEGIN
|
||||
SET @Event = N'SELECT';
|
||||
SET @TSQL = N'SELECT * FROM ' + @Schema + '.' + @Object
|
||||
END
|
||||
ELSE IF DATEPART(ms, @PostTime) % 4 = 1
|
||||
BEGIN
|
||||
SET @Event = N'INSERT';
|
||||
SET @TSQL = N'INSERT ' + @Schema + '.' + @Object + ' SELECT * FROM ' + @Schema + '.' + @Object
|
||||
END
|
||||
ELSE IF DATEPART(ms, @PostTime) % 4 = 2
|
||||
BEGIN
|
||||
SET @Event = N'UPDATE';
|
||||
SET @TSQL = N'UPDATE ' + @Schema + '.' + @Object + ' SET 1=1';
|
||||
END
|
||||
ELSE IF DATEPART(ms, @PostTime) % 4 = 3
|
||||
BEGIN
|
||||
SET @Event = N'DELETE';
|
||||
SET @TSQL = N'DELETE FROM ' + @Schema + '.' + @Object + ' WHERE 1=1';
|
||||
END
|
||||
|
||||
IF @Optimized = 1
|
||||
INSERT TestSequentialKey_Optimized (PostTime, DatabaseUser, [Event], [Schema], [Object], [TSQL])
|
||||
VALUES (@PostTime, @User, @Event, @Schema, @Object, @TSQL);
|
||||
ELSE
|
||||
INSERT TestSequentialKey (PostTime, DatabaseUser, [Event], [Schema], [Object], [TSQL])
|
||||
VALUES (@PostTime, @User, @Event, @Schema, @Object, @TSQL);
|
||||
|
||||
GO
|
||||
+1
@@ -1,2 +1,3 @@
|
||||
#!/bin/bash -e
|
||||
apt update
|
||||
apt install microsoft-mlserver-mml-r-9.3.0
|
||||
@@ -41,12 +41,14 @@ if NOT EXIST tpcxbb_1gb.bak (
|
||||
set SQLCMDSERVER=%SQL_MASTER_INSTANCE%
|
||||
set SQLCMDUSER=sa
|
||||
set SQLCMDPASSWORD=%SQL_MASTER_SA_PASSWORD%
|
||||
for /F "usebackq" %%v in (`sqlcmd -I -b -h-1 -Q "print RTRIM((CAST(SERVERPROPERTY('ProductLevel') as nvarchar(128))));"`) do SET CTP_VERSION=%%v
|
||||
if /i "%CTP_VERSION%" EQU "CTP2.4" (set MASTER_POD_NAME=mssql-master-pool-0) else (set MASTER_POD_NAME=master-0)
|
||||
for /F "usebackq tokens=1,2" %%v in (`sqlcmd -I -b -h-1 -W -Q "SET NOCOUNT ON; SELECT @@SERVERNAME, SERVERPROPERTY('IsHadrEnabled');"`) do (
|
||||
SET MASTER_POD_NAME=%%v
|
||||
SET HADR_ENABLED=%%w
|
||||
)
|
||||
|
||||
REM Copy the backup file, restore the database, create necessary objects and data file
|
||||
echo Copying sales database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp tpcxbb_1gb.bak %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data -c mssql-server || goto exit
|
||||
%DEBUG% kubectl cp tpcxbb_1gb.bak %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data/ -c mssql-server || goto exit
|
||||
|
||||
REM Download and copy the sample backup files
|
||||
if /i "%AW_WWI_SAMPLES%" EQU "--install-extra-samples" (
|
||||
@@ -57,7 +59,10 @@ if /i "%AW_WWI_SAMPLES%" EQU "--install-extra-samples" (
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/%%f" -o %%f
|
||||
)
|
||||
echo Copying %%f database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data -c mssql-server || goto exit
|
||||
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data/ -c mssql-server || goto exit
|
||||
|
||||
echo Removing database backup file...
|
||||
%DEBUG% kubectl exec %MASTER_POD_NAME% -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/%%f"
|
||||
)
|
||||
|
||||
set FILES=WideWorldImporters-Full.bak WideWorldImportersDW-Full.bak
|
||||
@@ -67,40 +72,50 @@ if /i "%AW_WWI_SAMPLES%" EQU "--install-extra-samples" (
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/%%f" -o %%f
|
||||
)
|
||||
echo Copying %%f database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data -c mssql-server || goto exit
|
||||
%DEBUG% kubectl cp %%f %CLUSTER_NAMESPACE%/%MASTER_POD_NAME%:var/opt/mssql/data/ -c mssql-server || goto exit
|
||||
|
||||
echo Removing database backup file...
|
||||
%DEBUG% kubectl exec %MASTER_POD_NAME% -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/%%f"
|
||||
)
|
||||
)
|
||||
|
||||
REM If HADR is enabled then port-forward 1533 temporarily to connect to the primary directly
|
||||
REM Default timeout for port-forward is 5 minutes so start command in background & it will terminate automatically
|
||||
if /i "%HADR_ENABLED%" EQU "1" (
|
||||
%DEBUG% start kubectl port-forward pods/%MASTER_POD_NAME% 1533:1533 -n %CLUSTER_NAMESPACE%
|
||||
SET SQLCMDSERVER=127.0.0.1,1533
|
||||
)
|
||||
|
||||
echo Configuring sample database(s)...
|
||||
%DEBUG% sqlcmd -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b -v SA_PASSWORD="%KNOX_PASSWORD%" || goto exit
|
||||
|
||||
REM remove files copied into the pod:
|
||||
echo Removing database backup files...
|
||||
%DEBUG% kubectl exec %MASTER_POD_NAME% -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
echo Removing database backup file...
|
||||
%DEBUG% kubectl exec %MASTER_POD_NAME% -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/tpcxbb_1gb.bak"
|
||||
|
||||
for %%F in (web_clickstreams inventory customer) do (
|
||||
if NOT EXIST %%F.csv (
|
||||
echo Exporting %%F data...
|
||||
if /i %%F EQU web_clickstreams (set DELIMITER=,) else (SET DELIMITER=^|)
|
||||
%DEBUG% bcp sales.dbo.%%F out "%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"!DELIMITER!" -o "%%F.out" -e "%%F.err" || goto exit
|
||||
%DEBUG% bcp sales.dbo.%%F out "%%F.csv" -S %SQLCMDSERVER% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"!DELIMITER!" -o "%%F.out" -e "%%F.err" || goto exit
|
||||
)
|
||||
)
|
||||
|
||||
if NOT EXIST product_reviews.csv (
|
||||
echo Exporting product_reviews data...
|
||||
%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit
|
||||
%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S %SQLCMDSERVER% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit
|
||||
)
|
||||
|
||||
REM Copy the data file to HDFS
|
||||
echo Uploading web_clickstreams data to HDFS...
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
|
||||
%DEBUG% curl -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
|
||||
:: del /q web_clickstreams.*
|
||||
|
||||
echo.
|
||||
echo Uploading product_reviews data to HDFS...
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
|
||||
%DEBUG% curl -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || goto exit
|
||||
%DEBUG% curl -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
|
||||
:: del /q product_reviews.*
|
||||
|
||||
REM %DEBUG% del /q *.out *.err *.csv
|
||||
@@ -122,4 +137,4 @@ goto :eof
|
||||
:usage
|
||||
echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>] [--install-extra-samples] [SQL_MASTER_PORT] [KNOX_PORT]
|
||||
echo Default ports are assumed for SQL Master instance ^& Knox gateway unless specified.
|
||||
exit /b 0
|
||||
exit /b 0
|
||||
|
||||
@@ -51,18 +51,10 @@ then
|
||||
$DEBUG curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
|
||||
fi
|
||||
|
||||
CTP_VERSION=$(sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b -h-1 -Q "print RTRIM((CAST(SERVERPROPERTY('ProductLevel') as nvarchar(128))));")
|
||||
|
||||
if [ "$CTP_VERSION" == "CTP2.4" ]
|
||||
then
|
||||
MASTER_POD_NAME=mssql-master-pool-0
|
||||
else
|
||||
MASTER_POD_NAME=master-0
|
||||
fi
|
||||
read -r MASTER_POD_NAME HADR_ENABLED <<<$(sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b -h-1 -Q "SET NOCOUNT ON; SELECT @@SERVERNAME, SERVERPROPERTY('IsHadrEnabled');")
|
||||
|
||||
echo Copying sales database backup file...
|
||||
$DEBUG kubectl cp tpcxbb_1gb.bak $CLUSTER_NAMESPACE/$MASTER_POD_NAME:var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
# $DEBUG rm tpcxbb_1gb.bak
|
||||
|
||||
if [ "$AW_WWI_SAMPLES" == "--install-extra-samples" ]
|
||||
then
|
||||
@@ -75,6 +67,9 @@ then
|
||||
fi
|
||||
echo Copying $file database backup file to SQL Master instance...
|
||||
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/$MASTER_POD_NAME:var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
|
||||
echo Removing database backup file...
|
||||
$DEBUG kubectl exec $MASTER_POD_NAME -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/$file"
|
||||
done
|
||||
|
||||
|
||||
@@ -87,17 +82,28 @@ then
|
||||
fi
|
||||
echo Copying $file database backup file to SQL Master instance...
|
||||
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/$MASTER_POD_NAME:var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
|
||||
echo Removing database backup file...
|
||||
$DEBUG kubectl exec $MASTER_POD_NAME -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/$file"
|
||||
done
|
||||
fi
|
||||
|
||||
# If HADR is enabled then port-forward 1533 temporarily to connect to the primary directly
|
||||
# Default timeout for port-forward is 5 minutes so start command in background & it will terminate automatically
|
||||
if [ "$HADR_ENABLED" == "1" ]
|
||||
then
|
||||
$DEBUG kubectl port-forward pods/$MASTER_POD_NAME 1533:1533 -n $CLUSTER_NAMESPACE &
|
||||
SQL_MASTER_INSTANCE=127.0.0.1,1533
|
||||
fi
|
||||
|
||||
echo Configuring sample database...
|
||||
# WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/SQLCMD.EXE"
|
||||
export SA_PASSWORD=$KNOX_PASSWORD
|
||||
$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b -i "$STARTUP_PATH/bootstrap-sample-db.sql" -o "bootstrap.out" || (echo $ERROR_MESSAGE && exit 2)
|
||||
|
||||
# remove files copied into the pod:
|
||||
echo Removing database backup files...
|
||||
$DEBUG kubectl exec $MASTER_POD_NAME -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
echo Removing database backup file...
|
||||
$DEBUG kubectl exec $MASTER_POD_NAME -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/tpcxbb_1gb.bak"
|
||||
|
||||
for table in web_clickstreams inventory customer
|
||||
do
|
||||
@@ -111,26 +117,26 @@ for table in web_clickstreams inventory customer
|
||||
# WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/bcp.exe"
|
||||
if [ ! -f $table.csv ]
|
||||
then
|
||||
$DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t"$DELIMITER" -e "$table.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
$DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t"$DELIMITER" -e "$table.err" > "$table.out" || (echo $ERROR_MESSAGE && exit 3)
|
||||
fi
|
||||
done
|
||||
|
||||
if [ ! -f product_reviews.csv ]
|
||||
then
|
||||
echo Exporting product_reviews data...
|
||||
$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3)
|
||||
$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" > "$table.out" || (echo $ERROR_MESSAGE && exit 3)
|
||||
fi
|
||||
|
||||
# Copy the data file to HDFS
|
||||
echo Uploading web_clickstreams data to HDFS...
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 4)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
|
||||
$DEBUG curl -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 4)
|
||||
$DEBUG curl -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
|
||||
#$DEBUG rm -f web_clickstreams.*
|
||||
|
||||
echo
|
||||
echo Uploading product_reviews data to HDFS...
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 6)
|
||||
$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
|
||||
$DEBUG curl -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 6)
|
||||
$DEBUG curl -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
|
||||
#$DEBUG rm -f product_reviews.*
|
||||
|
||||
echo
|
||||
|
||||
@@ -18,7 +18,16 @@ BEGIN
|
||||
END;
|
||||
GO
|
||||
|
||||
CREATE OR ALTER PROCEDURE #restore_database (@backup_file nvarchar(255))
|
||||
-- Enable option to allow INSERT against external table defined on HADOOP data source
|
||||
DECLARE @config_option nvarchar(100) = 'allow polybase export';
|
||||
IF NOT EXISTS(SELECT * FROM sys.configurations WHERE name = @config_option and value_in_use = 1)
|
||||
BEGIN
|
||||
EXECUTE sp_configure @config_option, 1;
|
||||
RECONFIGURE WITH OVERRIDE;
|
||||
END;
|
||||
GO
|
||||
|
||||
CREATE OR ALTER PROCEDURE #restore_database (@backup_file nvarchar(255), @db_name nvarchar(128))
|
||||
AS
|
||||
BEGIN
|
||||
DECLARE @restore_filelist_tmpl nvarchar(1000) = N'RESTORE FILELISTONLY FROM DISK = N''/var/opt/mssql/data/%F''';
|
||||
@@ -53,7 +62,7 @@ BEGIN
|
||||
INSERT INTO @files
|
||||
EXECUTE(@restore_cmd);
|
||||
|
||||
SET @restore_cmd = REPLACE(REPLACE(@restore_database_tmpl, '%F', @backup_file), '%D', LEFT(@backup_file, CHARINDEX('.', @backup_file)-1));
|
||||
SET @restore_cmd = REPLACE(REPLACE(@restore_database_tmpl, '%F', @backup_file), '%D', @db_name);
|
||||
SET @restore_cur = CURSOR FAST_FORWARD FOR SELECT LogicalName, REVERSE(LEFT(REVERSE(PhysicalName), CHARINDEX('\', REVERSE(PhysicalName))-1)) FROM @files;
|
||||
OPEN @restore_cur;
|
||||
WHILE(1=1)
|
||||
@@ -84,50 +93,58 @@ BEGIN
|
||||
WITH (LOCATION = 'sqlhdfs://controller-svc/default');
|
||||
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP3.1'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='master-svc:8032'
|
||||
);
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP3.2'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
END;
|
||||
GO
|
||||
|
||||
--- Sample dbs:
|
||||
DECLARE @sample_dbs CURSOR, @proc nvarchar(255);
|
||||
SET @sample_dbs = CURSOR FAST_FORWARD FOR
|
||||
SELECT file_or_directory_name
|
||||
FROM sys.dm_os_enumerate_filesystem('/var/opt/mssql/data', '*.bak')
|
||||
WHERE DB_ID(REPLACE(REPLACE(file_or_directory_name, 'tpcxbb_1gb', 'sales'), '.bak', '')) IS NULL;
|
||||
DECLARE @file nvarchar(260);
|
||||
SELECT file_or_directory_name, d.db_name
|
||||
FROM sys.dm_os_enumerate_filesystem('/var/opt/mssql/data', '*.bak') as f
|
||||
CROSS APPLY (VALUES(REPLACE(REPLACE(file_or_directory_name, 'tpcxbb_1gb', 'sales'), '.bak', ''))) as d(db_name)
|
||||
WHERE DB_ID(d.db_name) IS NULL;
|
||||
DECLARE @file nvarchar(260), @db_name nvarchar(128);
|
||||
OPEN @sample_dbs;
|
||||
WHILE(1=1)
|
||||
BEGIN
|
||||
FETCH @sample_dbs INTO @file;
|
||||
FETCH @sample_dbs INTO @file, @db_name;
|
||||
IF @@FETCH_STATUS < 0 BREAK;
|
||||
|
||||
-- Restore the sample databases:
|
||||
EXECUTE #restore_database @file;
|
||||
EXECUTE #restore_database @file, @db_name;
|
||||
|
||||
-- Get database name used in restore:
|
||||
SET @proc = CONCAT(QUOTENAME(LEFT(@file, CHARINDEX('.', @file)-1)), N'.sys.sp_executesql');
|
||||
SET @db_name = QUOTENAME(@db_name);
|
||||
SET @proc = CONCAT(@db_name, N'.sys.sp_executesql');
|
||||
|
||||
EXECUTE @proc N'#create_data_sources';
|
||||
|
||||
-- Set compatibility level to 150:
|
||||
EXECUTE @proc N'ALTER DATABASE CURRENT SET COMPATIBILITY_LEVEL = 150';
|
||||
|
||||
-- Rename TPCx-BB database:
|
||||
IF DB_ID('tpcxbb_1gb') IS NOT NULL
|
||||
ALTER DATABASE tpcxbb_1gb MODIFY NAME = sales;
|
||||
-- Check for HADR & add database to containedag:
|
||||
IF SERVERPROPERTY('IsHadrEnabled') = 1
|
||||
BEGIN
|
||||
DECLARE @command nvarchar(1000);
|
||||
IF EXISTS(SELECT * FROM sys.databases WHERE name = PARSENAME(@db_name,1) and recovery_model_desc = 'SIMPLE')
|
||||
BEGIN
|
||||
-- Set recovery to full
|
||||
EXECUTE @proc N'ALTER DATABASE CURRENT SET RECOVERY FULL';
|
||||
|
||||
SET @command = CONCAT(N'BACKUP DATABASE ', @db_name, ' TO DISK = ''NUL'';' );
|
||||
EXEC(@command);
|
||||
END;
|
||||
|
||||
-- Add database to AG
|
||||
SET @command = CONCAT(N'ALTER AVAILABILITY GROUP containedag ADD DATABASE ', @db_name);
|
||||
EXEC(@command);
|
||||
END;
|
||||
END;
|
||||
GO
|
||||
|
||||
|
||||
@@ -4,12 +4,8 @@ GO
|
||||
-- Create external data source for Data Pool inside a SQL big data cluster
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlDataPool')
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP3.0'
|
||||
CREATE EXTERNAL DATA SOURCE SqlDataPool
|
||||
WITH (LOCATION = 'sqldatapool://controller-svc:8080/datapools/default');
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP3.1'
|
||||
CREATE EXTERNAL DATA SOURCE SqlDataPool
|
||||
WITH (LOCATION = 'sqldatapool://controller-svc/default');
|
||||
CREATE EXTERNAL DATA SOURCE SqlDataPool
|
||||
WITH (LOCATION = 'sqldatapool://controller-svc/default');
|
||||
|
||||
-- Create external table in a data pool in SQL Server 2019 big data cluster.
|
||||
-- The SqlDataPool data source is a special data source that is available in
|
||||
|
||||
+6
-14
@@ -17,20 +17,12 @@ GO
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP3.1'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='master-svc:8032'
|
||||
);
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP3.2'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
|
||||
|
||||
-- Create file format for RCFILE with appropriate properties.
|
||||
|
||||
+6
-14
@@ -7,20 +7,12 @@ GO
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP3.1'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='master-svc:8032'
|
||||
);
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP3.2'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
|
||||
|
||||
-- Create file format for orc file with appropriate properties.
|
||||
|
||||
+6
-14
@@ -7,20 +7,12 @@ GO
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP3.1'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='master-svc:8032'
|
||||
);
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP3.2'
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://nmnode-0-svc:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='sparkhead-svc:8032'
|
||||
);
|
||||
|
||||
|
||||
-- Create file format for orc file with appropriate properties.
|
||||
|
||||
@@ -42,7 +42,7 @@ python deploy-sql-big-data-aks.py
|
||||
|
||||
When prompted, provide your input for Azure subscription ID, Azure resource group to create the resources in, and Docker credentials. Optionally, you can also provide your input for below configurations or use the defaults provided:
|
||||
- azure_region
|
||||
- vm_size - we recommend to use a VM size to accommodate your workload. For an optimal experience while you are validating basic scenarios, we recommend at least 8 vCPUs and 32GB memory across all agent nodes in the cluster. The script uses **Standard_L8s** as default.
|
||||
- vm_size - we recommend to use a VM size to accommodate your workload. For an optimal experience while you are validating basic scenarios, we recommend at least 8 vCPUs and 64GB memory across all agent nodes in the cluster. The script uses **Standard_L8s** as default. A default size configuration also uses about 24 disks for persistent volume claims across all components.
|
||||
- aks_node_count - this is the number of the worker nodes for the AKS cluster, excluding master node. The script is using a default of 1 agent node. This is the minimum required for this VM size to have enough resources and disks to provision all the necessary persistent volumes.
|
||||
- cluster_name - this value is used for both AKS cluster and SQL big data cluster created on top of AKS. Note that the name of the SQL big data cluster is going to be a Kubernetes namespace
|
||||
- password - same value is going to be used for all accounts that require user password input: SQL Server master instance SA account, controller user and Knox user
|
||||
|
||||
@@ -68,7 +68,7 @@ command="az group create --name "+GROUP_NAME+" --location "+AZURE_REGION
|
||||
executeCmd (command)
|
||||
|
||||
print("Creating AKS cluster: "+CLUSTER_NAME)
|
||||
command = "az aks create --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --generate-ssh-keys --node-vm-size "+VM_SIZE+" --node-count "+AKS_NODE_COUNT+" --kubernetes-version 1.12.8"
|
||||
command = "az aks create --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --generate-ssh-keys --node-vm-size "+VM_SIZE+" --node-count "+AKS_NODE_COUNT+" --kubernetes-version 1.13.10"
|
||||
executeCmd (command)
|
||||
|
||||
command = "az aks get-credentials --overwrite-existing --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --admin"
|
||||
@@ -78,7 +78,7 @@ print("Creating SQL Big Data cluster:" +CLUSTER_NAME)
|
||||
command="azdata bdc config init --source aks-dev-test --target custom --force"
|
||||
executeCmd (command)
|
||||
|
||||
command="azdata bdc config replace -c custom/cluster.json -j ""metadata.name=" + CLUSTER_NAME + ""
|
||||
command="azdata bdc config replace -c custom/bdc.json -j ""metadata.name=" + CLUSTER_NAME + ""
|
||||
executeCmd (command)
|
||||
|
||||
# Use this only if you are using a private registry different than default Micrososft registry (mcr).
|
||||
|
||||
@@ -7,4 +7,8 @@ This folder contains scripts that provide a template for deploying a Kubernetes
|
||||
|
||||
## __[ubuntu-single-node-vm](ubuntu-single-node-vm/)__
|
||||
|
||||
This folder contains a sample script that can be used to deploy a single-node Kubernetes cluster on a Linux machine.
|
||||
This folder contains a sample script that can be used to create a single-node Kubernetes cluster on a Linux machine and deploy SQL Server big data cluster.
|
||||
|
||||
## __[ubuntu-single-node-vm-ad](ubuntu-single-node-vm-ad/)__
|
||||
|
||||
This folder contains a sample script that can be used to create a single-node Kubernetes cluster on a Linux machine and deploy SQL Server big data cluster with Active Directory integration.
|
||||
|
||||
+13
-14
@@ -1,20 +1,19 @@
|
||||
{
|
||||
"patch": [
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "spec.pools[?(@.spec.type=='Master')].spec",
|
||||
"value": {
|
||||
"type": "Master",
|
||||
"dnsName": "mastersql.contoso.local",
|
||||
"replicas": 1,
|
||||
"endpoints": [
|
||||
{
|
||||
"name": "Master",
|
||||
"serviceType": "NodePort",
|
||||
"port": 31433
|
||||
}
|
||||
]
|
||||
}
|
||||
"op": "add",
|
||||
"path": "spec.resources.master.spec.endpoints[0].dnsName",
|
||||
"value": "mastersql.contoso.local"
|
||||
},
|
||||
{
|
||||
"op": "add",
|
||||
"path": "spec.resources.gateway.spec.endpoints[0].dnsName",
|
||||
"value": "knox.contoso.local"
|
||||
},
|
||||
{
|
||||
"op": "add",
|
||||
"path": "spec.resources.appproxy.spec.endpoints[0].dnsName",
|
||||
"value": "app.contoso.local"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
+9
-9
@@ -4,10 +4,9 @@
|
||||
"op": "add",
|
||||
"path": "security",
|
||||
"value": {
|
||||
"useInternalDomain": false,
|
||||
"ouDistinguishedName":"OU=bdc,DC=contoso,DC=local",
|
||||
"dnsIpAddresses": ["11.11.111.11"],
|
||||
"domainControllerFullyQualifiedDns": ["VM.CONTOSO.LOCAL"],
|
||||
"dnsIpAddresses": ["00.00.000.00"],
|
||||
"domainControllerFullyQualifiedDns": ["DC.CONTOSO.LOCAL"],
|
||||
"realm":"CONTOSO.LOCAL",
|
||||
"domainDnsName":"contoso.local",
|
||||
"bdcAdminPrincipals": [
|
||||
@@ -20,12 +19,13 @@
|
||||
},
|
||||
{
|
||||
"op": "add",
|
||||
"path": "spec.endpoints/0",
|
||||
"value": {
|
||||
"name": "Kerberos",
|
||||
"serviceType": "NodePort",
|
||||
"port": 30088
|
||||
}
|
||||
"path": "spec.endpoints[0].dnsName",
|
||||
"value": "controller.contoso.local"
|
||||
},
|
||||
{
|
||||
"op": "add",
|
||||
"path": "spec.endpoints[1].dnsName",
|
||||
"value": "serviceproxy.contoso.local"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
+17
-24
@@ -22,19 +22,7 @@ while true; do
|
||||
[ "$password" = "$password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
echo ""
|
||||
# Get docker credentials for private release.
|
||||
#
|
||||
read -p "Enter Docker username: " DOCKER_USERNAME
|
||||
while true; do
|
||||
read -s -p "Enter Docker Password: " docker_password
|
||||
echo
|
||||
read -s -p "Confirm Docker Password: " docker_password2
|
||||
echo
|
||||
[ "$docker_password" = "$docker_password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
export DOCKER_PASSWORD=$docker_password
|
||||
|
||||
echo ""
|
||||
|
||||
# Get Domain Service Account Username and Password.
|
||||
@@ -75,9 +63,9 @@ RETRY_INTERVAL=5
|
||||
|
||||
# Variables for pulling dockers.
|
||||
#
|
||||
export DOCKER_REGISTRY="private-repo.microsoft.com"
|
||||
export DOCKER_REPOSITORY="mssql-private-preview"
|
||||
export DOCKER_TAG="ctp3.2.1"
|
||||
export DOCKER_REGISTRY="mcr.microsoft.com"
|
||||
export DOCKER_REPOSITORY="mssql/bdc"
|
||||
export DOCKER_TAG="2019-RC1-ubuntu"
|
||||
|
||||
# Variables used for azdata cluster creation.
|
||||
#
|
||||
@@ -91,9 +79,10 @@ export STORAGE_CLASS=local-storage
|
||||
export PV_COUNT="30"
|
||||
|
||||
IMAGES=(
|
||||
mssql-app-service-proxy
|
||||
mssql-appdeploy-init
|
||||
mssql-app-service-proxy
|
||||
mssql-control-watchdog
|
||||
mssql-controller
|
||||
mssql-dns
|
||||
mssql-hadoop
|
||||
mssql-mleap-serving-runtime
|
||||
mssql-mlserver-py-runtime
|
||||
@@ -105,10 +94,13 @@ IMAGES=(
|
||||
mssql-monitor-influxdb
|
||||
mssql-monitor-kibana
|
||||
mssql-monitor-telegraf
|
||||
mssql-security-domainctl
|
||||
mssql-security-knox
|
||||
mssql-security-support
|
||||
mssql-server
|
||||
mssql-server-controller
|
||||
mssql-server-data
|
||||
mssql-server-ha
|
||||
mssql-service-proxy
|
||||
mssql-ssis-app-runtime
|
||||
)
|
||||
@@ -320,25 +312,24 @@ echo "Kubernetes master setup done."
|
||||
|
||||
# Pull docker images of SQL Server big data cluster.
|
||||
#
|
||||
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to pull docker images..."
|
||||
echo "Pulling images from repository: " $DOCKER_REGISTRY"/"$DOCKER_REPOSITORY
|
||||
|
||||
docker login $DOCKER_REGISTRY -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
|
||||
for image in "${IMAGES[@]}";
|
||||
do
|
||||
docker pull $DOCKER_REGISTRY/$DOCKER_REPOSITORY/$image:$DOCKER_TAG
|
||||
echo "Docker image" $image " pulled."
|
||||
done
|
||||
docker logout $DOCKER_REGISTRY
|
||||
echo "Docker images pulled."
|
||||
|
||||
# Deploy azdata bdc create cluster.
|
||||
#
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to deploy azdata cluster..."
|
||||
echo "Starting to deploy big data cluster..."
|
||||
|
||||
# Command to create cluster for single node cluster.
|
||||
#
|
||||
@@ -346,13 +337,15 @@ azdata bdc config init --source kubeadm-dev-test --target kubeadm-custom -f
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.repository=$DOCKER_REPOSITORY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.registry=$DOCKER_REGISTRY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.imageTag=$DOCKER_TAG"
|
||||
azdata bdc config replace -c kubeadm-custom/cluster.json -j "$.spec.pools[?(@.spec.type == "Data")].spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/bdc.json -j "$.spec.resources.data-0.spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
|
||||
azdata bdc config patch -c kubeadm-custom/control.json -p $STARTUP_PATH/security-patch.json
|
||||
azdata bdc config patch -c kubeadm-custom/cluster.json -p $STARTUP_PATH/endpoint-patch.json
|
||||
azdata bdc config patch -c kubeadm-custom/bdc.json -p $STARTUP_PATH/endpoint-patch.json
|
||||
|
||||
azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
|
||||
echo "Azdata cluster created."
|
||||
|
||||
echo "Big data cluster created."
|
||||
|
||||
# Setting context to cluster.
|
||||
#
|
||||
|
||||
+9
-5
@@ -46,7 +46,7 @@ RETRY_INTERVAL=5
|
||||
#
|
||||
export DOCKER_REGISTRY="mcr.microsoft.com"
|
||||
export DOCKER_REPOSITORY="mssql/bdc"
|
||||
export DOCKER_TAG="2019-CTP3.2-ubuntu"
|
||||
export DOCKER_TAG="2019-RC1-ubuntu"
|
||||
|
||||
# Variables used for azdata cluster creation.
|
||||
#
|
||||
@@ -60,9 +60,10 @@ export STORAGE_CLASS=local-storage
|
||||
export PV_COUNT="30"
|
||||
|
||||
IMAGES=(
|
||||
mssql-app-service-proxy
|
||||
mssql-appdeploy-init
|
||||
mssql-app-service-proxy
|
||||
mssql-control-watchdog
|
||||
mssql-controller
|
||||
mssql-dns
|
||||
mssql-hadoop
|
||||
mssql-mleap-serving-runtime
|
||||
mssql-mlserver-py-runtime
|
||||
@@ -74,10 +75,13 @@ IMAGES=(
|
||||
mssql-monitor-influxdb
|
||||
mssql-monitor-kibana
|
||||
mssql-monitor-telegraf
|
||||
mssql-security-domainctl
|
||||
mssql-security-knox
|
||||
mssql-security-support
|
||||
mssql-server
|
||||
mssql-server-controller
|
||||
mssql-server-data
|
||||
mssql-server-ha
|
||||
mssql-service-proxy
|
||||
mssql-ssis-app-runtime
|
||||
)
|
||||
@@ -313,11 +317,11 @@ azdata bdc config init --source kubeadm-dev-test --target kubeadm-custom -f
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.repository=$DOCKER_REPOSITORY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.registry=$DOCKER_REGISTRY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.imageTag=$DOCKER_TAG"
|
||||
azdata bdc config replace -c kubeadm-custom/cluster.json -j "$.spec.pools[?(@.spec.type == "Data")].spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/bdc.json -j "$.spec.resources.data-0.spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
|
||||
azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
|
||||
echo "Azdata cluster created."
|
||||
echo "Big data cluster created."
|
||||
|
||||
# Setting context to cluster.
|
||||
#
|
||||
|
||||
+24
-20
@@ -26,26 +26,30 @@ TARGET_DOCKER_USERNAME = input("Provide Docker username for the target registry:
|
||||
TARGET_DOCKER_PASSWORD = getpass.getpass("Provide Docker password for the target registry:")
|
||||
TARGET_DOCKER_TAG = input("Provide Docker tag for the images at the target: ") or "latest"
|
||||
|
||||
images = [ 'mssql-appdeploy-init',
|
||||
'mssql-monitor-fluentbit',
|
||||
'mssql-monitor-collectd',
|
||||
'mssql-server-data',
|
||||
'mssql-hadoop',
|
||||
'mssql-monitor-elasticsearch',
|
||||
'mssql-monitor-influxdb',
|
||||
'mssql-security-knox',
|
||||
'mssql-mlserver-r-runtime',
|
||||
'mssql-mlserver-py-runtime',
|
||||
'mssql-controller',
|
||||
'mssql-mleap-serving-runtime',
|
||||
'mssql-server-controller',
|
||||
'mssql-monitor-grafana',
|
||||
'mssql-monitor-kibana',
|
||||
'mssql-service-proxy',
|
||||
'mssql-app-service-proxy',
|
||||
'mssql-ssis-app-runtime',
|
||||
'mssql-monitor-telegraf',
|
||||
'mssql-security-support']
|
||||
images = ['mssql-app-service-proxy',
|
||||
'mssql-control-watchdog'
|
||||
'mssql-controller'
|
||||
'mssql-dns'
|
||||
'mssql-hadoop'
|
||||
'mssql-mleap-serving-runtime'
|
||||
'mssql-mlserver-py-runtime'
|
||||
'mssql-mlserver-r-runtime'
|
||||
'mssql-monitor-collectd'
|
||||
'mssql-monitor-elasticsearch'
|
||||
'mssql-monitor-fluentbit'
|
||||
'mssql-monitor-grafana'
|
||||
'mssql-monitor-influxdb'
|
||||
'mssql-monitor-kibana'
|
||||
'mssql-monitor-telegraf'
|
||||
'mssql-security-domainctl'
|
||||
'mssql-security-knox'
|
||||
'mssql-security-support'
|
||||
'mssql-server'
|
||||
'mssql-server-controller'
|
||||
'mssql-server-data'
|
||||
'mssql-server-ha'
|
||||
'mssql-service-proxy'
|
||||
'mssql-ssis-app-runtime']
|
||||
|
||||
# Use this only if your source is a private Docker registry
|
||||
# print("Execute docker login to source registry: " + SOURCE_DOCKER_REGISTRY)
|
||||
|
||||
@@ -9,7 +9,7 @@ One of the most popular tools for calling an API on `http:` endpoints is [curl](
|
||||
|
||||
[About this sample](#about-this-sample)<br/>
|
||||
[Build the CLR/CURL extension](#build-functions)<br/>
|
||||
[Add RegEx functions to your SQL database](#add-functions)<br/>
|
||||
[Add CURL functions to your SQL database](#add-functions)<br/>
|
||||
[Test the functions](#test)<br/>
|
||||
[Disclaimers](#disclaimers)<br/>
|
||||
[Appendix](#appendix) - quick install script for your dev environment.<br/>
|
||||
|
||||
+11
-1
@@ -18,6 +18,11 @@ select property = 'TEMPDB:'+y.v.value('local-name(.)', 'nvarchar(300)'),
|
||||
value = y.v.value('.[1]', 'nvarchar(300)')
|
||||
from @source.nodes('//tempdb') x(v)
|
||||
cross apply x.v.nodes('*') y(v)
|
||||
UNION ALL
|
||||
select property = 'INSTANCE:'+y.v.value('local-name(.)', 'nvarchar(300)'),
|
||||
value = y.v.value('.[1]', 'nvarchar(300)')
|
||||
from @source.nodes('//instance') x(v)
|
||||
cross apply x.v.nodes('*') y(v)
|
||||
),
|
||||
tgt as(
|
||||
select property = x.v.value('name[1]', 'nvarchar(300)'),
|
||||
@@ -33,6 +38,11 @@ select property = 'TEMPDB:'+y.v.value('local-name(.)', 'nvarchar(300)'),
|
||||
value = y.v.value('.[1]', 'nvarchar(300)')
|
||||
from @target.nodes('//tempdb') x(v)
|
||||
cross apply x.v.nodes('*') y(v)
|
||||
UNION ALL
|
||||
select property = 'INSTANCE:'+y.v.value('local-name(.)', 'nvarchar(300)'),
|
||||
value = y.v.value('.[1]', 'nvarchar(300)')
|
||||
from @target.nodes('//instance') x(v)
|
||||
cross apply x.v.nodes('*') y(v)
|
||||
),
|
||||
diff as (
|
||||
select property = isnull(src.property, tgt.property),
|
||||
@@ -43,7 +53,7 @@ where (src.value <> tgt.value
|
||||
or src.value is null and tgt.value is not null
|
||||
or src.value is not null and tgt.value is null)
|
||||
)
|
||||
select *
|
||||
select property, source, [target]
|
||||
from diff
|
||||
where is_missing = 0 or @verbose = 1 -- in the earlier versions you had to comment out this line. Now just set the value of the flag up
|
||||
order by property
|
||||
|
||||
+16
-2
@@ -53,6 +53,20 @@ where name in ('cost threshold for parallelism','cursor threshold','fill factor
|
||||
for xml raw, elements
|
||||
);
|
||||
set @result += (select name = 'version', value = @@VERSION for xml raw, elements)
|
||||
select cast(@result as xml);
|
||||
end;
|
||||
|
||||
set @result += isnull
|
||||
((SELECT scheduler_count, scheduler_total_count FROM sys.dm_os_sys_info
|
||||
for xml raw('instance'), elements),''
|
||||
);
|
||||
|
||||
set @result +=
|
||||
isnull((SELECT name = REPLACE([type], 'MEMORYCLERK_', 'MEMORY:')
|
||||
, value = CAST(sum(pages_kb)/1024.1/1024 AS NUMERIC(6,1))
|
||||
FROM sys.dm_os_memory_clerks
|
||||
GROUP BY type
|
||||
HAVING sum(pages_kb) /1024. /1024 > 1
|
||||
for xml raw, elements),'');
|
||||
|
||||
select cast(@result as xml);
|
||||
|
||||
end;
|
||||
|
||||
Reference in New Issue
Block a user