github/sql-server-samples
1
0
Fork 0
mirror of https://github.com/Microsoft/sql-server-samples.git synced 2025-12-08 14:58:54 +00:00
Code Issues Releases Wiki Activity

Added new notebooks to IQP demos

Browse Source
This commit is contained in:
pmasl
2019-11-04 22:21:09 -08:00
parent 84b793edec
commit 51a6e8adc1
7 changed files with 6890 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/features/intelligent-query-processing/notebooks/Approximate_QP.ipynb
+385
View File
@@ -0,0 +1,385 @@
{
"metadata": {
"kernelspec": {
"name": "SQL",
"display_name": "SQL",
"language": "sql"
},
"language_info": {
"name": "sql",
"version": ""
}
},
"nbformat_minor": 2,
"nbformat": 4,
"cells": [
{
"cell_type": "markdown",
"source": [
"# Approximate Query Processing\r\n",
"Approximate Query Processing is a new feature family. It aggregates across large datasets where responsiveness is more critical than absolute precision. In this first version, a new T-SQL aggregate function `APPROX_COUNT_DISTINCT` returns the approximate number of unique non-null values in a group, and is a feature under the [**Intelligent Query Processing**](https://aka.ms/iqp) suite of features.\r\n",
"\r\n",
"**Note:** Being new T-SQL notation, `APPROX_COUNT_DISTINCT` can be used even if not upgrading to **Database Compatibility Level 150** .\r\n",
"\r\n",
"The new `APPROX_COUNT_DISTINCT` function implementation guarantees up to a 2% error rate within a 97% probability. This is appropriate for dashboard scenarios and trend analysis against big data sets with many distinct values (for example, distinct orders counts over a time period) – and many concurrent users, where exact values are not necessary. However, this should not be used with applications where an exact value is required.\r\n",
"\r\n",
"\r\n",
"\r\n",
"More information about this feature is available [here](https://docs.microsoft.com/sql/relational-databases/performance/intelligent-query-processing?view=sql-server-ver15#approximate-query-processing)."
],
"metadata": {
"azdata_cell_guid": "7870ea3f-b017-46d9-bec6-15369f4ade69"
}
},
{
"cell_type": "markdown",
"source": [
"## Step 1: Setup WideWorldImportersDW database\r\n",
"\r\n",
"You could choose to use a container to evaluate this feature. Create an instance of SQL Server 2019 using a Docker image and restore the WideWorldImportersDW database backup\r\n",
"\r\n",
"You will need the **WideWorldImportersDW** database for this exercise. If you don't have this sample database, then you download the sample database [here](https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImportersDW-Full.bak \"WideWorldImportersDW-Full download\").\r\n",
"\r\n",
"Restore the copied WideWorldImportersDW database backup into the container and restore the backup.\r\n",
"\r\n",
"##### Docker Commands\r\n",
"```\r\n",
"docker pull mcr.microsoft.com/mssql/server:2019-latest\r\n",
"\r\n",
"docker run -e \"ACCEPT_EULA=Y\" -e \"SA_PASSWORD=`<A Strong Password`>\" -p 1445:1433 --name sql2019demo -d mcr.microsoft.com/mssql/server:2019-latest\r\n",
"\r\n",
"docker cp \".\\Downloads\\WideWorldImportersDW-Full.bak\" sql2019demo:/var/opt/mssql/data\r\n",
"```\r\n",
"\r\n",
"**Note**: *For Linux installations the default path to use is /var/opt/mssql*\r\n",
""
],
"metadata": {
"azdata_cell_guid": "975f5a9f-300f-450b-8981-cebb9ed6c719"
}
},
{
"cell_type": "code",
"source": [
"USE [master]\r\n",
"GO\r\n",
"IF EXISTS (SELECT [database_id] FROM sys.databases WHERE [name] = 'WideWorldImportersDW')\r\n",
"ALTER DATABASE [WideWorldImportersDW] SET SINGLE_USER WITH ROLLBACK IMMEDIATE\r\n",
"GO\r\n",
"\r\n",
"DECLARE @datafilepath VARCHAR(8000) = CAST(SERVERPROPERTY('InstanceDefaultDataPath') AS VARCHAR(4000)) + 'WideWorldImportersDW.mdf'\r\n",
"DECLARE @logfilepath VARCHAR(8000) = CAST(SERVERPROPERTY('InstanceDefaultLogPath') AS VARCHAR(4000)) + 'WideWorldImportersDW.ldf'\r\n",
"DECLARE @inmemfilepath VARCHAR(8000) = CAST(SERVERPROPERTY('InstanceDefaultDataPath') AS VARCHAR(4000)) + 'WideWorldImportersDW_InMemory_Data_1'\r\n",
"DECLARE @secondaryfilepath VARCHAR(8000) = CAST(SERVERPROPERTY('InstanceDefaultDataPath') AS VARCHAR(4000))+ 'WideWorldImportersDW_2.ndf'\r\n",
"\r\n",
"-- Change @backupfile file path as needed\r\n",
"DECLARE @backupfile VARCHAR(8000) = 'E:\\SampleDBs\\WideWorldImportersDW-Full.bak'\r\n",
"RESTORE DATABASE WideWorldImportersDW\r\n",
"FROM DISK = @backupfile \r\n",
"WITH MOVE 'WWI_Primary' TO @datafilepath,\r\n",
" MOVE 'WWI_UserData' TO @secondaryfilepath,\r\n",
" MOVE 'WWIDW_InMemory_Data_1' TO @inmemfilepath,\r\n",
" MOVE 'WWI_Log' TO @logfilepath, NOUNLOAD, REPLACE, STATS = 10\r\n",
"GO\r\n",
"\r\n",
"USE [master]\r\n",
"GO\r\n",
"ALTER DATABASE [WideWorldImportersDW] MODIFY FILE ( NAME = N'WWI_Log', SIZE = 4GB )\r\n",
"GO"
],
"metadata": {
"azdata_cell_guid": "b48b5a47-55d7-4c3c-ba51-2efa8d801029"
},
"outputs": [],
"execution_count": 7
},
{
"cell_type": "markdown",
"source": [
"## Step 2: Enlarge the WideWorldImportersDW database"
],
"metadata": {
"azdata_cell_guid": "4e07d47f-3833-40dd-8203-1d43af8bacd8"
}
},
{
"cell_type": "code",
"source": [
"USE WideWorldImportersDW;\r\n",
"GO\r\n",
"\r\n",
"/*\r\n",
"Assumes a fresh restore of WideWorldImportersDW\r\n",
"*/\r\n",
"\r\n",
"IF OBJECT_ID('Fact.OrderHistory') IS NULL \r\n",
"BEGIN\r\n",
" SELECT [Order Key], [City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key]\r\n",
" INTO Fact.OrderHistory\r\n",
" FROM Fact.[Order];\r\n",
"END;\r\n",
"\r\n",
"ALTER TABLE Fact.OrderHistory\r\n",
"ADD CONSTRAINT PK_Fact_OrderHistory PRIMARY KEY NONCLUSTERED ([Order Key] ASC, [Order Date Key] ASC) WITH (DATA_COMPRESSION = PAGE);\r\n",
"GO\r\n",
"\r\n",
"CREATE INDEX IX_Stock_Item_Key\r\n",
"ON Fact.OrderHistory ([Stock Item Key])\r\n",
"INCLUDE(Quantity)\r\n",
"WITH (DATA_COMPRESSION = PAGE);\r\n",
"GO\r\n",
"\r\n",
"CREATE INDEX IX_OrderHistory_Quantity\r\n",
"ON Fact.OrderHistory ([Quantity])\r\n",
"INCLUDE([Order Key])\r\n",
"WITH (DATA_COMPRESSION = PAGE);\r\n",
"GO\r\n",
"\r\n",
"CREATE INDEX IX_OrderHistory_CustomerKey\r\n",
"ON Fact.OrderHistory([Customer Key])\r\n",
"INCLUDE ([Total Including Tax])\r\n",
"WITH (DATA_COMPRESSION = PAGE);\r\n",
"GO\r\n",
"\r\n",
"IF (SELECT COUNT(*) FROM [Fact].[OrderHistory]) < 3702592\r\n",
"BEGIN\r\n",
"\tDECLARE @i smallint\r\n",
"\tSET @i = 0\r\n",
"\tWHILE @i < 4\r\n",
"\tBEGIN\r\n",
"\t\tINSERT INTO [Fact].[OrderHistory] ([City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key])\r\n",
"\t\tSELECT [City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key]\r\n",
"\t\tFROM [Fact].[OrderHistory];\r\n",
"\r\n",
"\t\tSET @i = @i +1\r\n",
"\tEND;\r\n",
"END\r\n",
"GO\r\n",
"\r\n",
"IF OBJECT_ID('Fact.OrderHistoryExtended') IS NULL \r\n",
"BEGIN\r\n",
" SELECT [Order Key], [City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key]\r\n",
" INTO Fact.OrderHistoryExtended\r\n",
" FROM Fact.[OrderHistory];\r\n",
"END;\r\n",
"\r\n",
"ALTER TABLE Fact.OrderHistoryExtended\r\n",
"ADD CONSTRAINT PK_Fact_OrderHistoryExtended PRIMARY KEY NONCLUSTERED ([Order Key] ASC, [Order Date Key] ASC)\r\n",
"WITH (DATA_COMPRESSION = PAGE);\r\n",
"GO\r\n",
"\r\n",
"CREATE INDEX IX_Stock_Item_Key\r\n",
"ON Fact.OrderHistoryExtended ([Stock Item Key])\r\n",
"INCLUDE (Quantity);\r\n",
"GO\r\n",
"\r\n",
"IF (SELECT COUNT(*) FROM [Fact].[OrderHistory]) < 29620736\r\n",
"BEGIN\r\n",
"\tDECLARE @i smallint\r\n",
"\tSET @i = 0\r\n",
"\tWHILE @i < 3\r\n",
"\tBEGIN\r\n",
"\t\tINSERT Fact.OrderHistoryExtended([City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key])\r\n",
"\t\tSELECT [City Key], [Customer Key], [Stock Item Key], [Order Date Key], [Picked Date Key], [Salesperson Key], [Picker Key], [WWI Order ID], [WWI Backorder ID], Description, Package, Quantity, [Unit Price], [Tax Rate], [Total Excluding Tax], [Tax Amount], [Total Including Tax], [Lineage Key]\r\n",
"\t\tFROM Fact.OrderHistoryExtended;\r\n",
"\r\n",
"\t\tSET @i = @i +1\r\n",
"\tEND;\r\n",
"END\r\n",
"GO\r\n",
"\r\n",
"UPDATE Fact.OrderHistoryExtended\r\n",
"SET [WWI Order ID] = [Order Key];\r\n",
"GO\r\n",
"\r\n",
"-- Repeat the following until log shrinks. These demos don't require much log space.\r\n",
"CHECKPOINT\r\n",
"GO\r\n",
"DBCC SHRINKFILE (N'WWI_Log' , 0, TRUNCATEONLY)\r\n",
"GO\r\n",
"SELECT * FROM sys.dm_db_log_space_usage\r\n",
"GO"
],
"metadata": {
"azdata_cell_guid": "a7421321-3652-49c5-b83a-e25e69c48aed"
},
"outputs": [],
"execution_count": 3
},
{
"cell_type": "markdown",
"source": [
"## Step 3: Execute a query with COUNT(DISTINCT) and APPROX_COUNT_DISTINCT()\r\n",
"\r\n",
"Note that we are purposefully disabling Batch Mode on Rowstore, so that we isolate and measure only the effects of executing a query with `COUNT(DISTINCT)` vs. `APPROX_COUNT_DISTINCT()`.\r\n",
""
],
"metadata": {
"azdata_cell_guid": "cc613f99-f3c2-462f-93df-bd197de31bbc"
}
},
{
"cell_type": "code",
"source": [
"USE [WideWorldImportersDW];\r\n",
"GO\r\n",
"ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;\r\n",
"GO\r\n",
"\r\n",
"SELECT COUNT(DISTINCT [WWI Order ID])\r\n",
"FROM [Fact].[OrderHistoryExtended]\r\n",
"OPTION (USE HINT('DISALLOW_BATCH_MODE'), RECOMPILE);\r\n",
"GO"
],
"metadata": {
"azdata_cell_guid": "71483984-6bdd-4866-8ddc-e40fe692c0db"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": "Commands completed successfully."
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "Commands completed successfully."
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "(1 row affected)"
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "Total execution time: 00:00:04.633"
},
"metadata": {}
},
{
"output_type": "execute_result",
"metadata": {},
"execution_count": 5,
"data": {
"application/vnd.dataresource+json": {
"schema": {
"fields": [
{
"name": "(No column name)"
}
]
},
"data": [
{
"0": "29620736"
}
]
},
"text/html": "<table><tr><th>(No column name)</th></tr><tr><td>29620736</td></tr></table>"
}
}
],
"execution_count": 5
},
{
"cell_type": "code",
"source": [
"USE [WideWorldImportersDW];\r\n",
"GO\r\n",
"ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;\r\n",
"GO\r\n",
"\r\n",
"SELECT APPROX_COUNT_DISTINCT([WWI Order ID])\r\n",
"FROM [Fact].[OrderHistoryExtended]\r\n",
"OPTION (USE HINT('DISALLOW_BATCH_MODE'), RECOMPILE);\r\n",
"GO"
],
"metadata": {
"azdata_cell_guid": "85113504-5b74-4b31-a894-d707dcd2fabb"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": "Commands completed successfully."
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "Commands completed successfully."
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "(1 row affected)"
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/html": "Total execution time: 00:00:03.084"
},
"metadata": {}
},
{
"output_type": "execute_result",
"metadata": {},
"execution_count": 4,
"data": {
"application/vnd.dataresource+json": {
"schema": {
"fields": [
{
"name": "(No column name)"
}
]
},
"data": [
{
"0": "30382637"
}
]
},
"text/html": "<table><tr><th>(No column name)</th></tr><tr><td>30382637</td></tr></table>"
}
}
],
"execution_count": 4
},
{
"cell_type": "markdown",
"source": [
"Notice the outputs above: \r\n",
"- 29620736 rows for the exact count distinct, and 30382637 for the approximate count distinct. That's 97.49% probability right there. \r\n",
"- Elapsed time is slightly less in the approximate count distinct (~3s vs. ~4.6s). But being faster is not necessarily the goal of this feature...\r\n",
"\r\n",
"Observe the query execution plans (or actual plans) for the queries. \r\n",
"\r\n",
"**Remember:** the \"Query Cost\" percentage seen in the plans is just based on estimated cost, not on runtime information.\r\n",
"\r\n",
"![ApproxQP_Plans](./media/ApproxQP_plans.PNG)\r\n",
"\r\n",
"Specifically, the memory each query was actually granted. 2.2GB for the count distinct vs. 200 bytes for the approximate count distinct:\r\n",
"\r\n",
"![ApproxQP_properties](./media/ApproxQP_properties.PNG)\r\n",
"\r\n",
"Being so efficient in terms of consumed memory, it represents a drastic improvement for scenarios where one or more of these queries have to execute at a very high rate, and do so concurrently with other business-critical workloads. Having several concurrent queries taking 2.2GB will definitely have an impact on concurrency, whereas a query that takes 200 bytes will not have that effect."
],
"metadata": {
"azdata_cell_guid": "7898e074-2d93-472b-84a5-2e52e3e43144"
}
}
]
}
samples/features/intelligent-query-processing/notebooks/Batch_Mode_on_Rowstore.ipynb
+16 -16
View File
@@ -228,18 +228,18 @@
"GO\r\n",
"\r\n",
"-- Row mode due to hint\r\n",
"SELECT\t[Tax Rate],\r\n",
"SELECT [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key],\r\n",
"\tSUM([Quantity])\t\t\t\t\tAS SUM_QTY,\r\n",
"\tSUM([Unit Price])\t\t\t\tAS SUM_BASE_PRICE,\r\n",
"\tCOUNT(*)\t\t\t\t\tAS COUNT_ORDER\r\n",
"FROM\t[Fact].[OrderHistoryExtended]\r\n",
"WHERE\t[Order Date Key]\t<= dateadd(dd, -73, '2015-11-13')\r\n",
"GROUP\tBY\t[Tax Rate],\r\n",
"\tSUM([Quantity]) AS SUM_QTY,\r\n",
"\tSUM([Unit Price]) AS SUM_BASE_PRICE,\r\n",
"\tCOUNT(*) AS COUNT_ORDER\r\n",
"FROM [Fact].[OrderHistoryExtended]\r\n",
"WHERE [Order Date Key]\t<= dateadd(dd, -73, '2015-11-13')\r\n",
"GROUP BY [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key]\r\n",
"ORDER\tBY\t[Tax Rate],\r\n",
"ORDER BY [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key]\r\n",
"OPTION (RECOMPILE, USE HINT('DISALLOW_BATCH_MODE'));\r\n",
@@ -1161,18 +1161,18 @@
"USE [WideWorldImportersDW]\r\n",
"GO\r\n",
"-- Batch mode on rowstore eligible\r\n",
"SELECT\t[Tax Rate],\r\n",
"SELECT [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key],\r\n",
"\tSUM([Quantity])\t\t\t\t\tAS SUM_QTY,\r\n",
"\tSUM([Unit Price])\t\t\t\tAS SUM_BASE_PRICE,\r\n",
"\tCOUNT(*)\t\t\t\t\tAS COUNT_ORDER\r\n",
"FROM\t[Fact].[OrderHistoryExtended]\r\n",
"WHERE\t[Order Date Key]\t<= dateadd(dd, -73, '2015-11-13')\r\n",
"GROUP\tBY\t[Tax Rate],\r\n",
"\tSUM([Quantity]) AS SUM_QTY,\r\n",
"\tSUM([Unit Price]) AS SUM_BASE_PRICE,\r\n",
"\tCOUNT(*) AS COUNT_ORDER\r\n",
"FROM [Fact].[OrderHistoryExtended]\r\n",
"WHERE [Order Date Key]\t<= dateadd(dd, -73, '2015-11-13')\r\n",
"GROUP BY [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key]\r\n",
"ORDER\tBY\t[Tax Rate],\r\n",
"ORDER BY [Tax Rate],\r\n",
"\t\t[Lineage Key],\r\n",
"\t\t[Salesperson Key]\r\n",
"OPTION (RECOMPILE);"
samples/features/intelligent-query-processing/notebooks/Table_Variable_Deferred_Compilation.ipynb
+6489
View File
File diff suppressed because one or more lines are too long
samples/features/intelligent-query-processing/notebooks/media/ApproxQP_plans.PNG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

samples/features/intelligent-query-processing/notebooks/media/ApproxQP_properties.PNG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

samples/features/intelligent-query-processing/notebooks/media/TVDC_disabled_plan.PNG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

samples/features/intelligent-query-processing/notebooks/media/TVDC_enabled_plan.PNG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB