From d80560971d22949c42a1ce8fac72708952de10ad Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 14:39:25 +0200 Subject: [PATCH 1/7] Initial commit for README.md --- .../manage/polybase/external-table/README.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 samples/manage/polybase/external-table/README.md diff --git a/samples/manage/polybase/external-table/README.md b/samples/manage/polybase/external-table/README.md new file mode 100644 index 00000000..412fb4ae --- /dev/null +++ b/samples/manage/polybase/external-table/README.md @@ -0,0 +1,109 @@ + +![](https://github.com/microsoft/sql-server-samples/blob/master/media/solutions-microsoft-logo-small.png) + +# Handling Update Statistics Error on External Tables in SQL Server + +This sample describes an option to update statistics on SQL Server PolyBase external tables! + +## Background + +In the process of configuring a maintenance plan for a SQL Server database with external tables and external data sources for PolyBase queries, an error occurred during the update statistics task. + +## Problem Encountered + +While attempting to update statistics on an external table, the following error message was encountered: + +```sql +UPDATE STATISTICS [dbo].[ExternalTableName] WITH FULLSCAN, COLUMNS + +Message 46519, level 16, state 22 + +The object Update Statistics isn't supported on External Table +``` + +### Contents + +[About this sample](#about-this-sample)
+[Before you begin](#before-you-begin)
+[Case history](#case-history)
+[Resolution steps](#resolution-steps)
+[Further considerations](#further-considerations)
+[Outcome](#outcome)
+[Disclaimers](#disclaimers)
+[Related links](#related-links)
+ + + +## About this sample + +- **Applies to:** SQL Server 2017 (or higher) +- **Key features:** Update statistics +- **Workload:** No workload related to this sample +- **Programming Language:** T-SQL +- **Authors:** [Sergio Govoni](https://www.linkedin.com/in/sgovoni/) | [Microsoft MVP Profile](https://mvp.microsoft.com/mvp/profile/c7b770c0-3c9a-e411-93f2-9cb65495d3c4) | [Blog](https://segovoni.medium.com/) | [GitHub](https://github.com/segovoni) | [Twitter](https://twitter.com/segovoni) + + + +## Before you begin + +To run this example, the following basic concepts are required. + +[SQL Server PolyBase](https://learn.microsoft.com/sql/relational-databases/polybase/polybase-guide?WT.mc_id=DP-MVP-4029181) enables your SQL Server instance to query data with T-SQL directly from SQL Server, Oracle, Teradata, MongoDB, Hadoop clusters, Cosmos DB, and S3-compatible object storage without separately installing client connection software. If you are new to PolyBase, you can find initial information in this article: [Polybase for beginners](https://techcommunity.microsoft.com/t5/sql-server-support-blog/polybase-for-beginners/ba-p/1075336). + + + +## Case history + +Upon investigation, it became apparent that SQL Server does not support the direct updating of statistics on external tables, as documented in the [CREATE STATISTICS documentation page](https://learn.microsoft.com/sql/t-sql/statements/create-statistics-transact-sql?WT.mc_id=DP-MVP-4029181#limitations-and-restrictions). + + + +## Resolution steps + +1. Understanding the Limitation: + +Referring to the documentation page, it explicitly states, "Updating statistics is not supported on external tables. To update statistics on an external table, drop and re-create the statistics." + +2. Available Solutions: + +Option 1: Ignore External Tables: +This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions. + +Option 2: Drop and Recreate Statistics: +Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. + +3. Implementation Guide: + +- Execute `sp_drop_create_stats_external_table` to generate T-SQL statements +- Execute DROP STATISTICS statements before the maintenance statistics task +- Execute CREATE STATISTICS statements after the maintenance statistics task +- Implement robust error handling during the maintenance plan execution +- Validate the presence of statistics on the external table upon completion + + + +## Further considerations + +- When dealing with large external tables, consider the performance implications of using default sampling versus full scan options +- Note that external tables utilizing DELIMITEDTEXT, CSV, PARQUET, or DELTA as data types only support statistics for one column per CREATE STATISTICS command + + + +## Outcome + +By integrating the `sp_drop_create_stats_external_table` stored procedure into the maintenance plan, developers can effectively manage statistics on external tables within SQL Server databases, ensuring optimal performance and reliability. + +From the maintenance plan prospective, CREATE and DROP STATISTICS statements can be stored on a temporary table or working table and executed separately. DROP STATISTICS statements can be executed before the maintenance statistics task and afterward the CREATE STATISTICS statements. Pay attention to the error handling during the maintenance plan because, at the end of the maintenance, statistics have to be in place on the external table. The output of the stored procedure cannot be missed. + + + +## Disclaimers + +This code sample is provided for demonstration and educational purposes only. It is recommended to use it with caution and fully understand its implications before applying it in a production environment. The provided code may not fully reflect the best development or security practices and may require adjustments to meet specific project requirements. The author disclaims any liability for any damages resulting from the use or interpretation of this material. + + + +## Related Links + + +For more information, see these articles... \ No newline at end of file From b03edd99d03b977bce7104f557b5ed649051e1b7 Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 15:01:05 +0200 Subject: [PATCH 2/7] Cosmetic care on README.md --- .../manage/polybase/external-table/README.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/samples/manage/polybase/external-table/README.md b/samples/manage/polybase/external-table/README.md index 412fb4ae..1e8e3ea4 100644 --- a/samples/manage/polybase/external-table/README.md +++ b/samples/manage/polybase/external-table/README.md @@ -1,9 +1,9 @@ ![](https://github.com/microsoft/sql-server-samples/blob/master/media/solutions-microsoft-logo-small.png) -# Handling Update Statistics Error on External Tables in SQL Server +# Handling UPDATE STATISTICS error on SQL Server PolyBase external tables -This sample describes an option to update statistics on SQL Server PolyBase external tables! +This sample describes an option to update statistics on SQL Server PolyBase external tables. ## Background @@ -37,7 +37,7 @@ The object Update Statistics isn't supported on External Table ## About this sample - **Applies to:** SQL Server 2017 (or higher) -- **Key features:** Update statistics +- **Key features:** UPDATE STATISTICS - **Workload:** No workload related to this sample - **Programming Language:** T-SQL - **Authors:** [Sergio Govoni](https://www.linkedin.com/in/sgovoni/) | [Microsoft MVP Profile](https://mvp.microsoft.com/mvp/profile/c7b770c0-3c9a-e411-93f2-9cb65495d3c4) | [Blog](https://segovoni.medium.com/) | [GitHub](https://github.com/segovoni) | [Twitter](https://twitter.com/segovoni) @@ -60,19 +60,19 @@ Upon investigation, it became apparent that SQL Server does not support the dire ## Resolution steps -1. Understanding the Limitation: +### Understanding the Limitation Referring to the documentation page, it explicitly states, "Updating statistics is not supported on external tables. To update statistics on an external table, drop and re-create the statistics." -2. Available Solutions: +### Available Solutions -Option 1: Ignore External Tables: -This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions. +* Option 1: Ignore External Tables: + * This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions. -Option 2: Drop and Recreate Statistics: -Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. +* Option 2: Drop and Recreate Statistics: + * Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. -3. Implementation Guide: +### Implementation Guide - Execute `sp_drop_create_stats_external_table` to generate T-SQL statements - Execute DROP STATISTICS statements before the maintenance statistics task @@ -106,4 +106,4 @@ This code sample is provided for demonstration and educational purposes only. It ## Related Links -For more information, see these articles... \ No newline at end of file +For more information, see these articles... From 1c11b10d600a3ed74319180a5b3f19e32b581ada Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 15:08:53 +0200 Subject: [PATCH 3/7] Add related links --- samples/manage/polybase/external-table/README.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/samples/manage/polybase/external-table/README.md b/samples/manage/polybase/external-table/README.md index 1e8e3ea4..c8c49f51 100644 --- a/samples/manage/polybase/external-table/README.md +++ b/samples/manage/polybase/external-table/README.md @@ -9,7 +9,7 @@ This sample describes an option to update statistics on SQL Server PolyBase exte In the process of configuring a maintenance plan for a SQL Server database with external tables and external data sources for PolyBase queries, an error occurred during the update statistics task. -## Problem Encountered +## Problem encountered While attempting to update statistics on an external table, the following error message was encountered: @@ -60,11 +60,11 @@ Upon investigation, it became apparent that SQL Server does not support the dire ## Resolution steps -### Understanding the Limitation +### Understanding the limitation Referring to the documentation page, it explicitly states, "Updating statistics is not supported on external tables. To update statistics on an external table, drop and re-create the statistics." -### Available Solutions +### Available solutions * Option 1: Ignore External Tables: * This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions. @@ -72,7 +72,7 @@ Referring to the documentation page, it explicitly states, "Updating statistics * Option 2: Drop and Recreate Statistics: * Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. -### Implementation Guide +### Implementation guide - Execute `sp_drop_create_stats_external_table` to generate T-SQL statements - Execute DROP STATISTICS statements before the maintenance statistics task @@ -103,7 +103,11 @@ This code sample is provided for demonstration and educational purposes only. It -## Related Links +## Related links -For more information, see these articles... +For more information, see these articles: + +- [Data virtualization with PolyBase in SQL Server](https://learn.microsoft.com/sql/relational-databases/polybase/polybase-guide?WT.mc_id=DP-MVP-4029181) +- [UPDATE STATISTICS (Transact-SQL)](https://learn.microsoft.com/sql/t-sql/statements/update-statistics-transact-sql?WT.mc_id=DP-MVP-4029181) +- [External tables: Why create statistics?](https://learn.microsoft.com/answers/questions/978155/external-tables-why-create-statistics?WT.mc_id=DP-MVP-4029181) From e299b4b35830d1ce9d301ce00c304059fe9eec53 Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 15:24:57 +0200 Subject: [PATCH 4/7] Initial commit sp_drop_create_stats_external_table --- .../sp-drop-create-stats-external-table.sql | 213 ++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 samples/manage/polybase/external-table/source/sp-drop-create-stats-external-table.sql diff --git a/samples/manage/polybase/external-table/source/sp-drop-create-stats-external-table.sql b/samples/manage/polybase/external-table/source/sp-drop-create-stats-external-table.sql new file mode 100644 index 00000000..d937266f --- /dev/null +++ b/samples/manage/polybase/external-table/source/sp-drop-create-stats-external-table.sql @@ -0,0 +1,213 @@ +------------------------------------------------------------------------ +-- Project: sp_drop_create_stats_external_table -- +-- -- +-- The stored procedure is able to generate all the -- +-- T-SQL statements for drop and create statistics -- +-- defined on SQL Server external tables in your -- +-- database! -- +-- -- +-- File: Stored procedure implementation -- +-- Author: Sergio Govoni https://www.linkedin.com/in/sgovoni/ -- +-- Notes: -- -- +------------------------------------------------------------------------ + + +IF OBJECT_ID('dbo.sp_drop_create_stats_external_table', 'P') IS NOT NULL + DROP PROCEDURE dbo.sp_drop_create_stats_external_table; +GO + +CREATE PROCEDURE dbo.sp_drop_create_stats_external_table +AS BEGIN + /* + Author: Sergio Govoni https://www.linkedin.com/in/sgovoni/ + Version: 1.0 + License: MIT License + */ + + DECLARE + -- Output table + @DropCreateSQLCmd TABLE + ( + ID INTEGER IDENTITY(1, 1) NOT NULL + ,SchemaName SYSNAME NOT NULL + ,TableName SYSNAME NOT NULL + ,ObjectType SYSNAME NOT NULL + ,OperationType NCHAR(1) NOT NULL + ,SQLCmd NVARCHAR(1024) NOT NULL + ); + + -- Generate CREATE STATISTICS statements + WITH StatCreate AS + ( + SELECT + 'A' AS RowType + ,T.[object_id] + ,T.stats_id + ,T.StatLevel + ,T.KeyOrdinal + ,T.SchemaName + ,T.TableName + ,CAST('CREATE ' + + 'STATISTICS [' + TRIM(T.StatsName) + + '] ON [' + TRIM(T.SchemaName) + + '].[' + TRIM(T.TableName) + + '] ( ' AS VARCHAR(MAX)) AS SQLCmd + FROM + ( + SELECT + DISTINCT + stat.[object_id] + ,stat.stats_id + ,CAST(0 AS INTEGER) AS StatLevel + ,CAST(0 AS INTEGER) AS KeyOrdinal + ,stat.name AS StatsName + ,sch.name AS SchemaName + ,obj.name AS TableName + FROM + sys.stats_columns AS statc + JOIN + sys.stats AS stat ON ((stat.stats_id = statc.stats_id) + AND (stat.[object_id] = statc.[object_id])) + JOIN + sys.objects AS obj ON statc.[object_id] = obj.[object_id] + JOIN + sys.external_tables external_tab ON ((external_tab.[object_id] = obj.[object_id]) + AND (external_tab.[schema_id] = obj.[schema_id])) + JOIN + sys.columns AS col ON ((col.column_id = statc.column_id) + AND (col.[object_id] = statc.[object_id])) + JOIN + sys.schemas AS sch ON obj.[schema_id] = sch.[schema_id] + WHERE + ((stat.auto_created = 1) OR (stat.user_created = 1)) + AND (obj.type = 'U') + ) AS T + + UNION ALL + + SELECT + 'R' AS RowType + ,statcol.[object_id] + ,statcol.stats_id + ,CAST(S.StatLevel + 1 AS INTEGER) AS IdxLevel + ,CAST(statcol.stats_column_id AS INTEGER) AS KeyOrdinal + ,S.SchemaName + ,S.TableName + ,CAST(S.SQLCmd + CASE(statcol.stats_column_id) WHEN 1 THEN '' ELSE ',' END + + '[' + TRIM(col.name) + + '] ' AS VARCHAR(MAX)) AS SQLCmd + FROM + StatCreate AS S + JOIN + sys.stats_columns AS statcol ON ((statcol.[object_id] = S.[object_id]) + AND (statcol.stats_id = S.stats_id)) + JOIN + sys.columns AS col ON ((col.column_id = statcol.column_id) + AND (col.[object_id] = statcol.[object_id])) + WHERE + (statcol.stats_column_id = (S.KeyOrdinal + 1)) + ), + StatCreateGroup AS + ( + SELECT + MAX(S.KeyOrdinal) AS MaxKeyOrdinal + ,S.[object_id] + ,S.stats_id + FROM + StatCreate AS S + JOIN + sys.objects AS O ON O.[object_id] = S.[object_id] + WHERE + (S.RowType = 'R') + GROUP BY + S.[object_id] + ,S.stats_id + ) + INSERT INTO @DropCreateSQLCmd + ( + SchemaName + ,TableName + ,ObjectType + ,OperationType + ,SQLCmd + ) + SELECT + StatCreate.SchemaName + ,StatCreate.TableName + ,'STATS' AS ObjecType + ,'C' AS OperationType + ,StatCreate.SQLCmd + ') WITH FULLSCAN;' AS SQLCmd + FROM + StatCreateGroup + JOIN + StatCreate ON ((StatCreate.[object_id] = StatCreateGroup.[object_id]) + AND (StatCreate.stats_id = StatCreateGroup.stats_id)) + AND (StatCreate.KeyOrdinal = StatCreateGroup.MaxKeyOrdinal); + + -- Generate DROP STATISTICS statements + WITH StatsDrop AS + ( + SELECT + T.SchemaName + ,T.TableName + ,'STATS' AS ObjectType + ,'D' AS OperationType + ,'DROP STATISTICS [' + + TRIM(SchemaName) + '].[' + + TRIM(TableName) + '].[' + + TRIM(StatisticName) + '];' AS SQLCmd + + FROM ( + SELECT + sch.[Name] as SchemaName + ,obj.[Name] as TableName + ,stat.[Name] as StatisticName + FROM + sys.stats AS stat + INNER JOIN + sys.objects AS obj ON stat.[object_id] = obj.[object_id] + INNER JOIN + sys.external_tables external_tab ON ((external_tab.[object_id] = obj.[object_id]) + AND (external_tab.[schema_id] = obj.[schema_id])) + INNER JOIN + sys.schemas AS sch ON obj.[schema_id] = sch.[schema_id] + WHERE + ((stat.auto_created = 1) OR (stat.user_created = 1)) + AND (obj.type = 'U') + ) AS T + ) + INSERT INTO @DropCreateSQLCmd + ( + SchemaName + ,TableName + ,ObjectType + ,OperationType + ,SQLCmd + ) + SELECT + SchemaName + ,TableName + ,ObjectType + ,OperationType + ,SQLCmd + FROM + StatsDrop; + + SELECT + ID + ,SchemaName + ,TableName + ,ObjectType + ,OperationType + ,SQLCmd + FROM + @DropCreateSQLCmd; + + RETURN; +END; +GO + +/* +EXEC dbo.sp_drop_create_stats_external_table; +GO +*/ \ No newline at end of file From 7bcef17abe61886fe44ab02fc314dc927cef43fc Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 15:43:55 +0200 Subject: [PATCH 5/7] Add link to SP source code --- samples/manage/polybase/external-table/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/manage/polybase/external-table/README.md b/samples/manage/polybase/external-table/README.md index c8c49f51..79e03bb5 100644 --- a/samples/manage/polybase/external-table/README.md +++ b/samples/manage/polybase/external-table/README.md @@ -70,7 +70,7 @@ Referring to the documentation page, it explicitly states, "Updating statistics * This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions. * Option 2: Drop and Recreate Statistics: - * Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. + * Employ a stored procedure, [sp_drop_create_stats_external_table](https://github.com/microsoft/sql-server-samples/tree/master/samples/manage/polybase/external-table/source/sp-drop-create-stats-external-table.sql), to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns. ### Implementation guide From e37dd54657d26b7c2bf703bab23a5adc5d22b9db Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Thu, 23 May 2024 16:01:20 +0200 Subject: [PATCH 6/7] Add link to new sample on database management --- samples/manage/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/samples/manage/README.md b/samples/manage/README.md index 0a24ac8f..adf3291c 100644 --- a/samples/manage/README.md +++ b/samples/manage/README.md @@ -17,3 +17,6 @@ This Solution Quick Start provides a solution for a Software-as-a-Solution (SaaS ## Windows Containers This includes samples for setting up mssql-server in Windows Containers. Currently it only includes a link to the separately maintained [mssql-docker](https://github.com/Microsoft/mssql-docker/blob/master/windows/README.md) instructions. + +## Handling UPDATE STATISTICS error on SQL Server PolyBase external tables +[This sample](https://github.com/microsoft/sql-server-samples/tree/master/samples/manage/polybase/external-table/README.md) describes an option to update statistics on SQL Server PolyBase external tables. From 60edf5f39297ef4c3c130c8eb68ac4b472072100 Mon Sep 17 00:00:00 2001 From: Sergio Govoni Date: Fri, 24 May 2024 15:01:19 +0200 Subject: [PATCH 7/7] Update the sample title --- samples/manage/README.md | 2 +- samples/manage/polybase/external-table/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/samples/manage/README.md b/samples/manage/README.md index adf3291c..47701537 100644 --- a/samples/manage/README.md +++ b/samples/manage/README.md @@ -18,5 +18,5 @@ This Solution Quick Start provides a solution for a Software-as-a-Solution (SaaS ## Windows Containers This includes samples for setting up mssql-server in Windows Containers. Currently it only includes a link to the separately maintained [mssql-docker](https://github.com/Microsoft/mssql-docker/blob/master/windows/README.md) instructions. -## Handling UPDATE STATISTICS error on SQL Server PolyBase external tables +## Handling UPDATE STATISTICS on SQL Server PolyBase external tables [This sample](https://github.com/microsoft/sql-server-samples/tree/master/samples/manage/polybase/external-table/README.md) describes an option to update statistics on SQL Server PolyBase external tables. diff --git a/samples/manage/polybase/external-table/README.md b/samples/manage/polybase/external-table/README.md index 79e03bb5..670b1c22 100644 --- a/samples/manage/polybase/external-table/README.md +++ b/samples/manage/polybase/external-table/README.md @@ -1,7 +1,7 @@ ![](https://github.com/microsoft/sql-server-samples/blob/master/media/solutions-microsoft-logo-small.png) -# Handling UPDATE STATISTICS error on SQL Server PolyBase external tables +# Handling UPDATE STATISTICS on SQL Server PolyBase external tables This sample describes an option to update statistics on SQL Server PolyBase external tables.