github/sql-server-samples
1
0
Fork 0
mirror of https://github.com/Microsoft/sql-server-samples.git synced 2025-12-08 14:58:54 +00:00
Code Issues Releases Wiki Activity

Merge branch 'segovoni/samples/manage/polyBase/external-table' of https://github.com/segovoni/sql-server-samples into segovoni/samples/manage/polyBase/external-table

Browse Source
This commit is contained in:
Sergio Govoni
2024-05-23 15:36:30 +02:00
parent e299b4b358 be47b2703e
commit 579ecd7ffd
12 changed files with 6607 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/manage/azure-arc-enabled-sql-server/modify-license-type/README.md
+1 -1
View File
@@ -30,7 +30,7 @@ The script accepts the following command line parameters:
|-SubId|subscription_id *or* a file_name|Optional: Subscription id or a .csv file with the list of subscriptions<sup>1</sup>. If not specified all subscriptions will be scanned|
|-ResourceGroup |resource_group_name|Optional: Limits the scope to a specific resource group|
|-MachineName |machine_name|Optional: Limits the scope to a specific machine|
|-LicenceType | "Paid", "PAYG" or "LicenseOnly"| Optional: Sets the license type to the specified value |
|-LicenseType | "Paid", "PAYG" or "LicenseOnly"| Optional: Sets the license type to the specified value |
|-EnableESU | "Yes", "No" | Optional. Enables the ESU policy the value is "Yes" or disables it if the value is "No". To enable, the license type must be "Paid" or "PAYG"|
|-Force| |Optional. Forces the change of the license type to the specified value on all installed extensions. If -Force is not specified, the -LicenseType value is set only if undefined. Ignored if -LicenseType is not specified|
samples/manage/polybase/external-table/README.md
+17 -13
View File
@@ -1,15 +1,15 @@
<!-- Always leave the MS logo -->
![](https://github.com/microsoft/sql-server-samples/blob/master/media/solutions-microsoft-logo-small.png)
# Handling Update Statistics Error on External Tables in SQL Server
# Handling UPDATE STATISTICS error on SQL Server PolyBase external tables
This sample describes an option to update statistics on SQL Server PolyBase external tables!
This sample describes an option to update statistics on SQL Server PolyBase external tables.
## Background
In the process of configuring a maintenance plan for a SQL Server database with external tables and external data sources for PolyBase queries, an error occurred during the update statistics task.
## Problem Encountered
## Problem encountered
While attempting to update statistics on an external table, the following error message was encountered:
@@ -37,7 +37,7 @@ The object Update Statistics isn't supported on External Table
## About this sample
- **Applies to:** SQL Server 2017 (or higher)
- **Key features:** Update statistics
- **Key features:** UPDATE STATISTICS
- **Workload:** No workload related to this sample
- **Programming Language:** T-SQL
- **Authors:** [Sergio Govoni](https://www.linkedin.com/in/sgovoni/) | [Microsoft MVP Profile](https://mvp.microsoft.com/mvp/profile/c7b770c0-3c9a-e411-93f2-9cb65495d3c4) | [Blog](https://segovoni.medium.com/) | [GitHub](https://github.com/segovoni) | [Twitter](https://twitter.com/segovoni)
@@ -60,19 +60,19 @@ Upon investigation, it became apparent that SQL Server does not support the dire
## Resolution steps
1. Understanding the Limitation:
### Understanding the limitation
Referring to the documentation page, it explicitly states, "Updating statistics is not supported on external tables. To update statistics on an external table, drop and re-create the statistics."
2. Available Solutions:
### Available solutions
Option 1: Ignore External Tables:
This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions.
* Option 1: Ignore External Tables:
* This option is not feasible for maintenance plans managed by SQL Server Management Studio, necessitating third-party solutions.
Option 2: Drop and Recreate Statistics:
Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns.
* Option 2: Drop and Recreate Statistics:
* Employ a stored procedure, `sp_drop_create_stats_external_table`, to generate T-SQL statements for dropping and creating statistics on external tables. This procedure supports statistics on multiple columns.
3. Implementation Guide:
### Implementation guide
- Execute `sp_drop_create_stats_external_table` to generate T-SQL statements
- Execute DROP STATISTICS statements before the maintenance statistics task
@@ -103,7 +103,11 @@ This code sample is provided for demonstration and educational purposes only. It
<a name=related-links></a>
## Related Links
## Related links
<!-- Links to more articles. Remember to delete "en-us" from the link path. -->
For more information, see these articles...
For more information, see these articles:
- [Data virtualization with PolyBase in SQL Server](https://learn.microsoft.com/sql/relational-databases/polybase/polybase-guide?WT.mc_id=DP-MVP-4029181)
- [UPDATE STATISTICS (Transact-SQL)](https://learn.microsoft.com/sql/t-sql/statements/update-statistics-transact-sql?WT.mc_id=DP-MVP-4029181)
- [External tables: Why create statistics?](https://learn.microsoft.com/answers/questions/978155/external-tables-why-create-statistics?WT.mc_id=DP-MVP-4029181)
samples/manage/sql-vm-io-analysis.ps1
+186
View File
@@ -0,0 +1,186 @@
# Enter parameters
$subscriptionId = Read-Host "<Subscription ID>"
$resourceGroup = Read-Host "<Resource Group>"
$vmName = Read-Host "<Virtual machine name>"
# Set resource details
$resourceType = "Microsoft.Compute/virtualMachines"
$resourceId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/$resourceType/$vmName"
# Get Azure access token
$accessToken = az account get-access-token --query accessToken -o tsv
# Invoke Azure Monitor Metrics API
function Get-Metrics {
[CmdletBinding()]
param (
[string]$accessToken,
[string]$resourceId,
[string]$metricNames,
[string]$apiVersion = "2023-10-01"
)
try {
$startTime = (Get-Date).AddHours(-24).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
$endTime = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
$timespan = "$startTime/$endTime"
Write-Verbose "Evaluating timespan: $timespan"
$uri = "https://management.azure.com$resourceId/providers/Microsoft.Insights/metrics?api-version=$apiVersion&metricnames=$metricNames&aggregation=maximum&interval=PT1M&timespan=$timespan"
$headers = @{ "Authorization" = "Bearer $accessToken"; "Content-Type" = "application/json" }
$response = Invoke-RestMethod -Uri $uri -Headers $headers -Method Get
if ($response) {
Write-Verbose "API response successfully retrieved."
return $response
} else {
Write-Error "No response from API."
}
} catch {
Write-Error "Error retrieving metrics: $_"
}
}
# Check if data disk latency violates thresholds
function Check-Latency {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[Object]$metrics,
[Parameter()]
[int]$latencyThreshold = 500,
[Parameter()]
[int]$consecutiveCount = 5
)
$violationTimes = @()
foreach ($metric in $metrics.value) {
if ($metric.name.value -eq "Data Disk Latency") {
$count = 0
foreach ($dataPoint in $metric.timeseries[0].data) {
if ($dataPoint.maximum -gt $latencyThreshold) {
$count++
if ($count -ge $consecutiveCount) {
$violationTimes += $dataPoint.timeStamp
$count = 0 # Reset count after recording a violation
}
} else {
$count = 0 # Reset count if the sequence is broken
}
}
}
}
if ($violationTimes.Count -gt 0) {
Write-Verbose "Latency violations detected."
return @{ "Flag" = $true; "Times" = $violationTimes }
} else {
Write-Verbose "No latency violations detected."
return @{ "Flag" = $false }
}
}
# Check metrics other than latency to evaluate for throttling
function Check-OtherMetricsThrottled {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[Object]$metrics,
[Parameter()]
[int]$PercentageThreshold = 90,
[Parameter()]
[int]$consecutiveCount = 5
)
$violatedMetrics = @()
foreach ($metric in $metrics.value) {
$count = 0
foreach ($dataPoint in $metric.timeseries[0].data) {
if ($dataPoint.maximum -gt $PercentageThreshold) {
$count++
if ($count -ge $consecutiveCount) {
$violatedMetrics += @{ "Metric" = $metric.name.localizedValue; "Time" = $dataPoint.timeStamp; "Value" = $dataPoint.maximum }
break
}
} else {
$count = 0
}
}
}
if ($violatedMetrics.Count -gt 0) {
Write-Verbose "Other metrics violations detected."
} else {
Write-Verbose "No other metrics violations detected."
}
return $violatedMetrics
}
# Compare times for latency & other throttled metrics. Logs the volations with values & timestamps
function CompareTimes {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[Hashtable]$latencyResult,
[Parameter(Mandatory = $true)]
[Array]$otherMetrics
)
foreach ($metric in $otherMetrics) {
$otherDateTime = [DateTime]$metric["Time"]
$isWithinFiveMinutes = $false
$closestLatencyTime = $null
$closestTimeDifference = [int]::MaxValue
foreach ($latencyTime in $latencyResult.Times) {
$latencyDateTime = [DateTime]$latencyTime
$timeDifference = [Math]::Abs(($otherDateTime - $latencyDateTime).TotalMinutes)
if ($timeDifference -le 5) {
$isWithinFiveMinutes = $true
if ($timeDifference -lt $closestTimeDifference) {
$closestTimeDifference = $timeDifference
$closestLatencyTime = $latencyTime
}
}
}
if ($isWithinFiveMinutes) {
if ($otherDateTime -lt $closestLatencyTime) {
Write-Host "`n $($metric["Metric"]) limit was hit before latency spiked at $closestLatencyTime with value $($metric["Value"]). `n"
} else {
Write-Host "`n $($metric["Metric"]) hit its limit with value $($metric["Value"]) at $($metric["Time"])."
Write-Host "Latency spiked at $closestLatencyTime before $($metric["Metric"]) hit its limit `n"
}
} else {
Write-Host "`n Metric: $($metric["Metric"]) exceeded its threshold with a value of $($metric["Value"]) at $($metric["Time"]), but this was not within 5 minutes of any latency spikes."
}
}
}
# Prompt user for latency threshold
$latencyThreshold = Read-Host "Enter Latency Threshold (default is 500)"
if (-not [int]::TryParse($latencyThreshold, [ref]0)) {
$latencyThreshold = 500 # Use default if invalid input
Write-Host "No valid input provided. Using Default 500ms for disk latency threshold"
}
# Execute main logic
$latencyMetrics = Get-Metrics -accessToken $accessToken -resourceId $resourceId -metricNames "Data Disk Latency"
$latencyResult = Check-Latency -metrics $latencyMetrics -latencyThreshold $latencyThreshold
if ($latencyResult.Flag) {
# If latency is flagged, check for other metrics. If there is no disk latency, machine is likely not throttled but only at high consumption
Write-Verbose "Checking the following metrics: Data Disk Bandwidth Consumed Percentage,Data Disk IOPS Consumed Percentage,VM Cached Bandwidth Consumed Percentage,VM Cached IOPS Consumed Percentage,VM Uncached Bandwidth Consumed Percentage,VM Uncached IOPS Consumed Percentage"
$DiskVMMetrics = Get-Metrics -accessToken $accessToken -resourceId $resourceId -metricNames "Data Disk Bandwidth Consumed Percentage,Data Disk IOPS Consumed Percentage,VM Cached Bandwidth Consumed Percentage,VM Cached IOPS Consumed Percentage,VM Uncached Bandwidth Consumed Percentage,VM Uncached IOPS Consumed Percentage"
$additionalMetrics = Check-OtherMetricsThrottled -metrics $DiskVMMetrics
if ($additionalMetrics.Count -gt 0) {
CompareTimes $latencyResult $additionalMetrics
} else {
Write-Host "No metrics violations detected besides latency."
}
} else {
Write-Host "No latency issues detected."
}