Determine the UTF8 storage requirements for all tables and string columns in the database in scope

The script below will determine the UTF8 storage requirements for all tables and string columns in the database in scope, and whether the defined data type length has to be changed when converting to UTF-8. The result will show the table and column names that map to CHAR, NCHAR, VARCHAR, and NVARCHAR data types (including UDTs), their current type size definition, actual byte size, and projected byte size once encoded into UTF-8.

For example, for the column EmailAddress of table Application.People, the data type definition is nvarchar(512) and the data length for the largest string found in that column is 66 bytes. If changed to varchar(512) under a _UTF8 collation, the data length would be reduced to 33 bytes. In this case, converting the column's collation and data type (without changing the size definition) will not cause data loss.

However, if the projected byte size for the column using UTF-8 would be larger than the current data type definition, then the defined data type length would also have to be enlarged. For example, if a column were defined originally as nvarchar(100)¹, and projected byte size for the column in UTF-8 would be 120, then changing the column type to a varchar(100)² would result in data loss. The column would have to be defined at least as varchar(120).

¹ Expresses byte-pairs, meaning the column is capable of storing up to 200 bytes of information
² Expresses bytes, meaning the column is capable of storing up to 100 bytes of information

In [7]:

SET NOCOUNT ON;
GO

USE WideWorldImporters
GO

DROP TABLE IF EXISTS #tmpObjects;
GO

CREATE TABLE #tmpObjects (ObjectName sysname, 
	ColumnName sysname, 
	ColumnType sysname, 
	DefinedTypeSize smallint, 
	ActualMaxBytes smallint,
	UTF8BytesNeeded smallint,
	[isdone] bit,
    CONSTRAINT PK_ObjName_ColName
        PRIMARY KEY NONCLUSTERED (ObjectName, ColumnName)
		WITH (IGNORE_DUP_KEY = ON)
	);

INSERT INTO #tmpObjects
SELECT QUOTENAME(SS.[name]) + '.' + QUOTENAME(STbl.[name]), QUOTENAME(SC.[name]), ST.[name], SC.max_length, NULL, NULL, 0
FROM sys.columns AS SC
INNER JOIN sys.types AS ST ON SC.user_type_id = ST.user_type_id
INNER JOIN sys.tables AS STbl ON STbl.[object_id] = SC.[object_id]
INNER JOIN sys.schemas AS SS ON STbl.[schema_id] = SS.[schema_id]
WHERE STbl.[type] = 'U' 
	AND STbl.is_ms_shipped = 0
	--AND STbl.temporal_type IN (0,1)
	AND ST.system_type_id IN (167, 175, 231, 239)
	AND ST.[name] <> 'sysname'
	AND SC.is_hidden = 0
	AND SC.max_length > 0;

DECLARE @OName sysname, @CName sysname, @CurrBytes smallint, @UTF8Bytes smallint, @sqlcmd NVARCHAR(4000), @params NVARCHAR(60), @cnt int, @maxcnt int

SELECT @maxcnt = COUNT(*) FROM #tmpObjects;
SET @cnt = 0 
SET @params = '@CurrBytesOut smallint OUTPUT, @UTF8BytesOut smallint OUTPUT'

WHILE @cnt < @maxcnt
BEGIN
	SELECT TOP 1 @OName = ObjectName, @CName = ColumnName FROM #tmpObjects WHERE isdone = 0
	SELECT @sqlcmd = 'SELECT @CurrBytesOut = MAX(DATALENGTH(' + @CName + ')), @UTF8BytesOut = MAX(DATALENGTH(CAST(' + @CName + ' AS VARCHAR(4000)) COLLATE Latin1_General_100_CI_AI_SC_UTF8)) FROM ' + @OName + ' WITH (NOLOCK)';  

	EXEC sp_executesql @sqlcmd, @params, @CurrBytesOut = @CurrBytes OUTPUT, @UTF8BytesOut = @UTF8Bytes OUTPUT

	UPDATE #tmpObjects
	SET ActualMaxBytes = @CurrBytes, UTF8BytesNeeded = @UTF8Bytes, isdone = 1 
	WHERE ObjectName = @OName AND ColumnName = @CName

	SET @cnt = @cnt + 1
END;

SELECT * FROM #tmpObjects;

Out [7]:

Commands completed successfully.

Total execution time: 00:00:00.0028408

Commands completed successfully.

Total execution time: 00:00:00.0015389

Commands completed successfully.

Total execution time: 00:00:00.0048617

Warning: Null value is eliminated by an aggregate or other SET operation.

Total execution time: 00:00:01.1670323

ObjectName	ColumnName	ColumnType	DefinedTypeSize	ActualMaxBytes	UTF8BytesNeeded	isdone
[Application].[Cities]	[CityName]	nvarchar	100	70	35	1
[Application].[Cities_Archive]	[CityName]	nvarchar	100	32	16	1
[Application].[Countries]	[Continent]	nvarchar	60	46	23	1
[Application].[Countries]	[CountryName]	nvarchar	120	42	23	1
[Application].[Countries]	[CountryType]	nvarchar	40	30	15	1
[Application].[Countries]	[FormalName]	nvarchar	120	104	52	1
[Application].[Countries]	[IsoAlpha3Code]	nvarchar	6	6	3	1
[Application].[Countries]	[Region]	nvarchar	60	16	8	1
[Application].[Countries]	[Subregion]	nvarchar	60	50	25	1
[Application].[Countries_Archive]	[Continent]	nvarchar	60	26	13	1
[Application].[Countries_Archive]	[CountryName]	nvarchar	120	36	18	1
[Application].[Countries_Archive]	[CountryType]	nvarchar	40	30	15	1
[Application].[Countries_Archive]	[FormalName]	nvarchar	120	74	37	1
[Application].[Countries_Archive]	[IsoAlpha3Code]	nvarchar	6	6	3	1
[Application].[Countries_Archive]	[Region]	nvarchar	60	16	8	1
[Application].[Countries_Archive]	[Subregion]	nvarchar	60	50	25	1
[Application].[DeliveryMethods]	[DeliveryMethodName]	nvarchar	100	54	27	1
[Application].[DeliveryMethods_Archive]	[DeliveryMethodName]	nvarchar	100	32	16	1
[Application].[PaymentMethods]	[PaymentMethodName]	nvarchar	100	22	11	1
[Application].[PaymentMethods_Archive]	[PaymentMethodName]	nvarchar	100	22	11	1
[Application].[People]	[EmailAddress]	nvarchar	512	66	33	1
[Application].[People]	[FaxNumber]	nvarchar	40	28	14	1
[Application].[People]	[FullName]	nvarchar	100	58	29	1
[Application].[People]	[LogonName]	nvarchar	100	64	32	1
[Application].[People]	[PhoneNumber]	nvarchar	40	28	14	1
[Application].[People]	[PreferredName]	nvarchar	100	40	20	1
[Application].[People]	[SearchName]	nvarchar	202	92	46	1
[Application].[People_Archive]	[EmailAddress]	nvarchar	512	66	33	1
[Application].[People_Archive]	[FaxNumber]	nvarchar	40	28	14	1
[Application].[People_Archive]	[FullName]	nvarchar	100	46	23	1
[Application].[People_Archive]	[LogonName]	nvarchar	100	64	32	1
[Application].[People_Archive]	[PhoneNumber]	nvarchar	40	28	14	1
[Application].[People_Archive]	[PreferredName]	nvarchar	100	40	20	1
[Application].[People_Archive]	[SearchName]	nvarchar	202	82	41	1
[Application].[StateProvinces]	[SalesTerritory]	nvarchar	100	28	14	1
[Application].[StateProvinces]	[StateProvinceCode]	nvarchar	10	4	2	1
[Application].[StateProvinces]	[StateProvinceName]	nvarchar	100	58	29	1
[Application].[StateProvinces_Archive]	[SalesTerritory]	nvarchar	100	28	14	1
[Application].[StateProvinces_Archive]	[StateProvinceCode]	nvarchar	10	4	2	1
[Application].[StateProvinces_Archive]	[StateProvinceName]	nvarchar	100	52	26	1
[Application].[SystemParameters]	[DeliveryAddressLine1]	nvarchar	120	16	8	1
[Application].[SystemParameters]	[DeliveryAddressLine2]	nvarchar	120	72	36	1
[Application].[SystemParameters]	[DeliveryPostalCode]	nvarchar	20	10	5	1
[Application].[SystemParameters]	[PostalAddressLine1]	nvarchar	120	26	13	1
[Application].[SystemParameters]	[PostalAddressLine2]	nvarchar	120	32	16	1
[Application].[SystemParameters]	[PostalPostalCode]	nvarchar	20	10	5	1
[Application].[TransactionTypes]	[TransactionTypeName]	nvarchar	100	58	29	1
[Application].[TransactionTypes_Archive]	[TransactionTypeName]	nvarchar	100	12	6	1
[Purchasing].[PurchaseOrderLines]	[Description]	nvarchar	200	170	85	1
[Purchasing].[PurchaseOrders]	[SupplierReference]	nvarchar	40	18	9	1
[Purchasing].[SupplierCategories]	[SupplierCategoryName]	nvarchar	100	54	27	1
[Purchasing].[SupplierCategories_Archive]	[SupplierCategoryName]	nvarchar	100	14	7	1
[Purchasing].[Suppliers]	[BankAccountBranch]	nvarchar	100	62	31	1
[Purchasing].[Suppliers]	[BankAccountCode]	nvarchar	40	12	6	1
[Purchasing].[Suppliers]	[BankAccountName]	nvarchar	100	48	24	1
[Purchasing].[Suppliers]	[BankAccountNumber]	nvarchar	40	20	10	1
[Purchasing].[Suppliers]	[BankInternationalCode]	nvarchar	40	10	5	1
[Purchasing].[Suppliers]	[DeliveryAddressLine1]	nvarchar	120	16	8	1
[Purchasing].[Suppliers]	[DeliveryAddressLine2]	nvarchar	120	52	26	1
[Purchasing].[Suppliers]	[DeliveryPostalCode]	nvarchar	20	10	5	1
[Purchasing].[Suppliers]	[FaxNumber]	nvarchar	40	28	14	1
[Purchasing].[Suppliers]	[PhoneNumber]	nvarchar	40	28	14	1
[Purchasing].[Suppliers]	[PostalAddressLine1]	nvarchar	120	24	12	1
[Purchasing].[Suppliers]	[PostalAddressLine2]	nvarchar	120	20	10	1
[Purchasing].[Suppliers]	[PostalPostalCode]	nvarchar	20	10	5	1
[Purchasing].[Suppliers]	[SupplierName]	nvarchar	200	48	24	1
[Purchasing].[Suppliers]	[SupplierReference]	nvarchar	40	22	11	1
[Purchasing].[Suppliers]	[WebsiteURL]	nvarchar	512	74	37	1
[Purchasing].[Suppliers_Archive]	[BankAccountBranch]	nvarchar	100	62	31	1
[Purchasing].[Suppliers_Archive]	[BankAccountCode]	nvarchar	40	12	6	1
[Purchasing].[Suppliers_Archive]	[BankAccountName]	nvarchar	100	48	24	1
[Purchasing].[Suppliers_Archive]	[BankAccountNumber]	nvarchar	40	20	10	1
[Purchasing].[Suppliers_Archive]	[BankInternationalCode]	nvarchar	40	10	5	1
[Purchasing].[Suppliers_Archive]	[DeliveryAddressLine1]	nvarchar	120	16	8	1
[Purchasing].[Suppliers_Archive]	[DeliveryAddressLine2]	nvarchar	120	52	26	1
[Purchasing].[Suppliers_Archive]	[DeliveryPostalCode]	nvarchar	20	10	5	1
[Purchasing].[Suppliers_Archive]	[FaxNumber]	nvarchar	40	28	14	1
[Purchasing].[Suppliers_Archive]	[PhoneNumber]	nvarchar	40	28	14	1
[Purchasing].[Suppliers_Archive]	[PostalAddressLine1]	nvarchar	120	24	12	1
[Purchasing].[Suppliers_Archive]	[PostalAddressLine2]	nvarchar	120	20	10	1
[Purchasing].[Suppliers_Archive]	[PostalPostalCode]	nvarchar	20	10	5	1
[Purchasing].[Suppliers_Archive]	[SupplierName]	nvarchar	200	48	24	1
[Purchasing].[Suppliers_Archive]	[SupplierReference]	nvarchar	40	22	11	1
[Purchasing].[Suppliers_Archive]	[WebsiteURL]	nvarchar	512	74	37	1
[Purchasing].[SupplierTransactions]	[SupplierInvoiceNumber]	nvarchar	40	8	4	1
[Sales].[BuyingGroups]	[BuyingGroupName]	nvarchar	100	26	13	1
[Sales].[BuyingGroups_Archive]	[BuyingGroupName]	nvarchar	100	NULL	NULL	1
[Sales].[CustomerCategories]	[CustomerCategoryName]	nvarchar	100	32	16	1
[Sales].[CustomerCategories_Archive]	[CustomerCategoryName]	nvarchar	100	16	8	1
[Sales].[Customers]	[CustomerName]	nvarchar	200	76	38	1
[Sales].[Customers]	[DeliveryAddressLine1]	nvarchar	120	18	9	1
[Sales].[Customers]	[DeliveryAddressLine2]	nvarchar	120	52	26	1
[Sales].[Customers]	[DeliveryPostalCode]	nvarchar	20	10	5	1
[Sales].[Customers]	[DeliveryRun]	nvarchar	10	0	0	1
[Sales].[Customers]	[FaxNumber]	nvarchar	40	28	14	1
[Sales].[Customers]	[PhoneNumber]	nvarchar	40	28	14	1
[Sales].[Customers]	[PostalAddressLine1]	nvarchar	120	22	11	1
[Sales].[Customers]	[PostalAddressLine2]	nvarchar	120	42	21	1
[Sales].[Customers]	[PostalPostalCode]	nvarchar	20	10	5	1
[Sales].[Customers]	[RunPosition]	nvarchar	10	0	0	1
[Sales].[Customers]	[WebsiteURL]	nvarchar	512	104	52	1
[Sales].[Customers_Archive]	[CustomerName]	nvarchar	200	46	23	1
[Sales].[Customers_Archive]	[DeliveryAddressLine1]	nvarchar	120	16	8	1
[Sales].[Customers_Archive]	[DeliveryAddressLine2]	nvarchar	120	48	24	1
[Sales].[Customers_Archive]	[DeliveryPostalCode]	nvarchar	20	10	5	1
[Sales].[Customers_Archive]	[DeliveryRun]	nvarchar	10	0	0	1
[Sales].[Customers_Archive]	[FaxNumber]	nvarchar	40	28	14	1
[Sales].[Customers_Archive]	[PhoneNumber]	nvarchar	40	28	14	1
[Sales].[Customers_Archive]	[PostalAddressLine1]	nvarchar	120	22	11	1
[Sales].[Customers_Archive]	[PostalAddressLine2]	nvarchar	120	34	17	1
[Sales].[Customers_Archive]	[PostalPostalCode]	nvarchar	20	10	5	1
[Sales].[Customers_Archive]	[RunPosition]	nvarchar	10	0	0	1
[Sales].[Customers_Archive]	[WebsiteURL]	nvarchar	512	96	48	1
[Sales].[InvoiceLines]	[Description]	nvarchar	200	170	85	1
[Sales].[Invoices]	[ConfirmedReceivedBy]	nvarchar	8000	58	29	1
[Sales].[Invoices]	[CustomerPurchaseOrderNumber]	nvarchar	40	10	5	1
[Sales].[Invoices]	[DeliveryRun]	nvarchar	10	0	0	1
[Sales].[Invoices]	[RunPosition]	nvarchar	10	0	0	1
[Sales].[OrderLines]	[Description]	nvarchar	200	170	85	1
[Sales].[Orders]	[CustomerPurchaseOrderNumber]	nvarchar	40	10	5	1
[Sales].[SpecialDeals]	[DealDescription]	nvarchar	60	48	24	1
[Warehouse].[Colors]	[ColorName]	nvarchar	40	22	11	1
[Warehouse].[Colors_Archive]	[ColorName]	nvarchar	40	8	4	1
[Warehouse].[PackageTypes]	[PackageTypeName]	nvarchar	100	12	6	1
[Warehouse].[PackageTypes_Archive]	[PackageTypeName]	nvarchar	100	NULL	NULL	1
[Warehouse].[StockGroups]	[StockGroupName]	nvarchar	100	38	19	1
[Warehouse].[StockGroups_Archive]	[StockGroupName]	nvarchar	100	16	8	1
[Warehouse].[StockItemHoldings]	[BinLocation]	nvarchar	40	8	4	1
[Warehouse].[StockItems]	[Barcode]	nvarchar	100	26	13	1
[Warehouse].[StockItems]	[Brand]	nvarchar	100	18	9	1
[Warehouse].[StockItems]	[Size]	nvarchar	40	26	13	1
[Warehouse].[StockItems]	[StockItemName]	nvarchar	200	170	85	1
[Warehouse].[StockItems_Archive]	[Barcode]	nvarchar	100	26	13	1
[Warehouse].[StockItems_Archive]	[Brand]	nvarchar	100	18	9	1
[Warehouse].[StockItems_Archive]	[Size]	nvarchar	40	26	13	1
[Warehouse].[StockItems_Archive]	[StockItemName]	nvarchar	200	170	85	1
[Warehouse].[VehicleTemperatures]	[FullSensorData]	nvarchar	2000	392	196	1
[Warehouse].[VehicleTemperatures]	[VehicleRegistration]	nvarchar	40	18	9	1

91 KiB Raw Blame History

Determine the UTF8 storage requirements for all tables and string columns in the database in scope

91 KiB

Raw Blame History