sql-server-samples/samples/features/in-memory-database/memory-optimized-tempdb-metadata/MemoryOptmizedTempDBMetadata-Python.ipynb

Exploring Memory-Optmized TempDB Metadata

TempDB metadata contention has historically been a bottleneck to scalability for many workloads running on SQL Server. SQL Server 2019 introduces a new feature that is part of the In-Memory Database feature family, memory-optimized tempdb metadata, which effectively removes this bottleneck and unlocks a new level of scalability for tempdb-heavy workloads. In SQL Server 2019, the system tables involved in managing temp table metadata can be moved into latch-free non-durable memory-optimized tables.

To learn more about tempdb metadata contention, along with other types of tempdb contention, check out the blog article TEMPDB - Files and Trace Flags and Updates, Oh My!. Keep reading to explore the new memory-optimized tempdb metadata feature.

Configure your environment

Contention in tempdb happens when a large number of concurrent threads are attemping to create, modify or drop temp tables. In order to simulate this situation, you'll need to have a SQL Server instance that has multiple cores (4 or more is recommended), and a way to simulate multiple concurrent threads. For this example, we'll be using the ostress.exe tool to generate multiple concurrent threads. If you have an existing multi-core SQL Server instance, you can follow the T-SQL instructions to set up the demo, otherwise you can try the docker container steps instead.

First, download the demo files to your local computer.

Docker Container Setup

Note that the docker commands may take some time to execute, but you will not see progress here until they are complete.

1. Make sure you have your docker environment configured, more information here.

NOTE
If you are using Docker Desktop for Windows or Mac, the default configuration will limit your containers to 2 cores, regardless of the number of cores on your computer. Be sure to configure docker to allow at least 4 cores and 4GB of RAM for this demo to run properly. To do this, right click on the Docker Desktop icon in the status bar and choose Settings -> Advanced.

2. Pull the demo container with the following command:

In [3]:

! docker pull bluefooted/sql2019tempdbdemo

Using default tag: latest
latest: Pulling from bluefooted/sql2019tempdbdemo
Digest: sha256:035a1bda5539bfe68ad1b2f032a6e389cea91a0cd880e75b83ef186c46b2e34f
Status: Image is up to date for bluefooted/sql2019tempdbdemo:latest
docker.io/bluefooted/sql2019tempdbdemo:latest

3. Start the demo container with the following command:

In [2]:

! docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=P@ssw0rd!" -p 1455:1433 --name sql2019tempdbdemo -d bluefooted/sql2019tempdbdemo

2a3dbfff94ce2bd277778e36ac268b4495afb77f1b6c5417478b10a9c545cca6

NOTE:
If you see the following error, you may already have run the docker run command with this image:

docker: Error response from daemon: Conflict. The container name "/sql2019tempdbdemo" is already in use by container "3f662e0fd9b8cbdc1013e874722e066aa8e81ec3a07423fc3ab95cb75e640af9". You have to remove (or rename) that container to be able to reuse that name. See 'docker run --help'.

If you see this message, you can start the container instead with the following command:

In [0]:

! docker start sql2019tempdbdemo

4. Connect to the demo SQL Server instance using Azure Data Studio or SQL Server Management Studio using the following information:

   Server Name: localhost,1455
   Username: sa
   Password: P@ssw0rd!

Existing SQL Server Instance Setup (skip if you are using the demo container)

If you already have a SQL Server instance with a minimum of 4 cores, you can download and restore the AdventureWorks database and use the scripts in this repo to configure the database. Follow the steps in the T-SQL notebook to complete this setup.

Detecting TempDB Metadata Contention

The first thing to figure out before you turn on this new feature is whether or not you are experiencing TempDB metadata contention. The main symptom of this contention is a number of sessions in Suspended state with a wait type of PAGELATCH_xx and a wait resource of a page that hosts a TempDB system table, such as 2:1:118. In order to know whether or not the page is part of a TempDB system table, you can use the new dynamic management function sys.dm_db_page_info() in SQL Server 2019 or later, or the older DBCC PAGE command in older versions. For this demo, let's focus on sys.dm_db_page_info(). Note that this command will take some time to complete, you'll want to proceed to the next step before it completes.

First, start the workload using the ostress.exe tool that is included in the downloads. Note that if you are not using the demo container, you will need to change the server name and login information.

In [11]:

! ostress.exe -Slocalhost,1455 -Usa -PP@ssw0rd!  -dAdventureWorks -Q"EXEC dbo.usp_EmployeeBirthdayList 4"  -mstress -quiet -n16 -r120 | FINDSTR "QEXEC Starting Creating elapsed"

10/08/19 15:39:54.739 [0x00007A74] -QEXEC dbo.usp_EmployeeBirthdayList 4
10/08/19 15:39:54.779 [0x00007A74] Starting query execution...
10/08/19 15:39:54.783 [0x00007A74] Creating 16 thread(s) to process queries
10/08/19 15:40:30.158 [0x00007A74] OSTRESS exiting normally, elapsed time: 00:00:35.419

While the above script is running, switch over to the T-SQL notebook and run the script to monitor your workload for page contention. You should see several sessions with a wait_type of PAGELATCH_EX or PAGELATCH_SH, often with an object_name of sysschobjs.

NOTE
If this query does not return any results, make sure the command above is still running. If it is not running, start it and try the query again. If you still do not see any sessions waiting, you may need to increase the number of CPUs available to your server, and/or increase the number of concurrent threads by increasing the -n parameter in the command. This demo was tested with 4 cores and 16 concurrent sessions, which should yield the expected results. If you would like more time to examine the contention, you can increase the -r parameter, which will increase the number of iterations.

Improve performance with Memory-Optimized TempDB Metadata

Now that you have observed TempDB metadata contention, let's see how SQL Server 2019 addresses this contention. Switch over to the T-SQL notebook to review and run the script to enable Memory-Optimized TempDB Metadata.

Once you have run the T-SQL command, you will need to restart the service. If you are using the demo container, you can do so with the following command:

In [10]:

! docker restart sql2019tempdbdemo

sql2019tempdbdemo

Once the server is restarted, you can use queries in the T-SQL notebook to verify that the feature has been enabled.

NOTE
It's a good idea to run a few T-SQL queries after the restart to make sure the server is up and running before you attempt the scenario again.

Now that we have enabled Memory-Optimized TempDB Metadata, let's try running the workload again:

In [9]:

! ostress.exe -Slocalhost,1455 -Usa -PP@ssw0rd!  -dAdventureWorks -Q"EXEC dbo.usp_EmployeeBirthdayList 4"  -mstress -quiet -n16 -r120 | FINDSTR "QEXEC Starting Creating elapsed"

10/08/19 15:38:27.261 [0x00009AA0] -QEXEC dbo.usp_EmployeeBirthdayList 4
10/08/19 15:38:27.316 [0x00009AA0] Starting query execution...
10/08/19 15:38:27.321 [0x00009AA0] Creating 16 thread(s) to process queries
10/08/19 15:38:54.045 [0x00009AA0] OSTRESS exiting normally, elapsed time: 00:00:26.784

While this is running, switch over to the T-SQL notebook to run the monitoring script again. This time, you should not see any sessions waiting on PAGELATCH_EX or PAGELATCH_SH. Also, the script should complete faster than before the change was made.