mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Removing code-snacks from \in-memory and \security folders until further review on nesting levels, naming, and location.
This commit is contained in:
Perry Skountrianos - MSFT
@@ -1,49 +0,0 @@
|
||||
# use glob syntax.
|
||||
syntax: glob
|
||||
*.ser
|
||||
*.class
|
||||
*~
|
||||
*.bak
|
||||
#*.off
|
||||
*.old
|
||||
|
||||
# eclipse conf file
|
||||
.settings
|
||||
.classpath
|
||||
.project
|
||||
.manager
|
||||
.scala_dependencies
|
||||
|
||||
# idea
|
||||
.idea
|
||||
*.iml
|
||||
|
||||
#visual studio
|
||||
.suo
|
||||
|
||||
# building
|
||||
target
|
||||
build
|
||||
null
|
||||
tmp*
|
||||
temp*
|
||||
dist
|
||||
test-output
|
||||
build.log
|
||||
out
|
||||
packages
|
||||
|
||||
# other scm
|
||||
.svn
|
||||
.CVS
|
||||
.hg*
|
||||
|
||||
# switch to regexp syntax.
|
||||
# syntax: regexp
|
||||
# ^\.pc/
|
||||
|
||||
#SHITTY output not in target directory
|
||||
build.log
|
||||
|
||||
#dropbox
|
||||
.DS_Store
|
||||
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 128 KiB |
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 128 KiB |
@@ -1,125 +0,0 @@
|
||||
# In-Memory & Columnar Store Code Snack
|
||||
In this code snack, developers will experience the benefit of performing real-time operation analytics enabled by leveraging a memory optimized table in combination with a columnstore index. The Visual Studio project contains a load generator that will be used to simulate a write heavy workload. They will initially run the simulator against a disk based table with a clustered index (btree) and take note of the rows inserted per second, and will measure the performance of a provided analytics query while the system is under the heavy write load. They will then author the T-SQL to create the memory optimized table with a columnstore index, update the load generator to target the memory optimized table and observe the improved performance characteristics. This code snack targets SQL Server on Linux.
|
||||
|
||||
## Requirements
|
||||
- Visual Studio 2015 with Update 3 (or later)
|
||||
- [SQL Server Data Tools for Visual Studio 2015](https://msdn.microsoft.com/en-us/mt186501)
|
||||
- [SQL Server on Linux](https://www.microsoft.com/en-us/sql-server/sql-server-on-linux) running in [Docker](https://docs.docker.com/engine/installation/#/on-macos-and-windows)
|
||||
- Your SQL on Linux Server should have at least 4 GB of RAM
|
||||
|
||||
## Clone the provided project
|
||||
Clone this repo on to your local machine.
|
||||
The recommended path is C:\In-Memory and Columnar\
|
||||
|
||||
## Download the sample data
|
||||
This project requires a sample set of data you will load into SQL Server.
|
||||
Download the data from: [http://bit.ly/2envb8m](http://bit.ly/2envb8m)
|
||||
|
||||
## Copy the sample data to your SQL on Linux host
|
||||
1. On your Docker host machine, first grab the container ID of your SQL Server on Linux container. The Container ID is the value present in the first column.
|
||||
```
|
||||
docker ps
|
||||
```
|
||||
2. Copy over the sample data from the host to the container running SQL on Linux by using docker cp as follows (replace ContainerID with the container ID your retrieved). Note that the path /var/opt/mssql will map to C: in any T-SQL scripts executed against this server.
|
||||
```
|
||||
docker cp datapoints.bcp [ContainerID]:/var/opt/mssql/data/datapoints.bcp
|
||||
```
|
||||
3. Verify that the copy was successful. By connecting to bash within the container. Connect to your container (substitute your container ID in the command below) and list out the files:
|
||||
```
|
||||
docker exec -t -i [ContainerID] /bin/bash
|
||||
ls /var/opt/mssql/data -1 -s -h
|
||||
```
|
||||
4. You should see datapoints.bcp in the listing
|
||||
```
|
||||
-rw-r--r-- 1 root root 256 Nov 13 22:49 Entropy.bin
|
||||
-rw-r--r-- 1 root root 14M Nov 13 22:50 MSDBData.mdf
|
||||
-rw-r--r-- 1 root root 768K Nov 14 22:48 MSDBLog.ldf
|
||||
-rw-r--r-- 1 root root 161M Nov 11 04:18 datapoints.bcp
|
||||
-rw-r--r-- 1 root root 4.0M Nov 14 22:55 master.mdf
|
||||
-rw-r--r-- 1 root root 768K Nov 14 23:07 mastlog.ldf
|
||||
-rw-r--r-- 1 root root 8.0M Nov 14 23:00 model.mdf
|
||||
-rw-r--r-- 1 root root 8.0M Nov 14 23:00 modellog.ldf
|
||||
-rw-r--r-- 1 root root 264M Nov 14 23:07 tempdb.mdf
|
||||
-rw-r--r-- 1 root root 8.0M Nov 14 23:07 templog.ldf
|
||||
drwxr-xr-x 3 root root 4.0K Nov 14 23:01 xtp
|
||||
```
|
||||
5. You are all set to continue the lab in Visual Studio 2015.
|
||||
|
||||
## Create the database and tables
|
||||
1. Open the SqlLoadgenerator solution using Visual Studio 2015.
|
||||
2. From Solution Explorer, expand the SqlGenerator solution, then SQL Resources folder and open "Create Database.sql".
|
||||
3. Adjust the file paths for the FILENAME attributes if you installed SQL Server to a different location.
|
||||
4. Select the Execute button
|
||||
5. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
|
||||
6. Wait for the script to complete successfully.
|
||||
7. Within Visual Studio, open "Create Table- Disk Based.sql"
|
||||
8. Execute the script to create the DataPointsDiskBased table.
|
||||
This table will be used to store simulated IoT device telemetry, using traditional disk based table as well as clustered and non-clustered indexes on the fields commonly used in both point queries and analytic queries.
|
||||
9. Within Visual Studio, open "Create Table- In Memory.sql"
|
||||
10. Execute the script to create the DataPointsInMem table.
|
||||
This table will be used to store the same simulated IoT device telemetry, but this time using a memory optimized table as well as clustered column store index against all fields (which will support analytic queries) and non-clustered hash indexes on the id field (which will support point lookups common to transactional queries).
|
||||
```
|
||||
CREATE TABLE [DataPointsInMem] (
|
||||
-- ID should be a Primary Key, fields with a b-tree or hash index
|
||||
Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
|
||||
[Value] decimal(18,5),
|
||||
[TimestampUtc] datetime,
|
||||
DeviceId int,
|
||||
-- This table should have a columnar index
|
||||
INDEX Transactions_CCI CLUSTERED COLUMNSTORE
|
||||
) WITH (
|
||||
-- This should be an in-memory table
|
||||
MEMORY_OPTIMIZED = ON
|
||||
);
|
||||
|
||||
-- In-memory tables should auto-elevate their transaction level to Snapshot
|
||||
ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ;
|
||||
```
|
||||
|
||||
## Load initial data
|
||||
|
||||
1. Within Visual Studio, open "Load Sample Data.sql"
|
||||
2. Adjust the path to the DataPoint.bcp file so it matches the location of your BCP file within the SQL on Linux container (if necessary) and save the script. Recall the path /var/opt/mssql will map to C: in any T-SQL scripts executed against SQL Server on Linux.
|
||||
3. Execute the script to load each table with 4 million rows worth of sample data. This will take some time to complete.
|
||||
|
||||
## Execute the sample analytics query
|
||||
|
||||
1. Within Visual Studio, open "SampleQueries - DiskBased.sql".
|
||||
2. Execute the script to summarize the time series data stored in the disk based table.
|
||||
3. When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute. Query time is shown in the bottom right of the document window in Visual Studio.
|
||||
![alt text][Disk Based Results]
|
||||
|
||||
[Disk Based Results]: Images/DiskBasedResults.png "Disk Based Results"
|
||||
4. Now, execute the script to summarize the time series data stored in the memory-optimized table, in "SampleQueries - InMemory.sql".
|
||||
When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute.
|
||||
You should notice that the performance of the query against the memory-optimized table runs between 2x-10x faster than the same query, running against the same data stored in a disk based table. Query time is shown in the bottom right of the document window in Visual Studio.
|
||||
![alt text][In-Memory Results]
|
||||
|
||||
[In-Memory Results]: Images/InMemoryResults.png "In-Memory Results"
|
||||
|
||||
## Execute the queries under load
|
||||
|
||||
1. Within Visual Studio, Solution Explorer, expand the SqlLoadGenerator project and then open "App.config".
|
||||
2. Locate the connection string with the name "SqlConnection" and modify it so it points to your instance of SQL Server on Linux.
|
||||
3. Save the App.config.
|
||||
4. From the Debug menu, select Start Without Debugging.
|
||||
5. At the prompt, choose option 1 to target the disk based table.
|
||||
You should see log entries when every 1000 rows are inserted.
|
||||
Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
|
||||
6. Open "SampleQueries - DiskBased.sql".
|
||||
7. Execute the script to summarize the time series data stored in the disk based table.
|
||||
8. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
|
||||
9. Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
|
||||
10. Close the console load generator.
|
||||
11. Run the SqlLoadGenerator again.
|
||||
This time at the prompt, choose option 2 to target the memory-optimized table.
|
||||
You should see log entries when every 1000 rows are inserted.
|
||||
12. Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
|
||||
13. Open "SampleQueries - In Memory.sql".
|
||||
14. Execute the script to summarize the time series data stored in the disk based table.
|
||||
15. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
|
||||
Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
|
||||
16. Close the console load generator.
|
||||
|
||||
## Conclusion
|
||||
You should observe that while neither query was affected by the heavy insert load, the query against the analytics query continued to run 2x-10x faster than the same query against the disk-based table.
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
|
||||
-- For SQL Server 2016, create the Database with a master data file (mdf),
|
||||
-- the log data file (ldf) and a separate filegroup to support memory-optimized tables.
|
||||
CREATE DATABASE [Telemetry]
|
||||
CONTAINMENT = NONE
|
||||
ON PRIMARY
|
||||
( NAME = N'Telemetry', FILENAME = N'C:\data\Telemetry.mdf' , SIZE = 128MB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB ),
|
||||
FILEGROUP [TelemetryInMem] CONTAINS MEMORY_OPTIMIZED_DATA DEFAULT
|
||||
( NAME = N'Telemetry_mem', FILENAME = N'C:\data\Telemetry_mem' , MAXSIZE = UNLIMITED)
|
||||
LOG ON
|
||||
( NAME = N'Telemetry_log', FILENAME = N'C:\data\Telemetry_log.ldf' , SIZE = 128MB , MAXSIZE = 2048GB , FILEGROWTH = 65536KB )
|
||||
GO
|
||||
|
||||
ALTER DATABASE [Telemetry] SET COMPATIBILITY_LEVEL = 130
|
||||
GO
|
||||
-24
@@ -1,24 +0,0 @@
|
||||
USE Telemetry;
|
||||
GO
|
||||
|
||||
DROP INDEX IF EXISTS dbo.DataPointsDiskBased.IX_DeviceId;
|
||||
DROP INDEX IF EXISTS dbo.DataPointsDiskBased.IX_Timestamp;
|
||||
DROP TABLE IF EXISTS dbo.DataPointsDiskBased;
|
||||
|
||||
CREATE TABLE [DataPointsDiskBased] (
|
||||
Id bigint IDENTITY NOT NULL PRIMARY KEY CLUSTERED,
|
||||
[Value] decimal(18,5),
|
||||
[TimestampUtc] datetime,
|
||||
DeviceId int,
|
||||
);
|
||||
|
||||
CREATE NONCLUSTERED INDEX IX_DeviceId
|
||||
ON dbo.DataPointsDiskBased (DeviceId);
|
||||
GO
|
||||
|
||||
CREATE NONCLUSTERED INDEX IX_Timestamp
|
||||
ON dbo.DataPointsDiskBased (TimestampUtc);
|
||||
GO
|
||||
|
||||
|
||||
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
USE Telemetry;
|
||||
GO
|
||||
|
||||
DROP INDEX IF EXISTS dbo.DataPointsInMem.IX_DeviceId;
|
||||
DROP INDEX IF EXISTS dbo.DataPointsInMem.IX_Timestamp;
|
||||
DROP TABLE IF EXISTS dbo.DataPointsInMem;
|
||||
|
||||
CREATE TABLE [DataPointsInMem] (
|
||||
-- ID should be a Primary Key, fields with a b-tree or hash index
|
||||
Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
|
||||
[Value] decimal(18,5),
|
||||
[TimestampUtc] datetime,
|
||||
DeviceId int,
|
||||
-- This table should have a columnar index
|
||||
INDEX Transactions_CCI CLUSTERED COLUMNSTORE
|
||||
) WITH (
|
||||
-- This should be an in-memory table
|
||||
MEMORY_OPTIMIZED = ON
|
||||
);
|
||||
|
||||
-- In-memory tables should auto-elevate their transaction level to Snapshot
|
||||
ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ;
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
USE Telemetry;
|
||||
GO
|
||||
|
||||
BULK INSERT Telemetry.dbo.DataPointsDiskBased
|
||||
FROM 'C:\data\datapoints.bcp'
|
||||
WITH (
|
||||
DATAFILETYPE = 'native'
|
||||
);
|
||||
GO
|
||||
|
||||
BULK INSERT Telemetry.dbo.DataPointsInMem
|
||||
FROM 'C:\data\datapoints.bcp'
|
||||
WITH (
|
||||
DATAFILETYPE = 'native'
|
||||
);
|
||||
GO
|
||||
|
||||
-21
@@ -1,21 +0,0 @@
|
||||
USE Telemetry;
|
||||
|
||||
SELECT
|
||||
Count(*) Counted,
|
||||
Max([Value]) MaxValue,
|
||||
Avg([Value]) AvgValue,
|
||||
Min([Value]) MinValue,
|
||||
DatePart(YYYY, TimestampUtc) [year],
|
||||
DatePart(MM, TimestampUtc) [month],
|
||||
DatePart(DD, TimestampUtc) [day],
|
||||
DatePart(hh, TimestampUtc) [hour],
|
||||
DatePart(mi, TimestampUtc) [minute],
|
||||
DatePart(ss, TimestampUtc) [second]
|
||||
FROM DataPointsDiskBased
|
||||
GROUP BY
|
||||
DatePart(YYYY, TimestampUtc),
|
||||
DatePart(MM, TimestampUtc),
|
||||
DatePart(DD, TimestampUtc),
|
||||
DatePart(hh, TimestampUtc),
|
||||
DatePart(mi, TimestampUtc),
|
||||
DatePart(ss, TimestampUtc)
|
||||
-24
@@ -1,24 +0,0 @@
|
||||
USE Telemetry;
|
||||
|
||||
SELECT
|
||||
Count(*) Counted,
|
||||
Max([Value]) MaxValue,
|
||||
Avg([Value]) AvgValue,
|
||||
Min([Value]) MinValue,
|
||||
DatePart(YYYY, TimestampUtc) [year],
|
||||
DatePart(MM, TimestampUtc) [month],
|
||||
DatePart(DD, TimestampUtc) [day],
|
||||
DatePart(hh, TimestampUtc) [hour],
|
||||
DatePart(mi, TimestampUtc) [minute],
|
||||
DatePart(ss, TimestampUtc) [second]
|
||||
FROM DataPointsInMem
|
||||
GROUP BY
|
||||
DatePart(YYYY, TimestampUtc),
|
||||
DatePart(MM, TimestampUtc),
|
||||
DatePart(DD, TimestampUtc),
|
||||
DatePart(hh, TimestampUtc),
|
||||
DatePart(mi, TimestampUtc),
|
||||
DatePart(ss, TimestampUtc)
|
||||
|
||||
|
||||
|
||||
-1031
File diff suppressed because it is too large
Load Diff
-28
@@ -1,28 +0,0 @@
|
||||
<?xml version="1.0" encoding="utf-8" ?>
|
||||
<configuration>
|
||||
<configSections>
|
||||
<sectionGroup name="userSettings" type="System.Configuration.UserSettingsGroup, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" >
|
||||
<section name="SqlLoadGenerator.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" allowExeDefinition="MachineToLocalUser" requirePermission="false" />
|
||||
</sectionGroup>
|
||||
</configSections>
|
||||
|
||||
<connectionStrings>
|
||||
<add name="SqlConnection"
|
||||
connectionString="data source=192.168.5.105;initial catalog=telemetry;user id=sa;password=Abc1234567890;MultipleActiveResultSets=True;App=LoadGenerator" providerName="System.Data.SqlClient" />
|
||||
</connectionStrings>
|
||||
|
||||
<startup>
|
||||
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
|
||||
</startup>
|
||||
|
||||
<userSettings>
|
||||
<SqlLoadGenerator.Properties.Settings>
|
||||
<setting name="Spike_NumRowsToInsert" serializeAs="String">
|
||||
<value>200000</value>
|
||||
</setting>
|
||||
<setting name="Spike_NumParallelClients" serializeAs="String">
|
||||
<value>1</value>
|
||||
</setting>
|
||||
</SqlLoadGenerator.Properties.Settings>
|
||||
</userSettings>
|
||||
</configuration>
|
||||
-157
@@ -1,157 +0,0 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text;
|
||||
using System.Threading.Tasks;
|
||||
using System.Data.SqlClient;
|
||||
using System.Diagnostics;
|
||||
|
||||
namespace SqlLoadGenerator
|
||||
{
|
||||
class Program
|
||||
{
|
||||
static string _connectionString;
|
||||
static int _numRowsToInsert = Properties.Settings.Default.Spike_NumRowsToInsert;
|
||||
static int _numTaskPerSpike = Properties.Settings.Default.Spike_NumParallelClients;
|
||||
static Stopwatch _stopwatch;
|
||||
static string _tableName = "DataPointsDiskBased";
|
||||
|
||||
static void Main(string[] args)
|
||||
{
|
||||
Console.WriteLine("Which table would you like to target for load? (press 1 or 2)");
|
||||
Console.WriteLine("1. Disk Based Table");
|
||||
Console.WriteLine("2. Memory-Optimized Table");
|
||||
if (Console.ReadKey().KeyChar == '1')
|
||||
{
|
||||
_tableName = "DataPointsDiskBased";
|
||||
}
|
||||
else
|
||||
{
|
||||
_tableName = "DataPointsInMem";
|
||||
}
|
||||
Console.WriteLine("");
|
||||
|
||||
_connectionString = System.Configuration.ConfigurationManager.ConnectionStrings["SqlConnection"].ConnectionString;
|
||||
|
||||
_stopwatch = Stopwatch.StartNew();
|
||||
|
||||
List<Task> tasks = new List<Task>();
|
||||
|
||||
tasks.AddRange(ScheduleLoadSpike(_numTaskPerSpike));
|
||||
|
||||
Task.WaitAll(tasks.ToArray());
|
||||
Console.WriteLine("Tasks completed.");
|
||||
|
||||
_stopwatch.Stop();
|
||||
|
||||
Console.ForegroundColor = ConsoleColor.Cyan;
|
||||
Console.WriteLine("{0} hours {1} minutes {2} seconds elapsed.",
|
||||
_stopwatch.Elapsed.Hours, _stopwatch.Elapsed.Minutes, _stopwatch.Elapsed.Seconds);
|
||||
|
||||
Console.ReadLine();
|
||||
}
|
||||
|
||||
static Task[] ScheduleLoadSpike(int numTasks)
|
||||
{
|
||||
Task[] tasks = new Task[numTasks];
|
||||
for (int i = 0; i < numTasks; i++)
|
||||
{
|
||||
tasks[i] = Task.Run(() => GenerateLoadSpike());
|
||||
}
|
||||
|
||||
return tasks;
|
||||
}
|
||||
|
||||
static int _numTasks = 0;
|
||||
static int GenerateLoadSpike()
|
||||
{
|
||||
Random Rand = new Random();
|
||||
|
||||
int taskID = System.Threading.Interlocked.Increment(ref _numTasks);
|
||||
Console.WriteLine("{0}: Preparing load spike...", taskID);
|
||||
|
||||
int numRowsAffected = 0;
|
||||
|
||||
try
|
||||
{
|
||||
|
||||
SqlConnection conn = new SqlConnection(_connectionString);
|
||||
|
||||
string commandText = String.Format("INSERT [dbo].[{0}] ([Value], TimestampUtc, DeviceId) " +
|
||||
"VALUES (@Value, @TimestampUtc, @DeviceId)", _tableName);
|
||||
|
||||
Random r = new Random(1);
|
||||
|
||||
conn.Open();
|
||||
|
||||
for (int i = 0; i < _numRowsToInsert; i++)
|
||||
{
|
||||
// if a transient error closed our connection, create a new one and open it
|
||||
if (conn.State == System.Data.ConnectionState.Closed)
|
||||
{
|
||||
Console.ForegroundColor = ConsoleColor.Red;
|
||||
Console.WriteLine("{0}: Re-creating closed connection", taskID);
|
||||
Console.ResetColor();
|
||||
conn = new SqlConnection(_connectionString);
|
||||
conn.Open();
|
||||
}
|
||||
|
||||
double value = 9999 + 100 * Rand.NextDouble();
|
||||
List<SqlParameter> parameters = new List<SqlParameter>() {
|
||||
new SqlParameter("@Value", value),
|
||||
new SqlParameter("@TimestampUtc", DateTime.UtcNow),
|
||||
new SqlParameter("@DeviceId", taskID)
|
||||
};
|
||||
|
||||
|
||||
try
|
||||
{
|
||||
using (SqlCommand cmd = new SqlCommand(commandText, conn))
|
||||
{
|
||||
cmd.CommandType = System.Data.CommandType.Text;
|
||||
cmd.Parameters.AddRange(parameters.ToArray());
|
||||
numRowsAffected += cmd.ExecuteNonQuery();
|
||||
|
||||
}
|
||||
}
|
||||
catch (SqlException sqlex)
|
||||
{
|
||||
Console.ForegroundColor = ConsoleColor.Green;
|
||||
Console.WriteLine(sqlex.Message);
|
||||
Console.ResetColor();
|
||||
|
||||
System.Threading.Thread.Sleep(200);
|
||||
}
|
||||
catch (Exception cmdex)
|
||||
{
|
||||
Console.ForegroundColor = ConsoleColor.Blue;
|
||||
Console.WriteLine(cmdex.Message);
|
||||
Console.ResetColor();
|
||||
|
||||
}
|
||||
|
||||
if (i % 1000 == 0)
|
||||
{
|
||||
Console.WriteLine("{0}: Inserted {1} new rows so far", taskID, numRowsAffected);
|
||||
}
|
||||
}
|
||||
|
||||
conn.Close();
|
||||
conn.Dispose();
|
||||
|
||||
Console.WriteLine("{0}: Inserted {1} new rows", taskID, numRowsAffected);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Console.ForegroundColor = ConsoleColor.Red;
|
||||
Console.WriteLine(ex.Message);
|
||||
Console.ResetColor();
|
||||
}
|
||||
|
||||
Console.WriteLine("{0}: Finished with load spike.", taskID);
|
||||
return numRowsAffected;
|
||||
}
|
||||
}
|
||||
}
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
using System.Reflection;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
// General Information about an assembly is controlled through the following
|
||||
// set of attributes. Change these attribute values to modify the information
|
||||
// associated with an assembly.
|
||||
[assembly: AssemblyTitle("SqlLoadGenerator")]
|
||||
[assembly: AssemblyDescription("")]
|
||||
[assembly: AssemblyConfiguration("")]
|
||||
[assembly: AssemblyCompany("")]
|
||||
[assembly: AssemblyProduct("SqlLoadGenerator")]
|
||||
[assembly: AssemblyCopyright("Copyright © 2016")]
|
||||
[assembly: AssemblyTrademark("")]
|
||||
[assembly: AssemblyCulture("")]
|
||||
|
||||
// Setting ComVisible to false makes the types in this assembly not visible
|
||||
// to COM components. If you need to access a type in this assembly from
|
||||
// COM, set the ComVisible attribute to true on that type.
|
||||
[assembly: ComVisible(false)]
|
||||
|
||||
// The following GUID is for the ID of the typelib if this project is exposed to COM
|
||||
[assembly: Guid("59486e0e-6932-4191-94ff-04082db46472")]
|
||||
|
||||
// Version information for an assembly consists of the following four values:
|
||||
//
|
||||
// Major Version
|
||||
// Minor Version
|
||||
// Build Number
|
||||
// Revision
|
||||
//
|
||||
// You can specify all the values or you can default the Build and Revision Numbers
|
||||
// by using the '*' as shown below:
|
||||
// [assembly: AssemblyVersion("1.0.*")]
|
||||
[assembly: AssemblyVersion("1.0.0.0")]
|
||||
[assembly: AssemblyFileVersion("1.0.0.0")]
|
||||
-50
@@ -1,50 +0,0 @@
|
||||
//------------------------------------------------------------------------------
|
||||
// <auto-generated>
|
||||
// This code was generated by a tool.
|
||||
// Runtime Version:4.0.30319.42000
|
||||
//
|
||||
// Changes to this file may cause incorrect behavior and will be lost if
|
||||
// the code is regenerated.
|
||||
// </auto-generated>
|
||||
//------------------------------------------------------------------------------
|
||||
|
||||
namespace SqlLoadGenerator.Properties {
|
||||
|
||||
|
||||
[global::System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
|
||||
[global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.VisualStudio.Editors.SettingsDesigner.SettingsSingleFileGenerator", "14.0.0.0")]
|
||||
internal sealed partial class Settings : global::System.Configuration.ApplicationSettingsBase {
|
||||
|
||||
private static Settings defaultInstance = ((Settings)(global::System.Configuration.ApplicationSettingsBase.Synchronized(new Settings())));
|
||||
|
||||
public static Settings Default {
|
||||
get {
|
||||
return defaultInstance;
|
||||
}
|
||||
}
|
||||
|
||||
[global::System.Configuration.UserScopedSettingAttribute()]
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute()]
|
||||
[global::System.Configuration.DefaultSettingValueAttribute("6000000")]
|
||||
public int Spike_NumRowsToInsert {
|
||||
get {
|
||||
return ((int)(this["Spike_NumRowsToInsert"]));
|
||||
}
|
||||
set {
|
||||
this["Spike_NumRowsToInsert"] = value;
|
||||
}
|
||||
}
|
||||
|
||||
[global::System.Configuration.UserScopedSettingAttribute()]
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute()]
|
||||
[global::System.Configuration.DefaultSettingValueAttribute("1")]
|
||||
public int Spike_NumParallelClients {
|
||||
get {
|
||||
return ((int)(this["Spike_NumParallelClients"]));
|
||||
}
|
||||
set {
|
||||
this["Spike_NumParallelClients"] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
<?xml version='1.0' encoding='utf-8'?>
|
||||
<SettingsFile xmlns="http://schemas.microsoft.com/VisualStudio/2004/01/settings" CurrentProfile="(Default)" GeneratedClassNamespace="SqlLoadGenerator.Properties" GeneratedClassName="Settings">
|
||||
<Profiles />
|
||||
<Settings>
|
||||
<Setting Name="Spike_NumRowsToInsert" Type="System.Int32" Scope="User">
|
||||
<Value Profile="(Default)">6000000</Value>
|
||||
</Setting>
|
||||
<Setting Name="Spike_NumParallelClients" Type="System.Int32" Scope="User">
|
||||
<Value Profile="(Default)">1</Value>
|
||||
</Setting>
|
||||
</Settings>
|
||||
</SettingsFile>
|
||||
-70
@@ -1,70 +0,0 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<Project ToolsVersion="14.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
|
||||
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
|
||||
<PropertyGroup>
|
||||
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
|
||||
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
|
||||
<ProjectGuid>{59486E0E-6932-4191-94FF-04082DB46472}</ProjectGuid>
|
||||
<OutputType>Exe</OutputType>
|
||||
<AppDesignerFolder>Properties</AppDesignerFolder>
|
||||
<RootNamespace>SqlLoadGenerator</RootNamespace>
|
||||
<AssemblyName>SqlLoadGenerator</AssemblyName>
|
||||
<TargetFrameworkVersion>v4.6.1</TargetFrameworkVersion>
|
||||
<FileAlignment>512</FileAlignment>
|
||||
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
|
||||
<PlatformTarget>AnyCPU</PlatformTarget>
|
||||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>full</DebugType>
|
||||
<Optimize>false</Optimize>
|
||||
<OutputPath>bin\Debug\</OutputPath>
|
||||
<DefineConstants>DEBUG;TRACE</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
|
||||
<PlatformTarget>AnyCPU</PlatformTarget>
|
||||
<DebugType>pdbonly</DebugType>
|
||||
<Optimize>true</Optimize>
|
||||
<OutputPath>bin\Release\</OutputPath>
|
||||
<DefineConstants>TRACE</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<Reference Include="System" />
|
||||
<Reference Include="System.Configuration" />
|
||||
<Reference Include="System.Core" />
|
||||
<Reference Include="System.Xml.Linq" />
|
||||
<Reference Include="System.Data.DataSetExtensions" />
|
||||
<Reference Include="Microsoft.CSharp" />
|
||||
<Reference Include="System.Data" />
|
||||
<Reference Include="System.Net.Http" />
|
||||
<Reference Include="System.Xml" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Compile Include="Program.cs" />
|
||||
<Compile Include="Properties\AssemblyInfo.cs" />
|
||||
<Compile Include="Properties\Settings.Designer.cs">
|
||||
<AutoGen>True</AutoGen>
|
||||
<DesignTimeSharedInput>True</DesignTimeSharedInput>
|
||||
<DependentUpon>Settings.settings</DependentUpon>
|
||||
</Compile>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<None Include="App.config" />
|
||||
<None Include="Properties\Settings.settings">
|
||||
<Generator>SettingsSingleFileGenerator</Generator>
|
||||
<LastGenOutput>Settings.Designer.cs</LastGenOutput>
|
||||
</None>
|
||||
</ItemGroup>
|
||||
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
|
||||
<!-- To modify your build process, add your task inside one of the targets below and uncomment it.
|
||||
Other similar extension points exist, see Microsoft.Common.targets.
|
||||
<Target Name="BeforeBuild">
|
||||
</Target>
|
||||
<Target Name="AfterBuild">
|
||||
</Target>
|
||||
-->
|
||||
</Project>
|
||||
@@ -1 +0,0 @@
|
||||
Coming soon
|
||||
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 132 KiB |
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 132 KiB |
@@ -1,95 +0,0 @@
|
||||
# In-Memory & Columnar Store Code Snack (SQL Server 2016)
|
||||
In this code snack, developers will experience the benefit of performing real-time operation analytics enabled by leveraging a memory optimized table in combination with a columnstore index. The Visual Studio project contains a load generator that will be used to simulate a write heavy workload. They will initially run the simulator against a disk based table with a clustered index (btree) and take note of the rows inserted per second, and will measure the performance of a provided analytics query while the system is under the heavy write load. They will then author the T-SQL to create the memory optimized table with a columnstore index, update the load generator to target the memory optimized table and observe the improved performance characteristics.
|
||||
|
||||
## Requirements
|
||||
- Visual Studio 2015 with Update 3 (or later)
|
||||
- [SQL Server Data Tools for Visual Studio 2015](https://msdn.microsoft.com/en-us/mt186501)
|
||||
- SQL Server 2016 Developer Edition (or higher)
|
||||
- Your developer machine should have at least 8 GB of RAM
|
||||
|
||||
## Clone the provided project
|
||||
Clone this repo on to your local machine.
|
||||
The recommended path is C:\In-Memory and Columnar\
|
||||
|
||||
## Download the sample data
|
||||
This project requires a sample set of data you will load into SQL Server.
|
||||
Download the data from: [http://bit.ly/2envb8m](http://bit.ly/2envb8m)
|
||||
|
||||
## Create the database and tables
|
||||
1. Open the SqlLoadgenerator solution using Visual Studio 2015.
|
||||
2. From Solution Explorer, expand the SqlGenerator solution, then SQL Resources folder and open "Create Database.sql".
|
||||
3. Adjust the file paths for the FILENAME attributes if you installed SQL Server to a different location.
|
||||
4. Select the Execute button
|
||||
5. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
|
||||
6. Wait for the script to complete successfully.
|
||||
7. Within Visual Studio, open "Create Table- Disk Based.sql"
|
||||
8. Execute the script to create the DataPointsDiskBased table.
|
||||
This table will be used to store simulated IoT device telemetry, using traditional disk based table as well as clustered and non-clustered indexes on the fields commonly used in both point queries and analytic queries.
|
||||
9. Within Visual Studio, open "Create Table- In Memory.sql"
|
||||
10. Execute the script to create the DataPointsInMem table.
|
||||
This table will be used to store the same simulated IoT device telemetry, but this time using a memory optimized table as well as clustered column store index against all fields (which will support analytic queries) and non-clustered hash indexes on the id field (which will support point lookups common to transactional queries).
|
||||
```
|
||||
CREATE TABLE [DataPointsInMem] (
|
||||
-- ID should be a Primary Key, fields with a b-tree or hash index
|
||||
Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
|
||||
[Value] decimal(18,5),
|
||||
[TimestampUtc] datetime,
|
||||
DeviceId int,
|
||||
-- This table should have a columnar index
|
||||
INDEX Transactions_CCI CLUSTERED COLUMNSTORE
|
||||
) WITH (
|
||||
-- This should be an in-memory table
|
||||
MEMORY_OPTIMIZED = ON
|
||||
);
|
||||
|
||||
-- In-memory tables should auto-elevate their transaction level to Snapshot
|
||||
ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ;
|
||||
```
|
||||
|
||||
## Load initial data
|
||||
|
||||
1. Within Visual Studio, open "Load Sample Data.sql"
|
||||
2. Adjust the path to the DataPoint.bcp file so it matches the location of your project and save the script.
|
||||
3. Execute the script to load each table with 4 million rows worth of sample data. This will take some time to complete.
|
||||
|
||||
## Execute the sample analytics query
|
||||
|
||||
1. Within Visual Studio, open "SampleQueries - DiskBased.sql".
|
||||
2. Execute the script to summarize the time series data stored in the disk based table.
|
||||
3. When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute. Query time is shown in the bottom right of the document window in Visual Studio.
|
||||
![alt text][Disk Based Results]
|
||||
|
||||
[Disk Based Results]: Images/DiskBasedResults.png "Disk Based Results"
|
||||
4. Now, execute the script to summarize the time series data stored in the memory-optimized table, in "SampleQueries - InMemory.sql".
|
||||
When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute.
|
||||
You should notice that the performance of the query against the memory-optimized table runs between 2x-10x faster than the same query, running against the same data stored in a disk based table. Query time is shown in the bottom right of the document window in Visual Studio.
|
||||
![alt text][In-Memory Results]
|
||||
|
||||
[In-Memory Results]: Images/InMemoryResults.png "In-Memory Results"
|
||||
|
||||
## Execute the queries under load
|
||||
|
||||
1. Within Visual Studio, Solution Explorer, expand the SqlLoadGenerator project and then open "App.config".
|
||||
2. Locate the connection string with the name "SqlConnection" and modify it so it points to your instance of SQL Server 2016.
|
||||
3. Save the App.config.
|
||||
4. From the Debug menu, select Start Without Debugging.
|
||||
5. At the prompt, choose option 1 to target the disk based table.
|
||||
You should see log entries when every 1000 rows are inserted.
|
||||
Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
|
||||
6. Open "SampleQueries - DiskBased.sql".
|
||||
7. Execute the script to summarize the time series data stored in the disk based table.
|
||||
8. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
|
||||
9. Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
|
||||
10. Close the console load generator.
|
||||
11. Run the SqlLoadGenerator again.
|
||||
This time at the prompt, choose option 2 to target the memory-optimized table.
|
||||
You should see log entries when every 1000 rows are inserted.
|
||||
12. Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
|
||||
13. Open "SampleQueries - In Memory.sql".
|
||||
14. Execute the script to summarize the time series data stored in the disk based table.
|
||||
15. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
|
||||
Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
|
||||
16. Close the console load generator.
|
||||
|
||||
## Conclusion
|
||||
You should observe that while neither query was affected by the heavy insert load, the query against the analytics query continued to run 2x-10x faster than the same query against the disk-based table.
|
||||
Binary file not shown.
Reference in New Issue
Block a user