Init Commit

samples/features/in-memory/code-snacks/sqlonwindows/in-mem_columnstore/README.md

@@ -0,0 +1,95 @@
+# In-Memory & Columnar Store Code Snack
+In this code snack, developers will experience the benefit of performing real-time operation analytics enabled by leveraging a memory optimized table in combination with a columnstore index. The Visual Studio project contains a load generator that will be used to simulate a write heavy workload. They will initially run the simulator against a disk based table with a clustered index (btree) and take note of the rows inserted per second, and will measure the performance of a provided analytics query while the system is under the heavy write load. They will then author the T-SQL to create the memory optimized table with a columnstore index, update the load generator to target the memory optimized table and observe the improved performance characteristics.
+
+## Requirements
+- Visual Studio 2015 with Update 3 (or later)
+- [SQL Server Data Tools for Visual Studio 2015](https://msdn.microsoft.com/en-us/mt186501) 
+- SQL Server 2016 Developer Edition (or higher)
+- Your developer machine should have at least 8 GB of RAM
+
+## Clone the provided project
+Clone this repo on to your local machine.
+The recommended path is C:\In-Memory and Columnar\
+
+## Download the sample data
+This project requires a sample set of data you will load into SQL Server.
+Download the data from: [http://bit.ly/2envb8m](http://bit.ly/2envb8m)
+
+## Create the database and tables
+1. Open the SqlLoadgenerator solution using Visual Studio 2015.
+2. From Solution Explorer, expand the SqlGenerator solution, then SQL Resources folder and open "Create Database.sql".
+3. Adjust the file paths for the FILENAME attributes if you installed SQL Server to a different location.
+4. Select the Execute button
+5. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
+6. Wait for the script to complete successfully.
+7. Within Visual Studio, open "Create Table- Disk Based.sql"
+8. Execute the script to create the DataPointsDiskBased table.
+This table  will be used to store simulated IoT device telemetry, using traditional disk based table as well as clustered and non-clustered indexes on the fields commonly used in both point queries and analytic queries.
+9. Within Visual Studio, open "Create Table- In Memory.sql"
+10. Execute the script to create the DataPointsInMem table.
+This table will be used to store the same simulated IoT device telemetry, but this time using a memory optimized table as well as clustered column store index against all fields (which will support analytic queries) and non-clustered hash indexes on the id field (which will support point lookups common to transactional queries).
+```
+CREATE TABLE [DataPointsInMem] (
+	-- ID should be a Primary Key, fields with a b-tree or hash index
+	Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
+	[Value] decimal(18,5),
+	[TimestampUtc] datetime,
+	DeviceId int,
+	--  This table should have a columnar index
+	INDEX Transactions_CCI CLUSTERED COLUMNSTORE
+) WITH (
+	--  This should be an in-memory table
+	MEMORY_OPTIMIZED = ON
+);
+
+--  In-memory tables should auto-elevate their transaction level to Snapshot
+ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ; 
+```
+
+## Load initial data
+
+1. Within Visual Studio, open "Load Sample Data.sql"
+2. Adjust the path to the DataPoint.bcp file so it matches the location of your project and save the script.
+3. Execute the script to load each table with 4 million rows worth of sample data. This will take some time to complete.
+
+## Execute the sample analytics query
+
+1. Within Visual Studio, open "SampleQueries - DiskBased.sql".
+2. Execute the script to summarize the time series data stored in the disk based table.
+3. When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute. Query time is shown in the bottom right of the document window in Visual Studio.
+![alt text][Disk Based Results]
+
+[Disk Based Results]: images/DiskBasedResults.png "Disk Based Results"
+4. Now, execute the script to summarize the time series data stored in the memory-optimized table, in "SampleQueries - InMemory.sql".
+When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute.
+You should notice that the performance of the query against the memory-optimized table runs between 2x-10x faster than the same query, running against the same data stored in a disk based table. Query time is shown in the bottom right of the document window in Visual Studio.
+![alt text][In-Memory Results]
+
+[In-Memory Results]: images/InMemoryResults.png "In-Memory Results"
+
+## Execute the queries under load
+
+1. Within Visual Studio, Solution Explorer, expand the SqlLoadGenerator project and then open "App.config".
+2. Locate the connection string with the name "SqlConnection" and modify it so it points to your instance of SQL Server 2016.
+3. Save the App.config.
+4. From the Debug menu, select Start Without Debugging.
+5. At the prompt, choose option 1 to target the disk based table.
+You should see log entries when every 1000 rows are inserted.
+Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
+6. Open "SampleQueries - DiskBased.sql".
+7. Execute the script to summarize the time series data stored in the disk based table.
+8. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
+9. Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
+10. Close the console load generator.
+11. Run the SqlLoadGenerator again.
+This time at the prompt, choose option 2 to target the memory-optimized table.
+You should see log entries when every 1000 rows are inserted.
+12. Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
+13. Open "SampleQueries - In Memory.sql".
+14. Execute the script to summarize the time series data stored in the disk based table.
+15. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
+Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
+16. Close the console load generator.
+
+## Conclusion
+You should observe that while neither query was affected by the heavy insert load, the query against the analytics query continued to run 2x-10x faster than the same query against the disk-based table.

samples/features/r-services/code-snacks/clustering_in_R/README.md

@@ -0,0 +1,83 @@
+# Clustering in R
+In this code snack, developers will experience authoring R code to help them run a clustering exercise that “magically” groups data into distinct populations by using an unsupervised clustering algorithm, k-means. The k-means script will be packaged within a SQL stored procedure for convenient execution from a .NET application.
+
+## About Clustering
+The goal of a clustering algorithm is to look at an input set of data and attempt to identify groups of data by virtue of the similarity between the features of each example in the data set. What makes clustering algorithms particularly powerful is that they do not need a training step like the other algorithms— you simply provide them the data, tell them how many clusters you want to create and they assign each example to a group. The canonical clustering algorithm is k-means. 
+
+## Requirements
+- Visual Studio 2015 with Update 3 (or later)
+- SQL Server 2016 Developer Edition (or higher)
+
+## Required SQL Server Configuration
+-	Make sure that your installation of SQL Server includes R Services, see [https://msdn.microsoft.com/en-us/library/mt696069.aspx](https://msdn.microsoft.com/en-us/library/mt696069.aspx)
+-	Using SQL Server Configuration Manager (which is launched from the Start menu), make sure that TCP/IP connections are enabled to your instance of SQL Server (under SQL Server Network Configuration). 
+![alt text][SQL Config]
+
+[SQL Config]: images/SqlConfig.png "SQL Server Network Configuration"
+- Be sure that the SQL Server, SQL Server Launchpad and SQL Server Browser  services are all running.
+
+## Clone the provided project
+Clone this repo on to your local machine.
+The recommended path is C:\Clustering in R
+
+## Create the database and tables
+1. Open the ClusteringConsole.sln solution using Visual Studio 2015
+2. From Solution Explorer, expand the ClusteringConsole solution, then Solution Items folder and open “Create Sample Database.sql”.
+3. Adjust the file path for the FROM clause in the BULK INSERT statement if you cloned the project to a different location.
+4. Select the Execute button
+5. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
+6. Wait for the script to complete successfully.
+
+## Create the Clustering Stored Procedure
+1. Within Visual Studio, open “Create Procedure ClusterTaxiData.sql”.
+This stored procedure queries the data in the nyctaxi_features table and creates four clusters of data based on the passenger_count (the number of passengers in the taxi cab) and direct_distance (the distance traveled, measured as the crow flies). It uses the rxKmeans method to accomplish this, which runs the K-Means algorithm to group the data into the configured number of clusters (four clusters in this case). The formula syntax "~ passenger_count + direct_distance” used in the first parameter simply means to cluster around those two columns from the input data.
+```
+CREATE PROCEDURE [dbo].[ClusterTaxiData]  
+AS  
+BEGIN  
+  DECLARE @inquery nvarchar(max) = N'  
+    select tipped,  passenger_count, trip_time_in_secs, trip_distance, direct_distance   
+    from nyctaxi_features   
+'  
+  
+  EXEC sp_execute_external_script
+	@language = N'R',  
+    @script = N'  
+
+	## Cluster the data 
+	clusters <- rxKmeans(~ passenger_count + direct_distance, data = InputDataSet, numClusters = 4, algorithm = "lloyd")  
+
+	## Return the result (by convention the result data set is retrieved from a variable named OutputDataSet).
+	OutputDataSet <- as.data.frame(clusters$centers) ;
+
+									',  
+    @input_data_1 = @inquery 
+	WITH RESULT SETS ((passenger_count real, direct_distance real))
+  ;  
+END  
+GO  
+```
+2. Execute the script to create the stored procedure.
+
+## Execute the Clustering Stored Procedure
+1. Within Visual Studio, open “Execute Procedure ClusterTaxiData.sql”.
+2. Select the Execute button
+3. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
+4. Wait for the script to complete successfully.
+5. Observe the results, you should have four clusters of data, each a row in the results. You might interpret these results in order as short trips with one passenger, long trips with two passengers, moderate trips with two passengers and short trips with lots of passengers.
+
+![alt text][Clustering Results]
+
+[Clustering Results]: images/ClusteringResults.png "Clustering Results"
+
+## Leverage Clustering from an Application
+1. Within Visual Studio, open app.config located underneath the SqlSecurity project in Solution Explorer.
+2. Set the connectionString value so that it points to your SQL Server.
+3. Save the file.
+4. From the Debug menu, select Start Without Debugging.
+
+Observe the clusters for the taxi rides as retrieved by the application, you have now integrated machine learning into your console application!
+
+![alt text][Application Results]
+
+[Application Results]: images/ApplicationResults.png "Application Results"

samples/features/security/code-snacks/sqlonwindows/README.md

@@ -0,0 +1,123 @@
+#  Dynamic Data Masking & Row Level Security Code Snack
+In this code snack, developers will create a database having human resources data, including a table containing simulated employee pay data. They will be guided thru the sample data to highlight the sensitive information it contains (e.g., social security numbers and salaries) and then configuring the masking of the sensitive data. In addition, they will enable Row Level Security to handle three different roles: contractors (who have no visibility to any rows except their own in the table), HR (who can view all employee rows except those of executives) and Executives (who can view all employee rows). They will complete a .NET application that queries the database to see the differing outcomes that result based on Row Level Security policy.
+
+## Requirements
+- Visual Studio 2015 with Update 3 (or later)
+- SQL Server 2016 Developer Edition (or higher)
+
+## Clone the provided project
+Clone this repo on to your local machine.
+The recommended path is C:\SQL Server Security
+
+## Create the database and tables
+1. Open the SqlSecurity.sln solution using Visual Studio 2015.
+2. From Solution Explorer, expand the SqlSecurity solution, then Solution Items folder and open “Create Sample Database.sql”.
+3. Adjust the file paths for the variables SqlSamplesDatabasePath and SqlSamplesSourceDataPath if you cloned to a different location.
+4. Using the toolbar, select SQLCMD Mode button so that query runs in that mode. 
+
+![alt text][SQLCMD]
+
+[SQLCMD]: images/sqlcmd.png "SQLCMD Mode Toggle"
+5. Select the Execute button.
+6. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
+7. Wait for the script to complete successfully.
+
+## Explore the Sample Data
+1. Within Visual Studio, open “Explore Data.sql"
+2. Execute the script to observe the sensitive fields in this query that summarize the pay rate for employees: NationalIDNumber (e.g., social security number) and Rate (e.g., pay rate)
+3. Notice the employee table has the NationalIDNumber which is sensitive field and the EmployePayHistory table has the Rate field which is sensitive because it captures the employees rate of pay.
+![alt text][Explore Data]
+
+[Explore Data]: images/exploredatacleartext.png "Explore Data"
+
+## Configure Masking
+1. Within Visual Studio, open “Configure Masking.sql"
+2. Execute the script to create mask both the NationalIDNumber and Rate fields.
+```
+-- Mask the NationalIDNumber column so it only displays the last two digits of field (for example: XX-XXX-XX43)
+ALTER TABLE HumanResources.Employee
+ALTER COLUMN NationalIDNumber ADD MASKED WITH(FUNCTION = 'partial(0,"XX-XXX-XX",2)')
+
+
+-- Mask the rate by providing a random value in place of the actual rate
+ALTER TABLE HumanResources.EmployeePayHistory
+ALTER COLUMN Rate ADD MASKED WITH (FUNCTION = 'random(20,150)')
+```
+3. Return to “Explore Data.sql” and execute this script again.
+4. Observe that even though you enabled masking on the table, these fields are still available to you (the administrative user) in their original unmasked format.
+5. To view the results with the masks applied, create a new user who can query from the database who does not have priveleges to see the unmasked data (in other words, they will always see the masked data).
+6. Open “Create Contract User.sql” and execute it to create a new login and user with the name Contractor and password Abc!1234.
+7. Return to “Explore Data.sql” and execute this script again.
+8. Select the Change Connection button from toolbar, and login to your SQL Server instance with the Contractor login (Login: Contractor and Password: Abc!1234)
+9. Execute this script again.
+10. Observe that now the NationalIDNumber only displays the last two digits, and the Rate values are different from before.
+![alt text][Masked Data]
+
+[Masked Data]: images/maskedata.png "Masked Data"
+
+## Configure Row Level Security
+1. Next, consider the scenario where you want to enforce a policy where only Executive users in the organization can see all employee rows in the Employee table. Users in the Human resources department can see all rows except those of the executives. Finally, all other users can only see their row.
+2. This is something you can accomplish using Row Level Security in a fashion that “just works” and applies transparently to the user issuing the query.
+3. Within Visual Studio, open “Configure Row Level Security.sql”.
+4. Execute the script. This will create a schema (to hold our security related functions), a predicate function that filters the rows based upon the user performing the querying, and a policy that is applied to the Employee table that uses the predicate function to filter the result set to only the rows the user should be seeing.
+```
+-- Best practice, create a schema to hold security predicates
+CREATE SCHEMA Security;  
+GO 
+
+
+-- Create the predicate function
+CREATE FUNCTION Security.LimitAccess(@LoginID nvarchar(256), @OrganizationLevel smallint)
+	RETURNS TABLE
+	WITH SCHEMABINDING
+AS
+    RETURN SELECT 1 as LimitAccess_Result
+	FROM HumanResources.EmployeeDepartmentHistory deptHist INNER JOIN HumanResources.Employee emp
+	ON deptHist.BusinessEntityID = emp.BusinessEntityID
+	WHERE	(emp.LoginID = CAST(SESSION_CONTEXT(N'LoginID') AS nvarchar(256)) AND EndDate is NULL AND DepartmentID = 9 AND @OrganizationLevel > 1) OR
+			(emp.LoginID = CAST(SESSION_CONTEXT(N'LoginID') AS nvarchar(256)) AND EndDate is NULL AND DepartmentID = 16) OR
+			CAST(SESSION_CONTEXT(N'LoginID') AS nvarchar(256)) = @LoginID;
+GO
+
+
+-- Create a policy that applies the predicate
+CREATE SECURITY POLICY Security.HumanResourcesPolicy
+	ADD FILTER PREDICATE Security.LimitAccess(LoginID, OrganizationLevel) ON HumanResources.Employee
+	WITH (STATE = ON);
+GO
+```
+5. Now, open “Explore Data with RLS.sql”. This query will show the differing result sets that appear for different users when the policy is in action. Notice that this query does not rely on the credentials used to connect to SQL Server, but rather the login store in the session context.
+6. This is a useful pattern when you have an application that uses one connection string to SQL Server, but is operating in an environment where your application handles the login, and that login is different from the credentials used to access SQL Server. You can uses this application login information to inform the Row Level Security policy.
+```
+-- SET the context to Contractor user and make it immutable for the duration of the connection
+EXEC sp_set_session_context @key=N'LoginID', @value=N'adventure-works\lynn0';  
+
+-- Query the table as usual: observe that only the one row is returned
+SELECT * FROM HumanResources.Employee;  
+```
+7. Execute the query. Observe the different result sets that appear for the exact same query— they are made different only because of the LoginID session context provided.
+![alt text][RLS Data]
+
+[RLS Data]: images/rlsresults.png "RLS Data"
+
+
+## Leverage Row Level Security from an Application
+1. Let’s put Row Level Security to work within the context of an application, in this case a .NET application.
+2. Within Visual Studio, open app.config located underneath the SqlSecurity project in Solution Explorer.
+3. Set the connectionString value so that it points to your SQL Server.
+4. Save the file.
+5. From the Debug menu, select Start Without Debugging.
+6. In the console dialog that appears, select option 1 to run the query as a Contractor.
+7. Observe the query that is run and that only 1 row is returned (the row for that user in the employee table).
+
+![alt text][RLS in App]
+
+[RLS in App]: images/rlsinapp.png "RLS in App"
+
+8. Run the console again, this time select option 2 (Human Resources).
+9. Observe that the same query is run as before, but 283 rows are returned. This represents all of the non-executive rows in the employee table.
+10. Run the console on last time and select option 3 (Executive).
+11. Observe that the same query is run, but that 290 rows are returned. This represents that all employee rows are returned, because an executive can should have access to all rows.
+12. To see how this is implemented, within Visual Studio, open Program.cs.
+13. Take a look at the QueryEmployeeTable method. Observe that is implements the same pattern as was shown in “Explore Data with RLS.sql”. First, the stored procedure sp_set_session_context is executed using the loginID selected (from the list of options made available when the console app starts) as the loginID parameter. Second, the query that counts all rows in the employee table is run.
+14. Notice that the policy is applied transparently to the application- the only change made between the queries is the loginID used to identify the user.