Merge pull request #349 from JocaPC/master

Adding CLR transitive closure sample
2025-12-08 14:58:54 +00:00 · 2017-12-22 23:14:52 +01:00
parent dc0494caaf 20a1e56cc2
commit 5dca06b633
7 changed files with 559 additions and 0 deletions
@@ -0,0 +1,7 @@
+*.cproj.user
+.vs/*
+.vscode/*
+bin/*
+obj/*
+Properties/PublishProfiles/*
+TransitiveClosure.sql
@@ -0,0 +1,36 @@
+using System.Reflection;
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+
+// General Information about an assembly is controlled through the following
+// set of attributes. Change these attribute values to modify the information
+// associated with an assembly.
+[assembly: AssemblyTitle("TransitiveClosureAggregatorLibrary")]
+[assembly: AssemblyDescription("")]
+[assembly: AssemblyConfiguration("")]
+[assembly: AssemblyCompany("")]
+[assembly: AssemblyProduct("TransitiveClosureAggregatorLibrary")]
+[assembly: AssemblyCopyright("Copyright ©  2017")]
+[assembly: AssemblyTrademark("")]
+[assembly: AssemblyCulture("")]
+
+// Setting ComVisible to false makes the types in this assembly not visible
+// to COM components.  If you need to access a type in this assembly from
+// COM, set the ComVisible attribute to true on that type.
+[assembly: ComVisible(false)]
+
+// The following GUID is for the ID of the typelib if this project is exposed to COM
+[assembly: Guid("097ef341-926d-4dd2-a434-08e9980d6089")]
+
+// Version information for an assembly consists of the following four values:
+//
+//      Major Version
+//      Minor Version
+//      Build Number
+//      Revision
+//
+// You can specify all the values or you can default the Build and Revision Numbers
+// by using the '*' as shown below:
+// [assembly: AssemblyVersion("1.0.*")]
+[assembly: AssemblyVersion("1.0.0.0")]
+[assembly: AssemblyFileVersion("1.0.0.0")]
@@ -0,0 +1,104 @@
+# Implementing Transitive Closure Clustering in SQL Server using CLR UDF 
+SQL Database don't have built-in support for transitive closure clustering, so the only workaround is to implement this algorithm in .Net framework and expose it as T-SQL function.
+A discussion on the problem, the algorithm and a pure T-SQL based solution can be found here: 
+- [Transitive Closure Clustering with SQL Server, UDA and JSON](https://medium.com/@mauridb/transitive-closure-clustering-with-sql-server-uda-and-json-dade18953fd2)
+- [T-SQL Puzzle Challenge Grouping Connected Items](http://www.itprotoday.com/microsoft-sql-server/t-sql-puzzle-challenge-grouping-connected-items)
+
+This code sample demonstrates how to create CLR User-Defined aggregate that implements clustering.
+
+### Contents
+
+[About this sample](#about-this-sample)<br/>
+[Build the CLR/TransitiveClosure aggregate](#build-functions)<br/>
+[Add RegEx functions to your SQL database](#add-functions)<br/>
+[Test the functions](#test)<br/>
+[Disclaimers](#disclaimers)<br/>
+
+<a name=about-this-sample></a>
+
+## About this sample 
+1. **Applies to:** SQL Server 2016+ Enterprise / Developer / Evaluation Edition
+2. **Key features:**
+    - CLR, JSON
+3. **Programming Language:** .NET C#
+4. **Author:** [Davide Mauri](https://github.com/yorek), Jovan Popovic [jovanpop-msft]
+
+<a name=build-functions></a>
+
+## Build the CLR/TransitiveClosure aggregate
+
+1. Download the source code and open the solution using Visual Studio.
+2. Change the password in .pfk file and rebuild the solution in **Release** mode.
+3. Open and save TransitiveClosure.tt to generate output T-SQL file that will contain script that inserts .dll file with the Transitive closure clustering aggregate.
+
+<a name=add-functions></a>
+## Add Clustering aggregate to your SQL database
+
+File TransitiveClosure.sql contains the code that will import aggregate into SQL Database.
+
+If you have not added CLR assemblies in your database, you should use the following script to enable CLR:
+```
+sp_configure @configname=clr_enabled, @configvalue=1
+GO
+RECONFIGURE
+GO
+```
+
+Once you enable CLR, you can use the T-SQL script to add the clustering aggregate. The script depends on the location where you have built the project, and might look like:
+```
+CREATE ASSEMBLY TransitiveClosure FROM 'D:\GitHub\sql-server-samples\samples\features\sql-clr\TransitiveClosure\bin\Release\TransitiveClosureAggregatorLibrary.dll' WITH PERMISSION_SET = SAFE;
+GO
+
+CREATE SCHEMA TC;
+GO
+
+CREATE AGGREGATE TC.CLUSTERING(@id1 INT, @id2 INT)  
+RETURNS NVARCHAR(MAX)  
+EXTERNAL NAME TransitiveClosure.[TransitiveClosure.Aggregate]; 
+```
+
+This code will import assembly in SQL Database and add an aggregate that provides clustering functionalities.
+
+<a name=test></a>
+
+## Test the function
+
+Once you create the assembly and expose the aggregate, you can use it to cluster some relational data in T-SQL code:
+
+```
+declare @edges table(n1 int, n2 int);
+
+insert into @edges
+values 
+    (1,2),(2,3),(3,4),(4,5),(2,21),(2,22),
+    (7,8),(8,9),(9,10);
+
+select TC.CLUSTERING(n1,n2)
+from @edges;
+```
+The result will be JSON document that groups the numbers that belong to the same cluster.
+```javascript
+{
+    "0":[1,2,3,4,5,21,22],
+    "1":[7,8,9,10]
+}
+```
+You can transform this JSON document into relational formatusing **OPENJSON** function:
+```
+select cluster = [key], elements = value
+from openjson(
+    (select TC.CLUSTERING(n1,n2) from @edges)
+);
+```
+The result of this query is:
+
+|cluster|elements|
+|---|---|
+|0|[1,2,3,4,5,21,22]|
+|1|[7,8,9,10]|
+
+<a name=disclaimers></a>
+
+## Disclaimers
+The code included in this sample is not intended to be a set of best practices on how to build scalable enterprise grade applications. This is beyond the scope of this sample.
+
@@ -0,0 +1,23 @@
+<#@output extension=".sql"#>
+<#@ template language="C#" hostspecific="True" #>
+
+DROP AGGREGATE IF EXISTS TC.CLUSTERING;
+GO
+
+DROP SCHEMA IF EXISTS TC;
+GO
+
+--Drop the assembly if it already exists
+DROP ASSEMBLY IF EXISTS TransitiveClosure;
+GO
+
+--Create the assembly
+CREATE ASSEMBLY TransitiveClosure FROM '<#= this.Host.ResolvePath("bin\\Release\\TransitiveClosureAggregatorLibrary.dll") #>' WITH PERMISSION_SET = SAFE;
+GO
+
+CREATE SCHEMA TC;
+GO
+
+CREATE AGGREGATE TC.CLUSTERING(@id1 INT, @id2 INT)  
+RETURNS NVARCHAR(MAX)  
+EXTERNAL NAME TransitiveClosure.[TransitiveClosure.Aggregate]; 
@@ -0,0 +1,316 @@
+using System;
+using System.IO;
+using System.Data.SqlTypes;
+using System.Text;
+using Microsoft.SqlServer.Server;
+using System.Collections.Generic;
+using System.Collections;
+
+namespace TransitiveClosure
+{
+    /// <summary>
+    /// Class that represents a group of numbers in the same cluster.
+    /// </summary>
+    public class Group: IEnumerable<int>
+    {
+        private int? _groupRoot = null;
+                
+        private Dictionary<int, bool> _group = new Dictionary<int, bool>();
+        public Dictionary<int, bool>.KeyCollection Elements => _group.Keys;
+        public int Count => _group.Keys.Count;
+
+        public bool ContainsElement(int element)
+        {
+            return _group.ContainsKey(element);
+        }
+
+        /// <summary>
+        /// Adds a pair of numbers to a group.
+        /// </summary>
+        /// <param name="from"></param>
+        /// <param name="to"></param>
+        public void AddUnique(int from, int to)
+        {
+            if (_groupRoot == null) _groupRoot = from; 
+            this.AddIfNotExists(from);
+            this.AddIfNotExists(to);
+        }
+        /// <summary>
+        /// Adds the element into the current group. 
+        /// </summary>
+        /// <param name="element">The number that should be added.</param>
+        public void Add(int element)
+        {
+            if (_groupRoot == null) _groupRoot = element;
+            _group.Add(element, true);
+        }
+
+        /// <summary>
+        /// Adds the element to a group if it is not already there. 
+        /// </summary>
+        /// <param name="element"></param>
+        public void AddIfNotExists(int element)
+        {
+            if (!_group.ContainsKey(element))
+            {
+                _group.Add(element, true);
+            }
+        }
+
+        public void MergeWith(Group source)
+        {
+            foreach (var e in source.Elements)
+            {
+                this.AddIfNotExists(e);
+            }
+        }
+
+        public IEnumerator<int> GetEnumerator()
+        {
+            foreach(var e in _group.Keys)
+            {
+                yield return e;
+            }
+        }
+
+        IEnumerator IEnumerable.GetEnumerator()
+        {
+            return (IEnumerator)GetEnumerator();
+        }
+
+        public override string ToString()
+        {
+            return string.Format($"[{_groupRoot.Value}]");
+        }
+    }
+
+    public class GroupSet: IEnumerable<Group>
+    {
+        private List<Group> _groupSet = new List<Group>();
+        // This keeps about the distinct numbers and in which group they are
+        private Dictionary<int, Group> _numbers = new Dictionary<int, Group>();
+        private int _merges = 0;
+
+        public int Groups => _groupSet.Count;
+        public int Numbers => _numbers.Count;
+        public int Merges => _merges;
+
+        public void Add(Group group)
+        {
+            _groupSet.Add(group);
+
+            foreach(int e in group)
+            {
+                if (!_numbers.ContainsKey(e))
+                    _numbers.Add(e, group);
+                else
+                    throw new ApplicationException("Element is already assigned to a group");
+            }
+        }
+
+        public List<Group> FindInGroups(int from, int to)
+        {
+            var result = new List<Group>();
+            
+            if (_numbers.ContainsKey(from)) result.Add(_numbers[from]);
+            if (_numbers.ContainsKey(to)) result.Add(_numbers[to]);
+
+            return result;                        
+        }
+
+        public void AddPair(int from, int to)
+        {
+            //Find if the inputValue is already in a group
+            var foundInGroups = FindInGroups(from, to);
+
+            // no item matches: create a new group and add both the values to it            
+            if (foundInGroups.Count == 0)
+            {
+                var ng = new Group();
+                ng.AddIfNotExists(from);
+                ng.AddIfNotExists(to);
+
+                _groupSet.Add(ng);
+
+                if (!_numbers.ContainsKey(from)) _numbers.Add(from, ng);
+                if (!_numbers.ContainsKey(to)) _numbers.Add(to, ng);
+            }
+
+            // one item match, add the related item to the same group
+            if (foundInGroups.Count == 1)
+            {
+                var g = foundInGroups[0];
+                g.AddUnique(from, to);
+
+                if (!_numbers.ContainsKey(from)) _numbers.Add(from, g); else _numbers[from] = g;
+                if (!_numbers.ContainsKey(to)) _numbers.Add(to, g); else _numbers[to] = g;
+            }
+
+            // if there is a match for both items but in two different groups
+            // merge them into just one group and delete the other
+            if (foundInGroups.Count == 2)
+            {
+                var g1 = foundInGroups[0];
+                var g2 = foundInGroups[1];
+
+                if (g1 == g2) return;
+
+                // Always move the smaller group
+                if (g2.Count > g1.Count)
+                {
+                    var t = g1;
+                    g1 = g2;
+                    g2 = t;
+                }
+
+                if (!_numbers.ContainsKey(from)) _numbers.Add(from, g1); else _numbers[from] = g1;
+                if (!_numbers.ContainsKey(to)) _numbers.Add(to, g1); else _numbers[to] = g1;
+
+                g1.MergeWith(g2);              
+
+                foreach(var e in g2)
+                {
+                    if (!_numbers.ContainsKey(e)) _numbers.Add(e, g1); else _numbers[e] = g1;
+                }
+
+                _merges += 1;
+
+                _groupSet.Remove(g2);
+            }
+        }
+
+        public IEnumerator<Group> GetEnumerator()
+        {
+            foreach(var g in _groupSet)
+            {
+                yield return g;
+            }
+        }
+
+        IEnumerator IEnumerable.GetEnumerator()
+        {
+            return (IEnumerator)GetEnumerator();
+        }
+    }
+
+    /// <summary>
+    /// Aggregate that takes a pair of numbers that represents an edge in some graph/relation.
+    /// As an output, returns groups of reachable edges, for example:
+    /// {
+    ///     "0":[1,2,3,4],
+    ///     "1":[5,6,7],
+    ///     "2":[8,9]
+    /// }
+    /// </summary>
+    [Serializable]
+    [SqlUserDefinedAggregateAttribute(Format.UserDefined, MaxByteSize = -1)]
+    public class Aggregate : IBinarySerialize
+    {
+        private GroupSet _groupSet;
+
+        public int Groups => _groupSet.Groups;
+        public int Numbers => _groupSet.Numbers;
+        public int Merges => _groupSet.Merges;
+
+        public void Init()
+        {
+            _groupSet = new GroupSet();
+        }
+
+        public void Accumulate(int inputValue1, int inputValue2)
+        {
+            _groupSet.AddPair(inputValue1, inputValue2);                       
+        }
+
+        public void Merge(Aggregate value)
+        {
+            foreach (var g in value._groupSet)
+            {
+                int? pe = null;
+                foreach (var ce in g)
+                {
+                    if (pe.HasValue)
+                    {
+                        this.Accumulate(pe.Value, ce);
+                    }
+                    pe = ce;
+                }
+            }
+        }     
+
+        public SqlString Terminate()
+        {
+            return this.ToString();
+        }
+
+        public override string ToString()
+        {
+            int c = 0;
+            StringBuilder sb = new StringBuilder();
+            sb.Append("{");
+            foreach (var g in this._groupSet)
+            {
+                sb.Append("\"" + c + "\":[");
+
+                var ea = new int[g.Elements.Count];
+                g.Elements.CopyTo(ea, 0);
+
+                sb.Append(string.Join(",", ea));
+                
+                sb.Append("],");
+
+                c += 1;
+            }
+            if (sb.Length > 1) sb.Remove(sb.Length - 1, 1);
+            sb.Append("}");
+            return sb.ToString();
+        }
+
+        public void Read(BinaryReader r)
+        {
+            if (r == null) throw new ArgumentNullException("r");
+            _groupSet = new GroupSet();
+
+            // Group Count
+            int g = r.ReadInt32();
+
+            // For Each Group
+            for (int j = 0; j < g; j++)
+            {
+                var l = new Group();               
+
+                // List Size (or Values Count)
+                int s = r.ReadInt32();
+
+                // Read values and put them in the list
+                for (int i = 0; i < s; i++)
+                {
+                    l.Add(r.ReadInt32());
+                }
+
+                // Add list to dictionary
+                _groupSet.Add(l);
+            }
+        }
+
+        public void Write(BinaryWriter w)
+        {
+            if (w == null) throw new ArgumentNullException("w IS NULL");
+
+            // Group count
+            w.Write(_groupSet.Groups);
+
+            foreach (var g in _groupSet)
+            {
+                // Values Count
+                w.Write(g.Count);
+
+                // Values
+                foreach (var e in g)
+                {
+                    w.Write(e);
+                }
+            }
+        }
+    }
+}
@@ -0,0 +1,73 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
+  <PropertyGroup>
+    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
+    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
+    <ProjectGuid>{097EF341-926D-4DD2-A434-08E9980D6089}</ProjectGuid>
+    <OutputType>Library</OutputType>
+    <AppDesignerFolder>Properties</AppDesignerFolder>
+    <RootNamespace>TransitiveClosureAggregatorLibrary</RootNamespace>
+    <AssemblyName>TransitiveClosureAggregatorLibrary</AssemblyName>
+    <TargetFrameworkVersion>v4.5.2</TargetFrameworkVersion>
+    <FileAlignment>512</FileAlignment>
+    <TargetFrameworkProfile />
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
+    <DebugSymbols>true</DebugSymbols>
+    <DebugType>full</DebugType>
+    <Optimize>false</Optimize>
+    <OutputPath>bin\Debug\</OutputPath>
+    <DefineConstants>DEBUG;TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+    <Prefer32Bit>false</Prefer32Bit>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
+    <DebugType>pdbonly</DebugType>
+    <Optimize>true</Optimize>
+    <OutputPath>bin\Release\</OutputPath>
+    <DefineConstants>TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+    <Prefer32Bit>false</Prefer32Bit>
+  </PropertyGroup>
+  <PropertyGroup>
+    <SignAssembly>true</SignAssembly>
+  </PropertyGroup>
+  <PropertyGroup>
+    <AssemblyOriginatorKeyFile>tcc.pfx</AssemblyOriginatorKeyFile>
+  </PropertyGroup>
+  <ItemGroup>
+    <Reference Include="System" />
+    <Reference Include="System.Core" />
+    <Reference Include="System.Xml.Linq" />
+    <Reference Include="System.Data.DataSetExtensions" />
+    <Reference Include="Microsoft.CSharp" />
+    <Reference Include="System.Data" />
+    <Reference Include="System.Net.Http" />
+    <Reference Include="System.Xml" />
+  </ItemGroup>
+  <ItemGroup>
+    <Compile Include="TransitiveClosureAggregate.cs" />
+    <Compile Include="Properties\AssemblyInfo.cs" />
+  </ItemGroup>
+  <ItemGroup>
+    <Content Include="TransitiveClosure.sql">
+      <AutoGen>True</AutoGen>
+      <DesignTime>True</DesignTime>
+      <DependentUpon>TransitiveClosure.tt</DependentUpon>
+    </Content>
+    <Content Include="TransitiveClosure.tt">
+      <Generator>TextTemplatingFileGenerator</Generator>
+      <LastGenOutput>TransitiveClosure.sql</LastGenOutput>
+    </Content>
+  </ItemGroup>
+  <ItemGroup>
+    <Service Include="{508349B6-6B84-4DF5-91F0-309BEEBAD82D}" />
+  </ItemGroup>
+  <ItemGroup>
+    <None Include="tcc.pfx" />
+  </ItemGroup>
+  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
+</Project>