From faca079f2cb5c3f398b4bfee286b83d2486d22b3 Mon Sep 17 00:00:00 2001 From: Jos de Bruijn Date: Wed, 1 Jun 2016 16:58:19 -0700 Subject: [PATCH 1/5] adding sprocs to docs --- .../documentation/wwi-olap-catalog.md | 35 +++++++++ .../documentation/wwi-oltp-htap-catalog.md | 72 +++++++++++++++++++ .../wwi-oltp-htap-installation.md | 12 +++- 3 files changed, 118 insertions(+), 1 deletion(-) diff --git a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md index 09e81ffd..c0981906 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md @@ -45,3 +45,38 @@ WideWorldImportersDW has the following dimension tables. The description include |Transaction|`Sales.CustomerTransactions` and `Purchasing.SupplierTransactions`|Measuring issue dates vs finalization dates, and amounts.| |Movement|`Warehouse.StockTransactions`|Movements over time.| |Stock Holding|`Warehouse.StockItemHoldings`|On-hand stock levels and value| + +## Stored procedures + +The stored procedures are used primarily for the ETL process and for configuration purposes. + +Any extensions of the sample are encouraged to use the `Reports` schema for Reporting Services reports, and the `PowerBI` schema for Power-BI access. + +### Application Schema + +These procedures are used to configure the sample. They are used to apply enterprise edition features to the standard edition version of the sample, add PolyBase, and reseed ETL. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|Configuration_ApplyPartitionedColumnstoreIndexing|Applies both partitioning and columnstore indexes for fact tables.| +|Configuration_ConfigureForEnterpriseEdition|Applies partitioning, columnstore indexing and in-memory.| +|Configuration_EnableInMemory|Replaces the integration staging tables with SCHEMA_ONLY memory-optimized tables to improve ETL performance.| +|Configuration_ApplyPolybase|Configures an external data source, file format, and table.| +|Configuration_PopulateLargeSaleTable|Applied enterprise edition changes, then populates a larger amount of data for the 2012 calendar year as additional history.| +|Configuration_ReseedETL|Removes existing data and restarts the ETL seeds. This allows for repopulating the OLAP database to match updated rows in the OLTP database.| + +### Integration Schema + +Procedures used in the ETL process fall in these categories: +- Helper procedures for the ETL package - All Get* procedures. +- Procedures used by the ETL package for migrating staged data into the DW tables - All Migrate* procedures. +- `PopulateDateDimensionForYear` - Takes a year and ensures that all dates for that year are populated in the `Dimension.Date` table. + +### Sequences Schema + +Procedures to configure the sequences in the database. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|ReseedAllSequences|Calls the procedure `ReseedSequenceBeyondTableValue` for all sequences.| +|ReseedSequenceBeyondTableValue|Used to reposition the next sequence value beyond the value in any table that uses the same sequence. (Like a `DBCC CHECKIDENT` for identity columns equivalent for sequences but across potentially multiple tables).| diff --git a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md index 75496ec4..a0f7215a 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md @@ -102,6 +102,78 @@ Details of stock items, their holdings and transactions. |VehicleTemperatures|Regularly recorded temperatures of vehicle chillers| |ColdRoomTemperatures|Regularly recorded temperatures of cold room chillers| +## Stored Procedures + +Stored procedures are organized in schemas. Most of the schemas are used for configuration or sample purposes. + +The `Website` schema contains the stored procedures that can be used by a Web front-end. + +The `Reports` and `PowerBI` schemas are meant for reporting services and PowerBI purposes. Any extensions of the sample are encouraged to use these schemas for reporting purposes. + +### Website schema + +These are the procedures used by a client application, such as a Web front-end. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|ActivateWebsiteLogon|Allows a person (from `Application.People`) to have access to the website.| +|ChangePassword|Changes a user’s password (for users that are not using external authentication mechanisms).| +|InsertCustomerOrders|Allows inserting one or more customer orders (including the order lines).| +|InvoiceCustomerOrders|Takes a list of orders to be invoiced and processes the invoices.| +|RecordColdRoomTemperatures|Takes a sensor data list, as a table-valued parameter (TVP), and applies the data to the `Warehouse.ColdRoomTemperatures` temporal table.| +|RecordVehicleTemperature|Takes a JSON array and uses it to update `Warehouse.VehicleTemperatures`.| +|SearchForCustomers|Searches for customers by name or part of name (either the company name or the person name).| +|SearchForPeople|Searches for people by name or part of name.| +|SearchForStockItems|Searches for stock items by name or part of name or marketing comments.| +|SearchForStockItemsByTags|Searches for stock items by tags.| +|SearchForSuppliers|Searches for suppliers by name or part of name (either the company name or the person name).| + +### Integration Schema + +The stored procedures in this schema are used by the ETL process. They obtain the data needed from various tables for the timeframe required by the [ETL package](wwi-etl.md). + +### DataLoadSimulation Schema + +Simulates a workload that inserts sales and purchases. The main stored procedure is `PopulateDataToCurrentDate`, which is used to insert sample data up to the current date. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|Configuration_ApplyDataLoadSimulationProcedures|Recreates the procedures needed for data load simulation. This is needed for bringing data up to the current date.| +|Configuration_RemoveDataLoadSimulationProcedures|This removes the procedures again after data simulation is complete.| +|DeactiveTemporalTablesBeforeDataLoad|Removes the temporal nature of all temporal tables and where applicable, applies a trigger so that changes can be made as though they were being applied at an earlier date than the sys-temporal tables allow.| +|PopulateDataToCurrentDate|Used to bring the data up to the current date. Should be run before any other configuration options after restoring the database from an initial backup.| +|ReactivateTemporalTablesAfterDataLoad|Re-establishes the temporal tables, including checking for data consistency. (Removes the associated triggers).| + + +### Application Schema + +These procedures are used to configure the sample. They are used to apply enterprise edition features to the standard edition version of the sample, and also to add auditing and full-text indexing. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|AddRoleMemberIfNonexistant|Adds a member to a role if the member isn’t already in the role| +|Configuration_ApplyAuditing|Adds auditing. Server auditing is applied for standard edition databases; additional database auditing is added for enterprise edition.| +|Configuration_ApplyColumnstoreIndexing|Applies columnstore indexing to `Sales.OrderLines` and `Sales.InvoiceLines` and reindexes appropriately.| +|Configuration_ApplyFullTextIndexing|Applies fulltext indexes to `Application.People`, `Sales.Customers`, `Purchasing.Suppliers`, and `Warehouse.StockItems`. Replaces `Website.SearchForPeople`, `Website.SearchForSuppliers`, `Website.SearchForCustomers`, `Website.SearchForStockItems`, `Website.SearchForStockItemsByTags` with replacement procedures that use fulltext indexing.| +|Configuration_ApplyPartitioning|Applies table partitioning to `Sales.CustomerTransactions and `Purchasing.SupplierTransactions`, and rearranges the indexes to suit.| +|Configuration_ApplyPolybase|Configures an external data source, file format, and table.| +|Configuration_ApplyRowLevelSecurity|Applies row level security to filter customers by sales territory related roles.| +|Configuration_ConfigureForEnterpriseEdition|Applies columnstore indexing, full text, in-memory, polybase, and partitioning.| +|Configuration_EnableInMemory|Adds a memory-optimized filegroup (when not working in Azure), replaces `Warehouse.ColdRoomTemperatures`, `Warehouse.VehicleTemperatures` with in-memory equivalents, and migrates the data, recreates the `Website.OrderIDList`, `Website.OrderList`, `Website.OrderLineList`, `Website.SensorDataList` table types with memory-optimized equivalents, drops and recreates the procedures `Website.InvoiceCustomerOrders`, `Website.InsertCustomerOrders`, and `Website.RecordColdRoomTemperatures` that uses these table types.| +|Configuration_RemoveAuditing|Removes the auditing configuration.| +|Configuration_RemoveRowLevelSecurity|Removes the row level security configuration (this is needed for changes to the associated tables).| +|CreateRoleIfNonExistant|Creates a database role if it doesn’t already exist.| + + +### Sequences Schema + +Procedures to configure the sequences in the database. + +|Procedure|Purpose| +|-----------------------------|---------------------| +|ReseedAllSequences|Calls the procedure ReseedSequenceBeyondTableValue for all sequences.| +|ReseedSequenceBeyondTableValue|Used to reposition the next sequence value beyond the value in any table that uses the same sequence. (Like a DBCC CHECKIDENT for identity columns equivalent for sequences but across potentially multiple tables).| + ## Design considerations diff --git a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-installation.md b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-installation.md index 35031721..0b4735c9 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-installation.md +++ b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-installation.md @@ -43,9 +43,19 @@ To import a bacpac into a new SQL Database, you can use Management Studio. ## Configuration -The sample database can make use of Full-Text Indexing. However, that feature is not installed by default with SQL Server - you need to select it during SQL Server setup. Therefore, a post-installation step is required. +### Full-Text Indexing + +The sample database can make use of Full-Text Indexing. However, that feature is not installed by default with SQL Server - you need to select it during SQL Server setup (it is enabled by default in Azure SQL DB). Therefore, a post-installation step is required. 1. In SQL Server Management Studio, connect to the WideWorldImporters database and open a new query window. 2. Run the following T-SQL command to enable the use of Full-Text Indexing in the database: EXECUTE Application.Configuration_ApplyFullTextIndexing + +### SQL Server Audit + +Enabling auditing in SQL Server requires server configuration. To enable SQL Server audit for the WideWorldImporters sample, run the following statement in the database: + + EXECUTE [Application].[Configuration_ApplyAuditing] + +In Azure SQL Database, Audit is configured through the [Azure portal](https://portal.azure.com/). From 039ebaab47cbf78c7e5fdb62d7abe0bc884ed975 Mon Sep 17 00:00:00 2001 From: Jos de Bruijn Date: Wed, 1 Jun 2016 17:00:30 -0700 Subject: [PATCH 2/5] moving section --- .../documentation/wwi-oltp-htap-catalog.md | 57 +++++++++---------- 1 file changed, 28 insertions(+), 29 deletions(-) diff --git a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md index a0f7215a..9daf521b 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md @@ -102,6 +102,34 @@ Details of stock items, their holdings and transactions. |VehicleTemperatures|Regularly recorded temperatures of vehicle chillers| |ColdRoomTemperatures|Regularly recorded temperatures of cold room chillers| + +## Design considerations + +Database design is subjective and there is no right or wrong way to design a database. The schemas and tables in this database show ideas for how you can design your own database. + +### Schema design + +WideWorldImporters uses a small number of schemas so that it is easy to understand the database system and demonstrate database principles. + +Wherever possible, the database collocates tables that are commonly queried together into the same schema to minimize join complexity. + +The database schema has been code-generated based on a series of metadata tables in another database WWI_Preparation. This gives WideWorldImporters a very high degree of design consistency, naming consistency, and completeness. For details on how the schema has been generated see the source code: [wide-world-importers/wwi-database-scripts](https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers/wwi-database-scripts) + +### Table design + +- All tables have single column primary keys for join simplicity. +- All schemas, tables, columns, indexes, and check constraints have a Description extended property that can be used to identify the purpose of the object or column. Memory-optimized tables are an exception to this since they don’t currently support extended properties. +- All foreign keys are automatically indexed unless there is another non-clustered index that has the same left-hand component. +- Auto-numbering in tables is based on sequences. These sequences are easier to work with across linked servers and similar environments than IDENTITY columns. Memory-optimized tables use IDENTITY columns since they don’t support in SQL Server 2016. +- A single sequence (TransactionID) is used for these tables: CustomerTransactions, SupplierTransactions, and StockItemTransactions. This demonstrates how a set of tables can have a single sequence. +- Some columns have appropriate default values. + +### Security schemas + +For security, WideWorldImporters does not allow external applications to access data schemas directly. To isolate access, WideWorldImporters uses security-access schemas that do not hold data, but contain views and stored procedures. External applications use the security schemas to retrieve the data that they are allowed to view. This way, users can only run the views and stored procedures in the secure-access schemas + +For example, this sample includes Power BI dashboards. An external application accesses these Power BI dashboards from the Power BI gateway as a user that has read-only permission on the PowerBI schema. For read-only permission, the user only needs SELECT and EXECUTE permission on the PowerBI schema. A database administrator at WWI assigns these permissions as needed. + ## Stored Procedures Stored procedures are organized in schemas. Most of the schemas are used for configuration or sample purposes. @@ -156,7 +184,6 @@ These procedures are used to configure the sample. They are used to apply enterp |Configuration_ApplyColumnstoreIndexing|Applies columnstore indexing to `Sales.OrderLines` and `Sales.InvoiceLines` and reindexes appropriately.| |Configuration_ApplyFullTextIndexing|Applies fulltext indexes to `Application.People`, `Sales.Customers`, `Purchasing.Suppliers`, and `Warehouse.StockItems`. Replaces `Website.SearchForPeople`, `Website.SearchForSuppliers`, `Website.SearchForCustomers`, `Website.SearchForStockItems`, `Website.SearchForStockItemsByTags` with replacement procedures that use fulltext indexing.| |Configuration_ApplyPartitioning|Applies table partitioning to `Sales.CustomerTransactions and `Purchasing.SupplierTransactions`, and rearranges the indexes to suit.| -|Configuration_ApplyPolybase|Configures an external data source, file format, and table.| |Configuration_ApplyRowLevelSecurity|Applies row level security to filter customers by sales territory related roles.| |Configuration_ConfigureForEnterpriseEdition|Applies columnstore indexing, full text, in-memory, polybase, and partitioning.| |Configuration_EnableInMemory|Adds a memory-optimized filegroup (when not working in Azure), replaces `Warehouse.ColdRoomTemperatures`, `Warehouse.VehicleTemperatures` with in-memory equivalents, and migrates the data, recreates the `Website.OrderIDList`, `Website.OrderList`, `Website.OrderLineList`, `Website.SensorDataList` table types with memory-optimized equivalents, drops and recreates the procedures `Website.InvoiceCustomerOrders`, `Website.InsertCustomerOrders`, and `Website.RecordColdRoomTemperatures` that uses these table types.| @@ -173,31 +200,3 @@ Procedures to configure the sequences in the database. |-----------------------------|---------------------| |ReseedAllSequences|Calls the procedure ReseedSequenceBeyondTableValue for all sequences.| |ReseedSequenceBeyondTableValue|Used to reposition the next sequence value beyond the value in any table that uses the same sequence. (Like a DBCC CHECKIDENT for identity columns equivalent for sequences but across potentially multiple tables).| - - -## Design considerations - -Database design is subjective and there is no right or wrong way to design a database. The schemas and tables in this database show ideas for how you can design your own database. - -### Schema design - -WideWorldImporters uses a small number of schemas so that it is easy to understand the database system and demonstrate database principles. - -Wherever possible, the database collocates tables that are commonly queried together into the same schema to minimize join complexity. - -The database schema has been code-generated based on a series of metadata tables in another database WWI_Preparation. This gives WideWorldImporters a very high degree of design consistency, naming consistency, and completeness. For details on how the schema has been generated see the source code: [wide-world-importers/wwi-database-scripts](https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers/wwi-database-scripts) - -### Table design - -- All tables have single column primary keys for join simplicity. -- All schemas, tables, columns, indexes, and check constraints have a Description extended property that can be used to identify the purpose of the object or column. Memory-optimized tables are an exception to this since they don’t currently support extended properties. -- All foreign keys are automatically indexed unless there is another non-clustered index that has the same left-hand component. -- Auto-numbering in tables is based on sequences. These sequences are easier to work with across linked servers and similar environments than IDENTITY columns. Memory-optimized tables use IDENTITY columns since they don’t support in SQL Server 2016. -- A single sequence (TransactionID) is used for these tables: CustomerTransactions, SupplierTransactions, and StockItemTransactions. This demonstrates how a set of tables can have a single sequence. -- Some columns have appropriate default values. - -### Security schemas - -For security, WideWorldImporters does not allow external applications to access data schemas directly. To isolate access, WideWorldImporters uses security-access schemas that do not hold data, but contain views and stored procedures. External applications use the security schemas to retrieve the data that they are allowed to view. This way, users can only run the views and stored procedures in the secure-access schemas - -For example, this sample includes Power BI dashboards. An external application accesses these Power BI dashboards from the Power BI gateway as a user that has read-only permission on the PowerBI schema. For read-only permission, the user only needs SELECT and EXECUTE permission on the PowerBI schema. A database administrator at WWI assigns these permissions as needed. From 4fa6501c896747dfda399dc2d87aad4ccc781b54 Mon Sep 17 00:00:00 2001 From: Jos de Bruijn Date: Wed, 1 Jun 2016 17:02:28 -0700 Subject: [PATCH 3/5] fix --- .../wide-world-importers/documentation/wwi-olap-catalog.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md index c0981906..a1fd0fd2 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md @@ -35,7 +35,7 @@ WideWorldImportersDW has the following dimension tables. The description include ### Fact tables -WideWorldImportersDW has the following dimension tables. The description includes the relationship with the source tables in the WideWorldImporters database, as well as the classes of analytics/reporting queries each fact table is typically used with. +WideWorldImportersDW has the following fact tables. The description includes the relationship with the source tables in the WideWorldImporters database, as well as the classes of analytics/reporting queries each fact table is typically used with. |Table|Source tables|Sample Analytics| |-----------------------------|---------------------| @@ -44,7 +44,8 @@ WideWorldImportersDW has the following dimension tables. The description include |Purchase|`Purchasing.PurchaseOrderLines`|Expected vs actual lead times| |Transaction|`Sales.CustomerTransactions` and `Purchasing.SupplierTransactions`|Measuring issue dates vs finalization dates, and amounts.| |Movement|`Warehouse.StockTransactions`|Movements over time.| -|Stock Holding|`Warehouse.StockItemHoldings`|On-hand stock levels and value| +|Stock Holding|`Warehouse.StockItemHoldings`|On-hand stock levels and value.| + ## Stored procedures From cdcc6e7620e617de1c20e8d1b89125e6a148071f Mon Sep 17 00:00:00 2001 From: Jos de Bruijn Date: Wed, 1 Jun 2016 17:03:55 -0700 Subject: [PATCH 4/5] z --- .../wide-world-importers/documentation/wwi-olap-catalog.md | 1 - 1 file changed, 1 deletion(-) diff --git a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md index a1fd0fd2..c2b4aef3 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md @@ -46,7 +46,6 @@ WideWorldImportersDW has the following fact tables. The description includes the |Movement|`Warehouse.StockTransactions`|Movements over time.| |Stock Holding|`Warehouse.StockItemHoldings`|On-hand stock levels and value.| - ## Stored procedures The stored procedures are used primarily for the ETL process and for configuration purposes. From 73dec5322657e874f73ca94f8505845afd2c92dd Mon Sep 17 00:00:00 2001 From: Jos de Bruijn Date: Wed, 1 Jun 2016 17:04:57 -0700 Subject: [PATCH 5/5] fix in table --- .../wide-world-importers/documentation/wwi-olap-catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md index c2b4aef3..88e5c093 100644 --- a/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md +++ b/samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md @@ -38,7 +38,7 @@ WideWorldImportersDW has the following dimension tables. The description include WideWorldImportersDW has the following fact tables. The description includes the relationship with the source tables in the WideWorldImporters database, as well as the classes of analytics/reporting queries each fact table is typically used with. |Table|Source tables|Sample Analytics| -|-----------------------------|---------------------| +|-----------------------------|---------------------|---------------------| |Order|`Sales.Orders` and `Sales.OrderLines`|Sales people, picker/packer productivity, and on time to pick orders. In addition, low stock situations leading to back orders.| |Sale|`Sales.Invoices` and `Sales.InvoiceLines`|Sales dates, delivery dates, profitability over time, profitability by sales person.| |Purchase|`Purchasing.PurchaseOrderLines`|Expected vs actual lead times|