mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Resolved merge conflict by keeping files from upstream
This commit is contained in:
msalexprotsenko
@@ -9,7 +9,7 @@ Installation instructions for SQL Server 2019 big data clusters can be found [he
|
||||
## Executing the sample scripts
|
||||
The scripts should be executed in a specific order to test the various features. Execute the scripts from each folder in below order:
|
||||
|
||||
1. __[spark/dataloading/transform-csv-files.ipynb](spark/dataloading/transform-csv-files.ipynb)__
|
||||
1. __[spark/data-loading/transform-csv-files.ipynb](spark/data-loading/transform-csv-files.ipynb)__
|
||||
1. __[data-virtualization/generic-odbc](data-virtualization/generic-odbc)__
|
||||
1. __[data-virtualization/hadoop](data-virtualization/hadoop)__
|
||||
1. __[data-virtualization/storage-pool](data-virtualization/storage-pool)__
|
||||
|
||||
@@ -7,10 +7,10 @@ Application deployment allows you to deploy applications into SQL Server big dat
|
||||
|
||||
## Pre-requisites
|
||||
* SQL Server big data cluster CTP 2.3 or later
|
||||
* `mssqlctl` CLI familiarity. If you are unfamiliar with `mssqlctl` please refer to - [App Deployment in SQL Server big data cluster](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) for more information.
|
||||
* `azdata` CLI familiarity. If you are unfamiliar with `azdata` please refer to - [App Deployment in SQL Server big data cluster](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) for more information.
|
||||
|
||||
* Tip
|
||||
**mssqlctl app -h** will display the various commands to manage the app
|
||||
**azdata app -h** will display the various commands to manage the app
|
||||
|
||||
## Templates
|
||||
Templates are used by our [App Deploy add-ins](https://docs.microsoft.com/en-us/sql/big-data-cluster/app-deployment-extension?view=sqlallproducts-allversions) and can be used to quickly deploy applications.
|
||||
|
||||
@@ -34,30 +34,30 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `roll-dice.R` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./RollDice
|
||||
azdata app create --spec ./RollDice
|
||||
```
|
||||
4. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n roll-dice -v [version]
|
||||
azdata app list -n roll-dice -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
5. Test the app by running the following command:
|
||||
```bash
|
||||
mssqlctl app run -n roll-dice -v [version] --input x=3
|
||||
azdata app run -n roll-dice -v [version] --input x=3
|
||||
```
|
||||
You should get output like the example for three dice below. The results of the dice rolled are in the `result` data frame:
|
||||
```json
|
||||
@@ -88,7 +88,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name roll-dice --version [version]
|
||||
azdata app delete --name roll-dice --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -23,7 +23,7 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server big data cluster.
|
||||
3. Optional: to see the SSIS package itself, install Visual Studio 2017 if you don't have it already. After that download and install [SSDT](https://docs.microsoft.com/en-us/sql/ssdt/download-sql-server-data-tools-ssdt?view=sql-server-2017#ssdt-for-vs-2017-standalone-installer).
|
||||
4. Optional: install [SSMS](https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-2017) if it is not already installed.
|
||||
|
||||
@@ -32,19 +32,19 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. Replace `[SA_PASSWORD]` in the `spec.yaml` file with the password for SQL user `sa`.
|
||||
4. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `back-up-db.dtsx` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./SSIS
|
||||
azdata app create --spec ./SSIS
|
||||
```
|
||||
5. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list --name back-up-db --version [version]
|
||||
azdata app list --name back-up-db --version [version]
|
||||
```
|
||||
Once the app is listed as `Ready` the job should run within a minute.
|
||||
You can check if the backup is created by running:
|
||||
@@ -56,7 +56,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name back-up-db --version [version]
|
||||
azdata app delete --name back-up-db --version [version]
|
||||
# delete backup files
|
||||
kubectl -n [your namespace] exec -it mssql-master-pool-0 -c mssql-server -- /bin/bash -c "rm /var/opt/mssql/data/*.DWConfigbak"
|
||||
```
|
||||
@@ -73,7 +73,7 @@ Here is the spec file for this application. This sample uses the `SSIS` runtime
|
||||
|Setting|Description|
|
||||
|-|-|
|
||||
|options|Specifies any command line parameters passed to the execution of the SSIS package|
|
||||
|schedule|Specifies when the job should run. This follows cron expressions. A value of '*/1 * * * *' means the job runs *every minute*. If omitted the package will not run automatically and you can run the package on demand using `mssqlctl run -n back-up-db -v [version]` or making a call to the API.|
|
||||
|schedule|Specifies when the job should run. This follows cron expressions. A value of '*/1 * * * *' means the job runs *every minute*. If omitted the package will not run automatically and you can run the package on demand using `azdata run -n back-up-db -v [version]` or making a call to the API.|
|
||||
|
||||
```yaml
|
||||
name: back-up-db
|
||||
|
||||
@@ -35,30 +35,30 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `add.py` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./addpy
|
||||
azdata app create --spec ./addpy
|
||||
```
|
||||
4. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n addpy -v [version]
|
||||
azdata app list -n addpy -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
5. Test the app by running the following command:
|
||||
```bash
|
||||
mssqlctl app run -n addpy -v [version] --input x=3,y=5
|
||||
azdata app run -n addpy -v [version] --input x=3,y=5
|
||||
```
|
||||
You should get output like the example below. The result of adding 3+5 are returned as `result`.
|
||||
```json
|
||||
@@ -75,7 +75,7 @@ To run this sample, you need the following prerequisites.
|
||||
```
|
||||
6. <a name=restapi></a>Any app you create is also accessible using a RESTful web service that is [Swagger](swagger.io) compliant. You can get the endpoint for the web service by running:
|
||||
```bash
|
||||
mssqlctl app describe --name addpy --version [version]
|
||||
azdata app describe --name addpy --version [version]
|
||||
```
|
||||
This will return an output much like the following:
|
||||
```json
|
||||
@@ -106,7 +106,7 @@ To run this sample, you need the following prerequisites.
|
||||
}
|
||||
```
|
||||
Note the IP address and the port number in this output. Open the following URL in your browser:
|
||||
`https://[IP]:[PORT]/api/docs/swagger.json`. You will have to log in with the same credentials you used for `mssqlctl login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io) to understand what methods are available:
|
||||
`https://[IP]:[PORT]/api/docs/swagger.json`. You will have to log in with the same credentials you used for `azdata login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io) to understand what methods are available:
|
||||
|
||||
|
||||
Notice the `app` GET method as well as the `token` POST method. Since the authentication for apps uses JWT tokens you will need to get a token my using your favorite tool to make a POST call to the `token` method. Here is an example of how to do just that in [Postman](https://www.getpostman.com/):
|
||||
@@ -114,20 +114,20 @@ To run this sample, you need the following prerequisites.
|
||||
|
||||
The result of this request will give you an `access_token`, which you will need to call the URL to run the app.
|
||||
|
||||
> *Optional*: If you want, you can open the URL for the `swagger` that was returned when you ran `mssqlctl app describe --name addpy --version [version]` in your browser. You will have to log in with the same credentials you used for `mssqlctl login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io). You will see that the web service exposes the `run` method.
|
||||
> *Optional*: If you want, you can open the URL for the `swagger` that was returned when you ran `azdata app describe --name addpy --version [version]` in your browser. You will have to log in with the same credentials you used for `azdata login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io). You will see that the web service exposes the `run` method.
|
||||
|
||||
You can use your favorite tool to call the `run` method (`https://[IP]:30778/api/app/addpy/[version]/run`), passing in the parameters in the body of your POST request as json. In this example we will use [Postman](https://www.getpostman.com/). Before making the call, you will need to set the `Authorization` to `Bearer Token` and paste in the token you retrieved earlier. This will set a header on your request. See the screenshot below.
|
||||
|
||||
Next, in the requests body, pass in the parameters to the app you are calling and set the `content-type` to `application/json`:
|
||||
|
||||
When you send the request, you will get the same output as you did when you ran the app through `mssqlctl app run`:
|
||||
When you send the request, you will get the same output as you did when you ran the app through `azdata app run`:
|
||||
|
||||
You have now successfully called the app through the web service!
|
||||
|
||||
7. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name addpy --version [version]
|
||||
azdata app delete --name addpy --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -34,30 +34,30 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `magic8ball.py` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./magic8ball
|
||||
azdata app create --spec ./magic8ball
|
||||
```
|
||||
4. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n magic8ball -v [version]
|
||||
azdata app list -n magic8ball -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
5. Test the app by running the following command:
|
||||
```bash
|
||||
mssqlctl app run -n magic8ball -v [version] --input txt="Will it rain tomorrow?"
|
||||
azdata app run -n magic8ball -v [version] --input txt="Will it rain tomorrow?"
|
||||
```
|
||||
You should get output like the example below. The answer to your question are returned as `result`.
|
||||
```json
|
||||
@@ -78,7 +78,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name magic8ball --version [version]
|
||||
azdata app delete --name magic8ball --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -31,27 +31,27 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. This example uses a TensorFlow Machine Learning Model that uses public US Census data predict income. [More details and information on the example are here](https://docs.microsoft.com/en-us/sql/big-data-cluster/train-and-create-machinelearning-models-with-spark?view=sqlallproducts-allversions). The application you will be deploying as part of this sample is a Random Forest Model that was built in Spark and has been [serialized as an MLeap bundle](https://docs.microsoft.com/en-us/sql/big-data-cluster/export-model-with-spark-mleap?view=sqlallproducts-allversions).
|
||||
|
||||
Deploy the app using the `create` command and pass the location of the spec file. In the example below, the spec file is expected to be in the `mleap` folder:
|
||||
```bash
|
||||
mssqlctl app create --spec ./mleap/
|
||||
azdata app create --spec ./mleap/
|
||||
```
|
||||
1. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n mleap-census -v [version]
|
||||
azdata app list -n mleap-census -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
2. Now that the app has been deployed you can test if the app works correctly by passing in a sample input that is available in the `mleap` folder. The deployed app is a RESTful webservice that is [Swagger](swagger.io) compliant. For this sample we will show you how you can test this using the CLI.
|
||||
@@ -59,7 +59,7 @@ To run this sample, you need the following prerequisites.
|
||||
To test the app, run the command below. The input parameter is a `MLeapFrame`, a `json` file that describes the parameters and the values provided to the model for predicting income. Note that the input parameter has a special character '@' to indicate that a `json` file is being passed. This command needs to be run within the `mleap` folder.
|
||||
|
||||
```bash
|
||||
mssqlctl app run --name mleap-census --version [version] --input schema=@census_frame.json
|
||||
azdata app run --name mleap-census --version [version] --input schema=@census_frame.json
|
||||
```
|
||||
|
||||
The result will be a json output that includes the predicted income along with additional data.
|
||||
@@ -69,7 +69,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name mleap-census --version [version]
|
||||
azdata app delete --name mleap-census --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -35,30 +35,30 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml`, `sentiment.rds` and `sentiment.R` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./sentiment-analysis
|
||||
azdata app create --spec ./sentiment-analysis
|
||||
```
|
||||
4. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n sentiment-r -v [version]
|
||||
azdata app list -n sentiment-r -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
5. Test the app by running the following command:
|
||||
```bash
|
||||
mssqlctl app run -n sentiment-r -v [version] --input reviewText="Absolutely the best movie experience I have ever had!"
|
||||
azdata app run -n sentiment-r -v [version] --input reviewText="Absolutely the best movie experience I have ever had!"
|
||||
```
|
||||
You should get output like the example below. The result of the sentiment analysis scoring is returned as a data frame in `out`. A `PredictedLabel` equal to `1` indicates the sentiment is deemed positive, whereas a `PredictedLabel` of `0` indicates a negative sentiment. The `Probability.1` indicates the level of certainty for the `PredictedLabel` to be the true sentiment.
|
||||
```json
|
||||
@@ -89,7 +89,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name sentiment-r --version [version]
|
||||
azdata app delete --name sentiment-r --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -35,30 +35,30 @@ To run this sample, you need the following prerequisites.
|
||||
**Software prerequisites:**
|
||||
|
||||
1. SQL Server big data cluster CTP 2.3 or later.
|
||||
2. `mssqlctl`. Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the `mssqlctl` and connecting to a SQL Server 2019 big data cluster.
|
||||
2. `azdata`. Refer to [installing azdata](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) document on setting up the `azdata` and connecting to a SQL Server 2019 big data cluster.
|
||||
|
||||
<a name=run-this-sample></a>
|
||||
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `controller-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
azdata login -e https://<ip-address-of-controller-svc-external>:30080 -u <user-name>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `sum_of_squares.R` files are located:
|
||||
```bash
|
||||
mssqlctl app create --spec ./sumofsq
|
||||
azdata app create --spec ./sumofsq
|
||||
```
|
||||
4. Check the deployment by running the following command:
|
||||
```bash
|
||||
mssqlctl app list -n sum-of-sq -v [version]
|
||||
azdata app list -n sum-of-sq -v [version]
|
||||
```
|
||||
Once the app is listed as `Ready` you can continue to the next step.
|
||||
5. Test the app by running the following command:
|
||||
```bash
|
||||
mssqlctl app run -n sum-of-sq -v [version] --input a=42,b=3435
|
||||
azdata app run -n sum-of-sq -v [version] --input a=42,b=3435
|
||||
```
|
||||
You should get output like the example below. The answer to your question are returned as `result`.
|
||||
```json
|
||||
@@ -79,7 +79,7 @@ To run this sample, you need the following prerequisites.
|
||||
6. You can clean up the sample by running the following commands:
|
||||
```bash
|
||||
# delete app
|
||||
mssqlctl app delete --name sum-of-sq --version [version]
|
||||
azdata app delete --name sum-of-sq --version [version]
|
||||
```
|
||||
|
||||
<a name=sample-details></a>
|
||||
|
||||
@@ -12,10 +12,10 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node
|
||||
|
||||
1. Connect to SQL Server Master instance.
|
||||
|
||||
1. Run the [../../spark/dataloading/transform-csv-files.ipynb](../../spark/dataloading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
|
||||
1. Run the [../../spark/data-loading/transform-csv-files.ipynb](../../spark/data-loading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
|
||||
|
||||
1. Execute the [web-clickstreams-hdfs-orc.sql](web-clickstreams-hdfs-orc.sql). This script demonstrates how to read ORC file(s) stored in HDFS.
|
||||
|
||||
1. Execute the [product-reviews-hdfs-orc.sql](product-reviews-hdfs-orc.sql). This script demonstrates how to read ORC file(s) stored in HDFS.
|
||||
|
||||
1. Execute the [inventory-hdfs-rcfile.sql](inventory-hdfs-rcfile.sql). This script demonstrates how to export data from SQL Server into HDFS using PolyBase v1 syntax. This script will export data from SQL Server into RCFILE format.
|
||||
1. Execute the [inventory-hdfs-rcfile.sql](inventory-hdfs-rcfile.sql). This script demonstrates how to export data from SQL Server into HDFS using PolyBase v1 syntax. This script will export data from SQL Server into RCFILE format.
|
||||
|
||||
@@ -8,7 +8,7 @@ SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server,
|
||||
|
||||
In this example, you are going to create an external table in a SQL Server instance over the inventory table that sits on an Oracle server. If you are using a SQL Server 2019 big data cluster then the scripts can be executed on the SQL Server Master instance.
|
||||
|
||||
**Before you begin**, you need to have an Oracle instance and credentials. Follow the instruction in the [setup\README.md](setup\README.md).
|
||||
**Before you begin**, you need to have an Oracle instance and credentials. Follow the instruction in the [setup\README.md](setup/README.md).
|
||||
|
||||
### Instructions
|
||||
|
||||
|
||||
+1
-1
@@ -49,7 +49,7 @@ GO
|
||||
SELECT pr.pr_item_sk, pc.pr_review_content, pr.pr_user_sk AS customerid
|
||||
FROM dbo.product_reviews as pr
|
||||
JOIN (SELECT TOP(100) * FROM dbo.product_reviews_hdfs_csv) AS pc ON pc.pr_review_sk = pr.pr_review_sk
|
||||
JOIN dbo.customer_ora AS c ON c.c_customer_sk = pr.pr_user_sk
|
||||
JOIN dbo.customer_ora AS c ON c.C_CUSTOMER_SK = pr.pr_user_sk
|
||||
JOIN dbo.item AS i ON i.i_item_sk = pr.pr_item_sk
|
||||
INNER JOIN (
|
||||
SELECT
|
||||
|
||||
+7
-7
@@ -20,22 +20,22 @@ IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'OracleSalesS
|
||||
-- As a result, the names are case-sensitive so specify the name in the external table definition
|
||||
-- that matches the exact case of the table and column names in the Oracle metadata.
|
||||
CREATE EXTERNAL TABLE [inventory_ora]
|
||||
([inv_date] DECIMAL(10,0) NOT NULL, [inv_item] DECIMAL(10,0) NOT NULL,
|
||||
[inv_warehouse] DECIMAL(10,0) NOT NULL, [inv_quantity_on_hand] DECIMAL(10,0))
|
||||
([INV_DATE] DECIMAL(10,0) NOT NULL, [INV_ITEM] DECIMAL(10,0) NOT NULL,
|
||||
[INV_WAREHOUSE] DECIMAL(10,0) NOT NULL, [INV_QUANTITY_ON_HAND] DECIMAL(10,0))
|
||||
WITH (DATA_SOURCE=[OracleSalesSrvr],
|
||||
LOCATION='<oracle_service_name,nvarchar(30),xe>.SALES.INVENTORY');
|
||||
GO
|
||||
|
||||
-- Find quantity of certain items from inventory for a specific category
|
||||
--
|
||||
SELECT TOP(100) w.w_warehouse_name, i.inv_item, SUM(i.inv_quantity_on_hand) as total_quantity
|
||||
SELECT TOP(100) w.w_warehouse_name, i.INV_ITEM, SUM(i.INV_QUANTITY_ON_HAND) as total_quantity
|
||||
FROM [inventory_ora] as i
|
||||
JOIN item as it
|
||||
ON it.i_item_sk = i.inv_item
|
||||
ON it.i_item_sk = i.INV_ITEM
|
||||
JOIN warehouse as w
|
||||
ON w.w_warehouse_sk = i.inv_warehouse
|
||||
WHERE it.i_category = 'Movies & TV' and i.inv_item BETWEEN 17401 and 17402 --> get items within specific range
|
||||
GROUP BY w.w_warehouse_name, i.inv_item;
|
||||
ON w.w_warehouse_sk = i.INV_WAREHOUSE
|
||||
WHERE it.i_category = 'Movies & TV' and i.INV_ITEM BETWEEN 17401 and 17402 --> get items within specific range
|
||||
GROUP BY w.w_warehouse_name, i.INV_ITEM;
|
||||
GO
|
||||
|
||||
-- Cleanup
|
||||
|
||||
+2
-2
@@ -9,14 +9,14 @@ if NOT DEFINED ORACLE_SERVER goto :usage
|
||||
if NOT DEFINED ORACLE_USER goto :usage
|
||||
if NOT DEFINED ORACLE_PASSWORD goto :usage
|
||||
|
||||
for %F in (sqlplus.exe sqlldr.exe) do (
|
||||
for %%F in (sqlplus.exe sqlldr.exe) do (
|
||||
echo Verifying %%F is in path & CALL WHERE /Q %%F || GOTO exit
|
||||
)
|
||||
|
||||
|
||||
for %%F in (sales-user.sql inventory.sql customer.sql) do (
|
||||
echo Executing [%%F]...
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit
|
||||
echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @%%F || GOTO exit
|
||||
)
|
||||
|
||||
for %%F in (inventory.ctl customer.ctl) do (
|
||||
|
||||
@@ -12,7 +12,7 @@ In SQL Server 2019 big data cluster, the storage pool consists of HDFS data node
|
||||
|
||||
1. Connect to SQL Server Master instance.
|
||||
|
||||
1. Run the [../../spark/dataloading/transform-csv-files.ipynb](../../spark/dataloading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
|
||||
1. Run the [../../spark/data-loading/transform-csv-files.ipynb](../../spark/data-loading/transform-csv-files.ipynb/) notebook to generate the sample parquet file(s).
|
||||
|
||||
1. Execute the [web-clickstreams-hdfs-csv.sql](web-clickstreams-hdfs-csv.sql). This script demonstrates how to read CSV file(s) stored in HDFS.
|
||||
|
||||
|
||||
+2
-2
@@ -90,8 +90,8 @@ AS
|
||||
AND w.wcs_user_sk IS NOT NULL)
|
||||
GROUP BY w.wcs_user_sk
|
||||
) AS q
|
||||
INNER JOIN customer as c ON q.wcs_user_sk = c.c_customer_sk
|
||||
INNER JOIN customer_demographics as cd ON c.c_current_cdemo_sk = cd.cd_demo_sk;
|
||||
INNER JOIN customer_ora as c ON q.wcs_user_sk = c.C_CUSTOMER_SK
|
||||
INNER JOIN customer_demographics as cd ON c.C_CURRENT_CDEMO_SK = cd.cd_demo_sk;
|
||||
GO
|
||||
|
||||
|
||||
|
||||
@@ -3,14 +3,14 @@
|
||||
|
||||
SQL Server 2019 big data cluster is deployed as docker containers on a Kubernetes cluster. These samples provide scripts that can be used to provision a Kubernetes clusters using different environments.
|
||||
|
||||
## Create a Kubernetes cluster using Kubeadm on Ubuntu 16.04 LTS or 18.04 LTS
|
||||
## __[Deploy a Kubernetes cluster using kubeadm](kubeadm/)__
|
||||
|
||||
Use the scripts in the **kubeadm** folder to deploy Kubernetes over multiple Linux machines (physical or virtualized) using `kubeadm` utility.
|
||||
Use the scripts in the **kubeadm** folder to deploy a Kubernetes cluster over one or more Linux machines (physical or virtualized) using `kubeadm` utility.
|
||||
|
||||
## Deploy a SQL Server big data cluster on Azure Kubernetes Service (AKS)
|
||||
## __[Deploy a SQL Server big data cluster on Azure Kubernetes Service (AKS)](aks/)__
|
||||
|
||||
Using the sample Python script in **aks** folder, you will deploy a Kubernetes cluster in Azure using AKS and a SQL Server big data cluster using on top of it.
|
||||
Using the sample Python script in **aks** folder, you will deploy a Kubernetes cluster in Azure using AKS and a SQL Server big data cluster using on top of it.
|
||||
|
||||
## Push SQL Server big data cluster images to your own private Docker repository
|
||||
## __[Push SQL Server big data cluster images to your own private Docker repository](offline/)__
|
||||
|
||||
Using the sample Python script in **offline** folder, you will push the necessary images required for the deployment to your own repository.
|
||||
@@ -16,9 +16,9 @@ Using this sample Python script, you will deploy a Kubernetes cluster in Azure u
|
||||
python -m pip install requests
|
||||
python -m pip install requests --upgrade
|
||||
```
|
||||
- Install mssqlctl CLI latest version using . Run the command below using elevated priviledges (sudo or admin cmd window):
|
||||
- Install latest version of the cluster management tool **azdata** (previously named mssqlctl) using below command. Run the command below using elevated priviledges (sudo or admin cmd window):
|
||||
```
|
||||
pip3 install -r https://private-repo.microsoft.com/python/ctp3.1/mssqlctl/requirements.txt
|
||||
pip3 install -r https://aka.ms/azdata
|
||||
```
|
||||
1. Login into your Azure account. Run this command:
|
||||
```
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#
|
||||
# Prerequisites:
|
||||
#
|
||||
# Azure CLI (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli), python3 (https://www.python.org/downloads), mssqlctl CLI (pip3 install -r https://private-repo.microsoft.com/python/ctp3.1/mssqlctl/requirements.txt)
|
||||
# Azure CLI (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli), python3 (https://www.python.org/downloads), azdata CLI (pip3 install -r https://aka.ms/azdata)
|
||||
#
|
||||
# Run `az login` at least once BEFORE running this script
|
||||
#
|
||||
@@ -24,8 +24,9 @@ def executeCmd (cmd):
|
||||
#
|
||||
SUBSCRIPTION_ID = input("Provide your Azure subscription ID:").strip()
|
||||
GROUP_NAME = input("Provide Azure resource group name to be created:").strip()
|
||||
DOCKER_USERNAME = input("Provide your Docker username:").strip()
|
||||
DOCKER_PASSWORD = getpass.getpass("Provide your Docker password:").strip()
|
||||
# Use this only if you are using a private registry different than default Micrososft registry (mcr).
|
||||
#DOCKER_USERNAME = input("Provide your Docker username:").strip()
|
||||
#DOCKER_PASSWORD = getpass.getpass("Provide your Docker password:").strip()
|
||||
|
||||
#
|
||||
# Optionally change these configuration settings
|
||||
@@ -38,21 +39,24 @@ AKS_NODE_COUNT=input("Provide number of worker nodes for AKS cluster - Press ENT
|
||||
CLUSTER_NAME=input("Provide name of AKS cluster and SQL big data cluster - Press ENTER for using `sqlbigdata`:").strip() or "sqlbigdata"
|
||||
|
||||
#This password will be use for Controller user, Knox user and SQL Server Master SA accounts
|
||||
#
|
||||
CONTROLLER_USERNAME=input("Provide username to be used for Controller user - Press ENTER for using `admin`:").strip() or "admin"
|
||||
PASSWORD = getpass.getpass("Provide password to be used for Controller user, Knox user and SQL Server Master SA accounts - Press ENTER for using `MySQLBigData2019`").strip() or "MySQLBigData2019"
|
||||
|
||||
#docker registry details
|
||||
DOCKER_REGISTRY="private-repo.microsoft.com"
|
||||
DOCKER_REPOSITORY="mssql-private-preview"
|
||||
DOCKER_IMAGE_TAG="ctp3.1"
|
||||
# Docker registry details
|
||||
# Use this only if you are using a private registry different than mcr. If so, make sure you are also setting the environment variables for DOCKER_USERNAME and DOCKER_PASSWORD
|
||||
# DOCKER_REGISTRY="<your private registry>"
|
||||
# DOCKER_REPOSITORY="<your private repository>"
|
||||
# DOCKER_IMAGE_TAG="<your Docker image tag>"
|
||||
|
||||
print ('Setting environment variables')
|
||||
os.environ['MSSQL_SA_PASSWORD'] = PASSWORD
|
||||
os.environ['CONTROLLER_USERNAME'] = CONTROLLER_USERNAME
|
||||
os.environ['CONTROLLER_PASSWORD'] = PASSWORD
|
||||
os.environ['KNOX_PASSWORD'] = PASSWORD
|
||||
os.environ['DOCKER_USERNAME']=DOCKER_USERNAME
|
||||
os.environ['DOCKER_PASSWORD']=DOCKER_PASSWORD
|
||||
# Use this only if you are using a private registry different than mcr. If so, you must set the environment variables for DOCKER_USERNAME and DOCKER_PASSWORD
|
||||
# os.environ['DOCKER_USERNAME']=DOCKER_USERNAME
|
||||
# os.environ['DOCKER_PASSWORD']=DOCKER_PASSWORD
|
||||
os.environ['ACCEPT_EULA']="Yes"
|
||||
|
||||
print ("Set azure context to subcription: "+SUBSCRIPTION_ID)
|
||||
@@ -71,29 +75,30 @@ command = "az aks get-credentials --overwrite-existing --name "+CLUSTER_NAME+" -
|
||||
executeCmd (command)
|
||||
|
||||
print("Creating SQL Big Data cluster:" +CLUSTER_NAME)
|
||||
command="mssqlctl bdc config init --source aks-dev-test --target custom --force"
|
||||
command="azdata bdc config init --source aks-dev-test --target custom --force"
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl bdc config section set -c custom -j ""metadata.name=" + CLUSTER_NAME + ""
|
||||
command="azdata bdc config replace -c custom/cluster.json -j ""metadata.name=" + CLUSTER_NAME + ""
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl bdc config section set -c custom -j ""$.spec.controlPlane.spec.docker.registry=" + DOCKER_REGISTRY + ""
|
||||
# Use this only if you are using a private registry different than default Micrososft registry (mcr).
|
||||
# command="azdata bdc config replace -c custom/control.json -j ""$.spec.controlPlane.spec.docker.registry=" + DOCKER_REGISTRY + ""
|
||||
# executeCmd (command)
|
||||
|
||||
# command="azdata bdc config replace -c custom/control.json -j ""$.spec.controlPlane.spec.docker.repository=" + DOCKER_REPOSITORY + ""
|
||||
# executeCmd (command)
|
||||
|
||||
# command="azdata bdc config replace -c custom/control.json -j ""$.spec.controlPlane.spec.docker.imageTag=" + DOCKER_IMAGE_TAG + ""
|
||||
# executeCmd (command)
|
||||
|
||||
command="azdata bdc create -c custom --accept-eula yes"
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl bdc config section set -c custom -j ""$.spec.controlPlane.spec.docker.repository=" + DOCKER_REPOSITORY + ""
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl bdc config section set -c custom -j ""$.spec.controlPlane.spec.docker.imageTag=" + DOCKER_IMAGE_TAG + ""
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl bdc create -c custom --accept-eula yes"
|
||||
executeCmd (command)
|
||||
|
||||
command="mssqlctl login --cluster-name " + CLUSTER_NAME
|
||||
command="azdata login --cluster-name " + CLUSTER_NAME
|
||||
executeCmd (command)
|
||||
|
||||
print("")
|
||||
print("SQL Server big data cluster endpoints: ")
|
||||
command="mssqlctl bdc endpoint list -o table"
|
||||
command="azdata bdc endpoint list -o table"
|
||||
executeCmd(command)
|
||||
|
||||
|
||||
@@ -1,30 +1,10 @@
|
||||
# Create a Kubernetes cluster using Kubeadm on Ubuntu 16.04 LTS or 18.04 LTS
|
||||
|
||||
|
||||
In this example, we will deploy Kubernetes over multiple Linux machines (physical or virtualized) using kubeadm utility. These instructions have been tested primarily with Ubuntu 16.04 LTS & 18.04 LTS versions.
|
||||
## __[ubuntu](ubuntu/)__
|
||||
|
||||
## Pre-requisites
|
||||
This folder contains scripts that provide a template for deploying a Kubernetes cluster using kubeadm on one or more Linux machines.
|
||||
|
||||
1. Multiple Linux machines or virtual machines. Recommended configuration is 8 CPUs, 32 GB memory each and at least 100 GB storage for each machine. Minimum number of machines required is three machines
|
||||
1. Designate one machine as the Kubernetes master
|
||||
1. Rest of the machines will be used as the Kubernetes agents
|
||||
## __[ubuntu-single-node-vm](ubuntu-single-node-vm/)__
|
||||
|
||||
**NOTE: Ensure there is sufficient local storage on your agents. Each volume will use up to 10GB by default. The script creates 25 volumes. Not all of the volumes will be used since it depends on the number of pods being deployed on each agent node. It is recommended to have at least 200 GB of storage on the agent nodes**
|
||||
|
||||
### Useful resources
|
||||
|
||||
[Deploy SQL Server 2019 big data cluster on Kubernetes](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)
|
||||
|
||||
[Creating a cluster using kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
|
||||
|
||||
[Troubleshooting kubeadm](https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/)
|
||||
|
||||
### Instructions
|
||||
|
||||
1. Start a sudo shell context
|
||||
1. Execute [ubuntu/setup-k8s-prereqs.sh](ubuntu/setup-k8s-prereqs.sh/) script on each machine
|
||||
1. Execute [ubuntu/setup-k8s-master.sh](ubuntu/setup-k8s-master.sh/) script on the machine designated as Kubernetes master
|
||||
1. After successful initialization of the Kubernetes master, follow the kubeadm join commands output by the setup script on each agent machine
|
||||
1. Execute [ubuntu/setup-volumes-agent.sh](ubuntu/setup-volumes-agent.sh/) script on each agent machine to create volumes for local storage
|
||||
1. Execute ***kubectl apply -f ubuntu/local-storage-provisioner.yaml*** against the Kubernetes cluster to create the local storage provisioner.
|
||||
1. Now, you can deploy the SQL Server 2019 big data cluster following instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)
|
||||
This folder contains a sample script that can be used to deploy a single-node Kubernetes cluster on a Linux machine.
|
||||
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
|
||||
# Deploy a SQL Server big data cluster on single node Kubernetes cluster (kubeadm)
|
||||
|
||||
Using this sample bash script, you will deploy a single node Kubernetes cluster using kubeadm and a SQL Server big data cluster that is integrated with Active Directory domain. The script must be run from the VM you are planning to use for your kubeadm deployment.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
1. A vanilla Ubuntu 16.04 or 18.04 virtual or physical machine on your corporate network. All dependencies will be setup by the script. Using Azure Linux VMs is not yet supported.
|
||||
1. Machine should have at least 8 CPUs, 64GB RAM and 100GB disk space. After installing the images you will be left with 50GB for data/logs across all components.
|
||||
1. Update existing packages using commands below to ensure that the OS image is up to date
|
||||
|
||||
``` bash
|
||||
sudo apt update&&apt upgrade -y
|
||||
sudo systemctl reboot
|
||||
```
|
||||
|
||||
## Recommended Virtual Machine settings
|
||||
|
||||
1. Use static memory configuration for the virtual machine. For example, in hyper-v installations do not use dynamic memory allocation but instead allocate the recommended 64 GB or higher.
|
||||
|
||||
1. Use checkpoint or snapshot capability in your hyper visor so that you can rollback the virtual machine to a clean state.
|
||||
|
||||
## Instructions to deploy SQL Server big data cluster
|
||||
|
||||
1. Download the script on the VM you are planning to use for the deployment
|
||||
|
||||
``` bash
|
||||
curl --output setup-bdc-ad.sh https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/setup-bdc-ad.sh
|
||||
curl --output endpoint-patch.json https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/endpoint-patch.json
|
||||
curl --output security-patch.json https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/security-patch.json
|
||||
```
|
||||
|
||||
1. Modify the endpoint-patch.json & security-patch.json files to supply the values specific to your Active Directory environment
|
||||
|
||||
1. Make the script executable
|
||||
|
||||
``` bash
|
||||
chmod +x setup-bdc-ad.sh
|
||||
```
|
||||
|
||||
1. Run the script (make sure you are running with sudo)
|
||||
|
||||
``` bash
|
||||
sudo ./setup-bdc-ad.sh
|
||||
```
|
||||
|
||||
1. Refresh alias setup for azdata
|
||||
|
||||
``` bash
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
When prompted, provide your input for the password that will be used for all external endpoints: controller, SQL Server master and gateway. The password should be sufficiently complex based on existing rules for SQL Server password. The controller username is defaulted to *admin*.
|
||||
|
||||
## Cleanup
|
||||
|
||||
1. The [cleanup-bdc.sh](cleanup-bdc.sh/) script is provided as convenience to reset the environment in case of errors. However, we recommend that you use a virtual machine for testing purposes and use the snapshot capability in your hyper-visor to rollback the virtual machine to a clean state.
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
#!/bin/bash
|
||||
|
||||
if [ "$EUID" -ne 0 ]
|
||||
then echo "Please run as root"
|
||||
exit
|
||||
fi
|
||||
DIR_PREFIX=$1
|
||||
|
||||
kubeadm reset --force
|
||||
unalias azdata
|
||||
|
||||
systemctl stop kubelet
|
||||
rm -rf /var/lib/cni/
|
||||
rm -rf /var/lib/etcd/
|
||||
rm -rf /run/flannel/
|
||||
rm -rf /var/lib/kubelet/*
|
||||
rm -rf /etc/cni/
|
||||
rm -rf /etc/kubernetes/
|
||||
|
||||
ip link set cni0 down
|
||||
#brctl delbr cni0
|
||||
ip link set flannel.1 down
|
||||
#brctl delbr flannel.1
|
||||
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
|
||||
|
||||
rm -rf .azdata/
|
||||
rm -rf bdcdeploy/
|
||||
|
||||
# Remove mounts.
|
||||
#
|
||||
SERVICE_STOP_FAILED=0
|
||||
|
||||
systemctl | grep "/var/lib/kubelet/pods" | while read -r line; do
|
||||
|
||||
# Retrieve the mount path
|
||||
#
|
||||
MOUNT_PATH=`echo "$line" | grep -v echo | egrep -oh -m 1 "(/var/lib/kubelet/pods).+"`
|
||||
|
||||
if [ -z "$MOUNT_PATH" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ ! -d "$MOUNT_PATH" ]] && [[ ! -f "$MOUNT_PATH" ]]; then
|
||||
|
||||
SERVICE=$(echo $line | cut -f1 -d' ')
|
||||
|
||||
echo "Mount "$MOUNT_PATH" no longer exists."
|
||||
echo "Stopping orphaned mount service: '$SERVICE'"
|
||||
|
||||
systemctl stop $SERVICE
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
SERVICE_STOP_FAILED=1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $SERVICE_STOP_FAILED -ne 0 ]; then
|
||||
echo "Not all services were stopped successfully. Please check the above output for more inforamtion."
|
||||
else
|
||||
echo "All orphaned services successfully stopped."
|
||||
fi
|
||||
|
||||
# Clean the mounted volumes.
|
||||
#
|
||||
|
||||
for i in $(seq 1 30); do
|
||||
|
||||
vol="vol$i"
|
||||
|
||||
sudo umount /mnt/local-storage/$vol
|
||||
|
||||
sudo rm -rf /mnt/local-storage/$vol
|
||||
|
||||
done
|
||||
|
||||
# Reset kube
|
||||
#
|
||||
sudo apt-get purge -y kubeadm --allow-change-held-packages
|
||||
sudo apt-get purge -y kubectl --allow-change-held-packages
|
||||
sudo apt-get purge -y kubelet --allow-change-held-packages
|
||||
sudo apt-get purge -y kubernetes-cni --allow-change-held-packages
|
||||
sudo apt-get purge -y kube* --allow-change-held-packages
|
||||
sudo apt -y autoremove
|
||||
sudo rm -rf ~/.kube
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"patch": [
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "spec.pools[?(@.spec.type=='Master')].spec",
|
||||
"value": {
|
||||
"type": "Master",
|
||||
"dnsName": "mastersql.contoso.local",
|
||||
"replicas": 1,
|
||||
"endpoints": [
|
||||
{
|
||||
"name": "Master",
|
||||
"serviceType": "NodePort",
|
||||
"port": 31433
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"patch": [
|
||||
{
|
||||
"op": "add",
|
||||
"path": "security",
|
||||
"value": {
|
||||
"useInternalDomain": false,
|
||||
"ouDistinguishedName":"OU=bdc,DC=contoso,DC=local",
|
||||
"dnsIpAddresses": ["11.11.111.11"],
|
||||
"domainControllerFullyQualifiedDns": ["VM.CONTOSO.LOCAL"],
|
||||
"realm":"CONTOSO.LOCAL",
|
||||
"domainDnsName":"contoso.local",
|
||||
"bdcAdminPrincipals": [
|
||||
"Domain Admins", "Enterprise Admins"
|
||||
],
|
||||
"bdcUserPrincipals": [
|
||||
"Domain Users"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"op": "add",
|
||||
"path": "spec.endpoints/0",
|
||||
"value": {
|
||||
"name": "Kerberos",
|
||||
"serviceType": "NodePort",
|
||||
"port": 30088
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+376
@@ -0,0 +1,376 @@
|
||||
#!/bin/bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
if [ "$EUID" -ne 0 ]
|
||||
then echo "Please run as root"
|
||||
exit
|
||||
fi
|
||||
|
||||
STARTUP_PATH=$(pwd)
|
||||
|
||||
# This is a script to create single-node Kubernetes cluster and deploy BDC on it.
|
||||
#
|
||||
export BDCDEPLOY_DIR=bdcdeploy
|
||||
|
||||
# Get password as input. It is used as default for controller, SQL Server Master instance (sa account) and Knox.
|
||||
#
|
||||
while true; do
|
||||
read -s -p "Create Password for Big Data Cluster: " password
|
||||
echo
|
||||
read -s -p "Confirm Password for Big Data Cluster: " password2
|
||||
echo
|
||||
[ "$password" = "$password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
echo ""
|
||||
# Get docker credentials for private release.
|
||||
#
|
||||
read -p "Enter Docker username: " DOCKER_USERNAME
|
||||
while true; do
|
||||
read -s -p "Enter Docker Password: " docker_password
|
||||
echo
|
||||
read -s -p "Confirm Docker Password: " docker_password2
|
||||
echo
|
||||
[ "$docker_password" = "$docker_password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
export DOCKER_PASSWORD=$docker_password
|
||||
echo ""
|
||||
|
||||
# Get Domain Service Account Username and Password.
|
||||
#
|
||||
read -p "Enter the Domain Service Account Username: " ds_username
|
||||
export DOMAIN_SERVICE_ACCOUNT_USERNAME=$ds_username
|
||||
|
||||
while true; do
|
||||
read -s -p "Enter Password for Domain Service Account: " ds_password
|
||||
echo
|
||||
read -s -p "Confirm your Domain Service Account Password: " ds_password2
|
||||
echo
|
||||
[ "$ds_password" = "$ds_password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
export DOMAIN_SERVICE_ACCOUNT_PASSWORD=$ds_password
|
||||
echo ""
|
||||
|
||||
# Name of virtualenv variable used.
|
||||
#
|
||||
export VIRTUALENV_NAME="bdcvenv"
|
||||
export LOG_FILE="bdcdeploy.log"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Requirements file.
|
||||
#
|
||||
export REQUIREMENTS_LINK="https://aka.ms/azdata"
|
||||
|
||||
# Kube version.
|
||||
#
|
||||
KUBE_DPKG_VERSION=1.15.0-00
|
||||
KUBE_VERSION=1.15.0
|
||||
|
||||
# Wait for 5 minutes for the cluster to be ready.
|
||||
#
|
||||
TIMEOUT=600
|
||||
RETRY_INTERVAL=5
|
||||
|
||||
# Variables for pulling dockers.
|
||||
#
|
||||
export DOCKER_REGISTRY="private-repo.microsoft.com"
|
||||
export DOCKER_REPOSITORY="mssql-private-preview"
|
||||
export DOCKER_TAG="ctp3.2.1"
|
||||
|
||||
# Variables used for azdata cluster creation.
|
||||
#
|
||||
export CONTROLLER_USERNAME=admin
|
||||
export CONTROLLER_PASSWORD=$password
|
||||
export MSSQL_SA_PASSWORD=$password
|
||||
export KNOX_PASSWORD=$password
|
||||
export ACCEPT_EULA=yes
|
||||
export CLUSTER_NAME=mssql-cluster
|
||||
export STORAGE_CLASS=local-storage
|
||||
export PV_COUNT="30"
|
||||
|
||||
IMAGES=(
|
||||
mssql-app-service-proxy
|
||||
mssql-appdeploy-init
|
||||
mssql-controller
|
||||
mssql-hadoop
|
||||
mssql-mleap-serving-runtime
|
||||
mssql-mlserver-py-runtime
|
||||
mssql-mlserver-r-runtime
|
||||
mssql-monitor-collectd
|
||||
mssql-monitor-elasticsearch
|
||||
mssql-monitor-fluentbit
|
||||
mssql-monitor-grafana
|
||||
mssql-monitor-influxdb
|
||||
mssql-monitor-kibana
|
||||
mssql-monitor-telegraf
|
||||
mssql-security-knox
|
||||
mssql-security-support
|
||||
mssql-server-controller
|
||||
mssql-server-data
|
||||
mssql-service-proxy
|
||||
mssql-ssis-app-runtime
|
||||
)
|
||||
|
||||
|
||||
# Make a directory for installing the scripts and logs.
|
||||
#
|
||||
mkdir -p $BDCDEPLOY_DIR
|
||||
cd $BDCDEPLOY_DIR/
|
||||
touch $LOG_FILE
|
||||
|
||||
{
|
||||
# Install all necessary packages: kuberenetes, docker, python3, python3-pip, request, azdata.
|
||||
#
|
||||
echo ""
|
||||
echo "######################################################################################"
|
||||
echo "Starting installing packages..."
|
||||
|
||||
# Install docker.
|
||||
#
|
||||
apt-get update -q
|
||||
|
||||
apt --yes install \
|
||||
software-properties-common \
|
||||
apt-transport-https \
|
||||
ca-certificates \
|
||||
curl
|
||||
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||
|
||||
add-apt-repository \
|
||||
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
||||
|
||||
apt update -q
|
||||
apt-get install -q --yes docker-ce=18.06.2~ce~3-0~ubuntu --allow-downgrades
|
||||
apt-mark hold docker-ce
|
||||
|
||||
usermod --append --groups docker $USER
|
||||
|
||||
# Install python3, python3-pip, requests.
|
||||
#
|
||||
apt-get install -q -y python3
|
||||
apt-get install -q -y python3-pip
|
||||
|
||||
pip3 install requests --upgrade
|
||||
|
||||
# Install and create virtualenv.
|
||||
#
|
||||
pip3 install --upgrade virtualenv
|
||||
virtualenv -p python3 $VIRTUALENV_NAME
|
||||
source $VIRTUALENV_NAME/bin/activate
|
||||
|
||||
# Install azdata cli.
|
||||
#
|
||||
pip3 install -r $REQUIREMENTS_LINK
|
||||
echo "Packages installed."
|
||||
|
||||
# Load all pre-requisites for Kubernetes.
|
||||
#
|
||||
echo "###########################################################################"
|
||||
echo "Starting to setup pre-requisites for kubernetes..."
|
||||
|
||||
# Setup the kubernetes preprequisites.
|
||||
#
|
||||
echo $(hostname -i) $(hostname) >> /etc/hosts
|
||||
|
||||
swapoff -a
|
||||
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
|
||||
|
||||
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
|
||||
|
||||
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
|
||||
|
||||
deb http://apt.kubernetes.io/ kubernetes-xenial main
|
||||
|
||||
EOF
|
||||
|
||||
# Install docker and packages to allow apt to use a repository over HTTPS.
|
||||
#
|
||||
apt-get update -q
|
||||
|
||||
apt-get install -q -y ebtables ethtool
|
||||
|
||||
#apt-get install -y docker.ce
|
||||
|
||||
apt-get install -q -y apt-transport-https
|
||||
|
||||
# Setup daemon.
|
||||
#
|
||||
cat > /etc/docker/daemon.json <<EOF
|
||||
{
|
||||
"exec-opts": ["native.cgroupdriver=systemd"],
|
||||
"log-driver": "json-file",
|
||||
"log-opts": {
|
||||
"max-size": "100m"
|
||||
},
|
||||
"storage-driver": "overlay2"
|
||||
}
|
||||
EOF
|
||||
|
||||
mkdir -p /etc/systemd/system/docker.service.d
|
||||
|
||||
# Restart docker.
|
||||
#
|
||||
systemctl daemon-reload
|
||||
systemctl restart docker
|
||||
|
||||
apt-get install -q -y kubelet=$KUBE_DPKG_VERSION kubeadm=$KUBE_DPKG_VERSION kubectl=$KUBE_DPKG_VERSION
|
||||
|
||||
# Holding the version of kube packages.
|
||||
#
|
||||
apt-mark hold kubelet kubeadm kubectl
|
||||
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
|
||||
|
||||
. /etc/os-release
|
||||
if [ "$UBUNTU_CODENAME" == "bionic" ]; then
|
||||
modprobe br_netfilter
|
||||
fi
|
||||
|
||||
# Disable Ipv6 for cluster endpoints.
|
||||
#
|
||||
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
|
||||
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
|
||||
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
|
||||
|
||||
echo net.ipv6.conf.all.disable_ipv6=1 > /etc/sysctl.conf
|
||||
echo net.ipv6.conf.default.disable_ipv6=1 > /etc/sysctl.conf
|
||||
echo net.ipv6.conf.lo.disable_ipv6=1 > /etc/sysctl.conf
|
||||
|
||||
|
||||
sysctl net.bridge.bridge-nf-call-iptables=1
|
||||
|
||||
# Setting up the persistent volumes for the kubernetes.
|
||||
#
|
||||
for i in $(seq 1 $PV_COUNT); do
|
||||
|
||||
vol="vol$i"
|
||||
|
||||
mkdir -p /mnt/local-storage/$vol
|
||||
|
||||
mount --bind /mnt/local-storage/$vol /mnt/local-storage/$vol
|
||||
|
||||
done
|
||||
echo "Kubernetes pre-requisites have been completed."
|
||||
|
||||
# Setup kubernetes cluster including remove taint on master.
|
||||
#
|
||||
echo ""
|
||||
echo "#############################################################################"
|
||||
echo "Starting to setup Kubernetes master..."
|
||||
|
||||
# Initialize a kubernetes cluster on the current node.
|
||||
#
|
||||
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=$KUBE_VERSION
|
||||
|
||||
mkdir -p $HOME/.kube
|
||||
mkdir -p /home/$SUDO_USER/.kube
|
||||
|
||||
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
|
||||
sudo chown $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.kube/config
|
||||
|
||||
# To enable a single node cluster remove the taint that limits the first node to master only service.
|
||||
#
|
||||
master_node=`kubectl get nodes --no-headers=true --output=custom-columns=NAME:.metadata.name`
|
||||
kubectl taint nodes ${master_node} node-role.kubernetes.io/master:NoSchedule-
|
||||
|
||||
# Local storage provisioning.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/local-storage-provisioner.yaml
|
||||
|
||||
# Install the software defined network.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
|
||||
|
||||
# helm init
|
||||
|
||||
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/rbac.yaml
|
||||
|
||||
# Verify that the cluster is ready to be used.
|
||||
#
|
||||
echo "Verifying that the cluster is ready for use..."
|
||||
while true ; do
|
||||
|
||||
if [[ "$TIMEOUT" -le 0 ]]; then
|
||||
echo "Cluster node failed to reach the 'Ready' state. Kubeadm setup failed."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
status=`kubectl get nodes --no-headers=true | awk '{print $2}'`
|
||||
|
||||
if [ "$status" == "Ready" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
sleep "$RETRY_INTERVAL"
|
||||
|
||||
TIMEOUT=$(($TIMEOUT-$RETRY_INTERVAL))
|
||||
|
||||
echo "Cluster not ready. Retrying..."
|
||||
done
|
||||
|
||||
|
||||
# Install the dashboard for Kubernetes.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
|
||||
|
||||
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
|
||||
echo "Kubernetes master setup done."
|
||||
|
||||
# Pull docker images of SQL Server big data cluster.
|
||||
#
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to pull docker images..."
|
||||
echo "Pulling images from repository: " $DOCKER_REGISTRY"/"$DOCKER_REPOSITORY
|
||||
|
||||
docker login $DOCKER_REGISTRY -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
|
||||
for image in "${IMAGES[@]}";
|
||||
do
|
||||
docker pull $DOCKER_REGISTRY/$DOCKER_REPOSITORY/$image:$DOCKER_TAG
|
||||
echo "Docker image" $image " pulled."
|
||||
done
|
||||
docker logout $DOCKER_REGISTRY
|
||||
echo "Docker images pulled."
|
||||
|
||||
# Deploy azdata bdc create cluster.
|
||||
#
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to deploy azdata cluster..."
|
||||
|
||||
# Command to create cluster for single node cluster.
|
||||
#
|
||||
azdata bdc config init --source kubeadm-dev-test --target kubeadm-custom -f
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.repository=$DOCKER_REPOSITORY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.registry=$DOCKER_REGISTRY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.imageTag=$DOCKER_TAG"
|
||||
azdata bdc config replace -c kubeadm-custom/cluster.json -j "$.spec.pools[?(@.spec.type == "Data")].spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
|
||||
azdata bdc config patch -c kubeadm-custom/control.json -p $STARTUP_PATH/security-patch.json
|
||||
azdata bdc config patch -c kubeadm-custom/cluster.json -p $STARTUP_PATH/endpoint-patch.json
|
||||
azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
|
||||
echo "Azdata cluster created."
|
||||
|
||||
# Setting context to cluster.
|
||||
#
|
||||
kubectl config set-context --current --namespace $CLUSTER_NAME
|
||||
|
||||
# Login and get endpoint list for the cluster.
|
||||
#
|
||||
azdata login -n $CLUSTER_NAME
|
||||
azdata bdc endpoint list --output table
|
||||
|
||||
if [ -d "$HOME/.azdata/" ]; then
|
||||
sudo chown -R $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.azdata/
|
||||
fi
|
||||
|
||||
if [ -d "$HOME/bdcdeploy/" ]; then
|
||||
sudo chown -R $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/bdcdeploy/
|
||||
fi
|
||||
|
||||
echo "alias azdata='$BDCDEPLOY_DIR/$VIRTUALENV_NAME/bin/azdata'" >> $HOME/.bashrc
|
||||
}| tee $LOG_FILE
|
||||
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
|
||||
# Deploy a SQL Server big data cluster on single node Kubernetes cluster (kubeadm)
|
||||
|
||||
Using this sample bash script, you will deploy a single node Kubernetes cluster using kubeadm and a SQL Server big data cluster on top of it. The script must be run from the VM you are planning to use for your kubeadm deployment.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
1. A vanilla Ubuntu 16.04 or 18.04 virtual or physical machine. All dependencies will be setup by the script. Using Azure Linux VMs is not yet supported.
|
||||
1. Machine should have at least 8 CPUs, 64GB RAM and 100GB disk space. After installing the images you will be left with 50GB for data/logs across all components.
|
||||
1. Update existing packages using commands below to ensure that the OS image is up to date
|
||||
|
||||
``` bash
|
||||
sudo apt update&&apt upgrade -y
|
||||
sudo systemctl reboot
|
||||
```
|
||||
|
||||
## Recommended Virtual Machine settings
|
||||
|
||||
1. Use static memory configuration for the virtual machine. For example, in hyper-v installations do not use dynamic memory allocation but instead allocate the recommended 64 GB or higher.
|
||||
|
||||
1. Use checkpoint or snapshot capability in your hyper visor so that you can rollback the virtual machine to a clean state.
|
||||
|
||||
## Instructions to deploy SQL Server big data cluster
|
||||
|
||||
1. Download the script on the VM you are planning to use for the deployment
|
||||
|
||||
``` bash
|
||||
curl --output setup-bdc.sh https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm/setup-bdc.sh
|
||||
```
|
||||
|
||||
2. Make the script executable
|
||||
|
||||
``` bash
|
||||
chmod +x setup-bdc.sh
|
||||
```
|
||||
|
||||
3. Run the script (make sure you are running with sudo)
|
||||
|
||||
``` bash
|
||||
sudo ./setup-bdc.sh
|
||||
```
|
||||
|
||||
4. Refresh alias setup for azdata
|
||||
|
||||
``` bash
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
When prompted, provide your input for the password that will be used for all external endpoints: controller, SQL Server master and gateway. The password should be sufficiently complex based on existing rules for SQL Server password. The controller username is defaulted to *admin*.
|
||||
|
||||
## Cleanup
|
||||
|
||||
1. The [cleanup-bdc.sh](cleanup-bdc.sh/) script is provided as convenience to reset the environment in case of errors. However, we recommend that you use a virtual machine for testing purposes and use the snapshot capability in your hyper-visor to rollback the virtual machine to a clean state.
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
#!/bin/bash
|
||||
|
||||
if [ "$EUID" -ne 0 ]
|
||||
then echo "Please run as root"
|
||||
exit
|
||||
fi
|
||||
DIR_PREFIX=$1
|
||||
|
||||
kubeadm reset --force
|
||||
unalias azdata
|
||||
|
||||
systemctl stop kubelet
|
||||
rm -rf /var/lib/cni/
|
||||
rm -rf /var/lib/etcd/
|
||||
rm -rf /run/flannel/
|
||||
rm -rf /var/lib/kubelet/*
|
||||
rm -rf /etc/cni/
|
||||
rm -rf /etc/kubernetes/
|
||||
|
||||
ip link set cni0 down
|
||||
#brctl delbr cni0
|
||||
ip link set flannel.1 down
|
||||
#brctl delbr flannel.1
|
||||
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
|
||||
|
||||
rm -rf .azdata/
|
||||
rm -rf bdcdeploy/
|
||||
|
||||
# Remove mounts.
|
||||
#
|
||||
SERVICE_STOP_FAILED=0
|
||||
|
||||
systemctl | grep "/var/lib/kubelet/pods" | while read -r line; do
|
||||
|
||||
# Retrieve the mount path
|
||||
#
|
||||
MOUNT_PATH=`echo "$line" | grep -v echo | egrep -oh -m 1 "(/var/lib/kubelet/pods).+"`
|
||||
|
||||
if [ -z "$MOUNT_PATH" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ ! -d "$MOUNT_PATH" ]] && [[ ! -f "$MOUNT_PATH" ]]; then
|
||||
|
||||
SERVICE=$(echo $line | cut -f1 -d' ')
|
||||
|
||||
echo "Mount "$MOUNT_PATH" no longer exists."
|
||||
echo "Stopping orphaned mount service: '$SERVICE'"
|
||||
|
||||
systemctl stop $SERVICE
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
SERVICE_STOP_FAILED=1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $SERVICE_STOP_FAILED -ne 0 ]; then
|
||||
echo "Not all services were stopped successfully. Please check the above output for more inforamtion."
|
||||
else
|
||||
echo "All orphaned services successfully stopped."
|
||||
fi
|
||||
|
||||
# Clean the mounted volumes.
|
||||
#
|
||||
|
||||
for i in $(seq 1 30); do
|
||||
|
||||
vol="vol$i"
|
||||
|
||||
sudo umount /mnt/local-storage/$vol
|
||||
|
||||
sudo rm -rf /mnt/local-storage/$vol
|
||||
|
||||
done
|
||||
|
||||
# Reset kube
|
||||
#
|
||||
sudo apt-get purge -y kubeadm --allow-change-held-packages
|
||||
sudo apt-get purge -y kubectl --allow-change-held-packages
|
||||
sudo apt-get purge -y kubelet --allow-change-held-packages
|
||||
sudo apt-get purge -y kubernetes-cni --allow-change-held-packages
|
||||
sudo apt-get purge -y kube* --allow-change-held-packages
|
||||
sudo apt -y autoremove
|
||||
sudo rm -rf ~/.kube
|
||||
+336
@@ -0,0 +1,336 @@
|
||||
#!/bin/bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
if [ "$EUID" -ne 0 ]
|
||||
then echo "Please run as root"
|
||||
exit
|
||||
fi
|
||||
|
||||
# This is a script to create single-node Kubernetes cluster and deploy BDC on it.
|
||||
#
|
||||
export BDCDEPLOY_DIR=bdcdeploy
|
||||
|
||||
# Get password as input. It is used as default for controller, SQL Server Master instance (sa account) and Knox.
|
||||
#
|
||||
while true; do
|
||||
read -s -p "Create Password for Big Data Cluster: " password
|
||||
echo
|
||||
read -s -p "Confirm your Password: " password2
|
||||
echo
|
||||
[ "$password" = "$password2" ] && break
|
||||
echo "Password mismatch. Please try again."
|
||||
done
|
||||
|
||||
|
||||
# Name of virtualenv variable used.
|
||||
#
|
||||
export VIRTUALENV_NAME="bdcvenv"
|
||||
export LOG_FILE="bdcdeploy.log"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Requirements file.
|
||||
#
|
||||
export REQUIREMENTS_LINK="https://aka.ms/azdata"
|
||||
|
||||
# Kube version.
|
||||
#
|
||||
KUBE_DPKG_VERSION=1.15.0-00
|
||||
KUBE_VERSION=1.15.0
|
||||
|
||||
# Wait for 5 minutes for the cluster to be ready.
|
||||
#
|
||||
TIMEOUT=600
|
||||
RETRY_INTERVAL=5
|
||||
|
||||
# Variables for pulling dockers.
|
||||
#
|
||||
export DOCKER_REGISTRY="mcr.microsoft.com"
|
||||
export DOCKER_REPOSITORY="mssql/bdc"
|
||||
export DOCKER_TAG="2019-CTP3.2-ubuntu"
|
||||
|
||||
# Variables used for azdata cluster creation.
|
||||
#
|
||||
export CONTROLLER_USERNAME=admin
|
||||
export CONTROLLER_PASSWORD=$password
|
||||
export MSSQL_SA_PASSWORD=$password
|
||||
export KNOX_PASSWORD=$password
|
||||
export ACCEPT_EULA=yes
|
||||
export CLUSTER_NAME=mssql-cluster
|
||||
export STORAGE_CLASS=local-storage
|
||||
export PV_COUNT="30"
|
||||
|
||||
IMAGES=(
|
||||
mssql-app-service-proxy
|
||||
mssql-appdeploy-init
|
||||
mssql-controller
|
||||
mssql-hadoop
|
||||
mssql-mleap-serving-runtime
|
||||
mssql-mlserver-py-runtime
|
||||
mssql-mlserver-r-runtime
|
||||
mssql-monitor-collectd
|
||||
mssql-monitor-elasticsearch
|
||||
mssql-monitor-fluentbit
|
||||
mssql-monitor-grafana
|
||||
mssql-monitor-influxdb
|
||||
mssql-monitor-kibana
|
||||
mssql-monitor-telegraf
|
||||
mssql-security-knox
|
||||
mssql-security-support
|
||||
mssql-server-controller
|
||||
mssql-server-data
|
||||
mssql-service-proxy
|
||||
mssql-ssis-app-runtime
|
||||
)
|
||||
|
||||
|
||||
# Make a directory for installing the scripts and logs.
|
||||
#
|
||||
mkdir -p $BDCDEPLOY_DIR
|
||||
cd $BDCDEPLOY_DIR/
|
||||
touch $LOG_FILE
|
||||
|
||||
{
|
||||
# Install all necessary packages: kuberenetes, docker, python3, python3-pip, request, azdata.
|
||||
#
|
||||
echo ""
|
||||
echo "######################################################################################"
|
||||
echo "Starting installing packages..."
|
||||
|
||||
# Install docker.
|
||||
#
|
||||
apt-get update -q
|
||||
|
||||
apt --yes install \
|
||||
software-properties-common \
|
||||
apt-transport-https \
|
||||
ca-certificates \
|
||||
curl
|
||||
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||
|
||||
add-apt-repository \
|
||||
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
||||
|
||||
apt update -q
|
||||
apt-get install -q --yes docker-ce=18.06.2~ce~3-0~ubuntu --allow-downgrades
|
||||
apt-mark hold docker-ce
|
||||
|
||||
usermod --append --groups docker $USER
|
||||
|
||||
# Install python3, python3-pip, requests.
|
||||
#
|
||||
apt-get install -q -y python3
|
||||
apt-get install -q -y python3-pip
|
||||
|
||||
pip3 install requests --upgrade
|
||||
|
||||
# Install and create virtualenv.
|
||||
#
|
||||
pip3 install --upgrade virtualenv
|
||||
virtualenv -p python3 $VIRTUALENV_NAME
|
||||
source $VIRTUALENV_NAME/bin/activate
|
||||
|
||||
# Install azdata cli.
|
||||
#
|
||||
pip3 install -r $REQUIREMENTS_LINK
|
||||
echo "Packages installed."
|
||||
|
||||
# Load all pre-requisites for Kubernetes.
|
||||
#
|
||||
echo "###########################################################################"
|
||||
echo "Starting to setup pre-requisites for kubernetes..."
|
||||
|
||||
# Setup the kubernetes preprequisites.
|
||||
#
|
||||
echo $(hostname -i) $(hostname) >> /etc/hosts
|
||||
|
||||
swapoff -a
|
||||
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
|
||||
|
||||
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
|
||||
|
||||
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
|
||||
|
||||
deb http://apt.kubernetes.io/ kubernetes-xenial main
|
||||
|
||||
EOF
|
||||
|
||||
# Install docker and packages to allow apt to use a repository over HTTPS.
|
||||
#
|
||||
apt-get update -q
|
||||
|
||||
apt-get install -q -y ebtables ethtool
|
||||
|
||||
#apt-get install -y docker.ce
|
||||
|
||||
apt-get install -q -y apt-transport-https
|
||||
|
||||
# Setup daemon.
|
||||
#
|
||||
cat > /etc/docker/daemon.json <<EOF
|
||||
{
|
||||
"exec-opts": ["native.cgroupdriver=systemd"],
|
||||
"log-driver": "json-file",
|
||||
"log-opts": {
|
||||
"max-size": "100m"
|
||||
},
|
||||
"storage-driver": "overlay2"
|
||||
}
|
||||
EOF
|
||||
|
||||
mkdir -p /etc/systemd/system/docker.service.d
|
||||
|
||||
# Restart docker.
|
||||
#
|
||||
systemctl daemon-reload
|
||||
systemctl restart docker
|
||||
|
||||
apt-get install -q -y kubelet=$KUBE_DPKG_VERSION kubeadm=$KUBE_DPKG_VERSION kubectl=$KUBE_DPKG_VERSION
|
||||
|
||||
# Holding the version of kube packages.
|
||||
#
|
||||
apt-mark hold kubelet kubeadm kubectl
|
||||
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
|
||||
|
||||
. /etc/os-release
|
||||
if [ "$UBUNTU_CODENAME" == "bionic" ]; then
|
||||
modprobe br_netfilter
|
||||
fi
|
||||
|
||||
# Disable Ipv6 for cluster endpoints.
|
||||
#
|
||||
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
|
||||
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
|
||||
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
|
||||
|
||||
echo net.ipv6.conf.all.disable_ipv6=1 > /etc/sysctl.conf
|
||||
echo net.ipv6.conf.default.disable_ipv6=1 > /etc/sysctl.conf
|
||||
echo net.ipv6.conf.lo.disable_ipv6=1 > /etc/sysctl.conf
|
||||
|
||||
|
||||
sysctl net.bridge.bridge-nf-call-iptables=1
|
||||
|
||||
# Setting up the persistent volumes for the kubernetes.
|
||||
#
|
||||
for i in $(seq 1 $PV_COUNT); do
|
||||
|
||||
vol="vol$i"
|
||||
|
||||
mkdir -p /mnt/local-storage/$vol
|
||||
|
||||
mount --bind /mnt/local-storage/$vol /mnt/local-storage/$vol
|
||||
|
||||
done
|
||||
echo "Kubernetes pre-requisites have been completed."
|
||||
|
||||
# Setup kubernetes cluster including remove taint on master.
|
||||
#
|
||||
echo ""
|
||||
echo "#############################################################################"
|
||||
echo "Starting to setup Kubernetes master..."
|
||||
|
||||
# Initialize a kubernetes cluster on the current node.
|
||||
#
|
||||
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=$KUBE_VERSION
|
||||
|
||||
mkdir -p $HOME/.kube
|
||||
mkdir -p /home/$SUDO_USER/.kube
|
||||
|
||||
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
|
||||
sudo chown $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.kube/config
|
||||
|
||||
# To enable a single node cluster remove the taint that limits the first node to master only service.
|
||||
#
|
||||
master_node=`kubectl get nodes --no-headers=true --output=custom-columns=NAME:.metadata.name`
|
||||
kubectl taint nodes ${master_node} node-role.kubernetes.io/master:NoSchedule-
|
||||
|
||||
# Local storage provisioning.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/local-storage-provisioner.yaml
|
||||
|
||||
# Install the software defined network.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
|
||||
|
||||
# helm init
|
||||
|
||||
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu/rbac.yaml
|
||||
|
||||
# Verify that the cluster is ready to be used.
|
||||
#
|
||||
echo "Verifying that the cluster is ready for use..."
|
||||
while true ; do
|
||||
|
||||
if [[ "$TIMEOUT" -le 0 ]]; then
|
||||
echo "Cluster node failed to reach the 'Ready' state. Kubeadm setup failed."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
status=`kubectl get nodes --no-headers=true | awk '{print $2}'`
|
||||
|
||||
if [ "$status" == "Ready" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
sleep "$RETRY_INTERVAL"
|
||||
|
||||
TIMEOUT=$(($TIMEOUT-$RETRY_INTERVAL))
|
||||
|
||||
echo "Cluster not ready. Retrying..."
|
||||
done
|
||||
|
||||
|
||||
# Install the dashboard for Kubernetes.
|
||||
#
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
|
||||
|
||||
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
|
||||
echo "Kubernetes master setup done."
|
||||
|
||||
# Pull docker images of SQL Server big data cluster.
|
||||
#
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to pull docker images..."
|
||||
echo "Pulling images from repository: " $DOCKER_REGISTRY"/"$DOCKER_REPOSITORY
|
||||
|
||||
for image in "${IMAGES[@]}";
|
||||
do
|
||||
docker pull $DOCKER_REGISTRY/$DOCKER_REPOSITORY/$image:$DOCKER_TAG
|
||||
echo "Docker image" $image " pulled."
|
||||
done
|
||||
echo "Docker images pulled."
|
||||
|
||||
# Deploy azdata bdc create cluster.
|
||||
#
|
||||
echo ""
|
||||
echo "############################################################################"
|
||||
echo "Starting to deploy azdata cluster..."
|
||||
|
||||
# Command to create cluster for single node cluster.
|
||||
#
|
||||
azdata bdc config init --source kubeadm-dev-test --target kubeadm-custom -f
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.repository=$DOCKER_REPOSITORY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.registry=$DOCKER_REGISTRY"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.imageTag=$DOCKER_TAG"
|
||||
azdata bdc config replace -c kubeadm-custom/cluster.json -j "$.spec.pools[?(@.spec.type == "Data")].spec.replicas=1"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
|
||||
azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
|
||||
azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
|
||||
echo "Azdata cluster created."
|
||||
|
||||
# Setting context to cluster.
|
||||
#
|
||||
kubectl config set-context --current --namespace $CLUSTER_NAME
|
||||
|
||||
# Login and get endpoint list for the cluster.
|
||||
#
|
||||
azdata login -n $CLUSTER_NAME
|
||||
azdata bdc endpoint list --output table
|
||||
|
||||
if [ -d "$HOME/.azdata/" ]; then
|
||||
sudo chown -R $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.azdata/
|
||||
fi
|
||||
|
||||
echo "alias azdata='$BDCDEPLOY_DIR/$VIRTUALENV_NAME/bin/azdata'" >> $HOME/.bashrc
|
||||
}| tee $LOG_FILE
|
||||
@@ -0,0 +1,29 @@
|
||||
# Create a Kubernetes cluster using Kubeadm on Ubuntu 16.04 LTS or 18.04 LTS
|
||||
|
||||
In this example, we will deploy Kubernetes over multiple Linux machines (physical or virtualized) using kubeadm utility. These instructions have been tested primarily with Ubuntu 16.04 LTS & 18.04 LTS versions.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
1. Multiple Ubuntu Linux machines or virtual machines. Recommended configuration is 8 CPUs, 32 GB memory each and at least 100 GB storage for each machine. Minimum number of machines required is three machines
|
||||
1. Designate one machine as the Kubernetes master
|
||||
1. Rest of the machines will be used as the Kubernetes agents
|
||||
|
||||
**NOTE: Ensure there is sufficient local storage on your agents. Each volume will use up to 10GB by default. The script creates 25 volumes. Not all of the volumes will be used since it depends on the number of pods being deployed on each agent node. It is recommended to have at least 200 GB of storage on the agent nodes**
|
||||
|
||||
### Useful resources
|
||||
|
||||
[Deploy SQL Server 2019 big data cluster on Kubernetes](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)
|
||||
|
||||
[Creating a cluster using kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
|
||||
|
||||
[Troubleshooting kubeadm](https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/)
|
||||
|
||||
### Instructions
|
||||
|
||||
1. Start a sudo shell context
|
||||
1. Execute [setup-k8s-prereqs.sh](setup-k8s-prereqs.sh/) script on each machine
|
||||
1. Execute [setup-k8s-master.sh](setup-k8s-master.sh/) script on the machine designated as Kubernetes master
|
||||
1. After successful initialization of the Kubernetes master, follow the kubeadm join commands output by the setup script on each agent machine
|
||||
1. Execute [setup-volumes-agent.sh](setup-volumes-agent.sh/) script on each agent machine to create volumes for local storage
|
||||
1. Execute ***kubectl apply -f local-storage-provisioner.yaml*** against the Kubernetes cluster to create the local storage provisioner.
|
||||
1. Now, you can deploy the SQL Server 2019 big data cluster following instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions)
|
||||
+1
-1
@@ -19,7 +19,7 @@ apt-get install -y kubelet=$KUBE_DPKG_VERSION kubeadm=$KUBE_DPKG_VERSION kubectl
|
||||
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
|
||||
|
||||
. /etc/os-release
|
||||
if [ "$VERSION_CODENAME" == "bionic" ]; then
|
||||
if [ "$UBUNTU_CODENAME" == "bionic" ]; then
|
||||
modprobe br_netfilter
|
||||
fi
|
||||
sysctl net.bridge.bridge-nf-call-iptables=1
|
||||
|
||||
@@ -24,7 +24,7 @@ python push-bdc-images-to-custom-private-repo.py
|
||||
>```
|
||||
|
||||
When prompted, provide your input for:
|
||||
- Docker registry, repository and credentials to access Microsoft private registry where the images will be pulled from (source)
|
||||
- Docker registry, repository to access Microsoft registry where the images will be pulled from (source). Press enter to use default mcr registry.
|
||||
- Docker registry, repository and credentials to access your private registry where the images will be pushed to (target)
|
||||
|
||||
## Deploy with from your private repository
|
||||
|
||||
+12
-9
@@ -13,10 +13,11 @@ def execute_cmd (cmd):
|
||||
if (stderr is not None):
|
||||
raise Exception(stderr)
|
||||
|
||||
SOURCE_DOCKER_REGISTRY = input("Provide Docker registry source - press ENTER for using `private-repo.microsoft.com`:") or "private-repo.microsoft.com"
|
||||
SOURCE_DOCKER_REPOSITORY = input("Provide Docker repository source - press ENTER for using `mssql-private-preview`:") or "mssql-private-preview"
|
||||
SOURCE_DOCKER_USERNAME = input("Provide Docker username for the source registry:")
|
||||
SOURCE_DOCKER_PASSWORD=getpass.getpass("Provide Docker password for the source registry:")
|
||||
SOURCE_DOCKER_REGISTRY = input("Provide Docker registry source - press ENTER for using `mcr.microsoft.com`:") or "mcr.microsoft.com"
|
||||
SOURCE_DOCKER_REPOSITORY = input("Provide Docker repository source - press ENTER for using `mssql/bdc`:") or "mssql/bdc"
|
||||
# Use this only if your source is a private Docker registry
|
||||
# SOURCE_DOCKER_USERNAME = input("Provide Docker username for the source registry:")
|
||||
# SOURCE_DOCKER_PASSWORD=getpass.getpass("Provide Docker password for the source registry:")
|
||||
SOURCE_DOCKER_TAG = input("Provide Docker tag for the images at the source: ") or "latest"
|
||||
|
||||
TARGET_DOCKER_REGISTRY = input("Provide Docker registry target:")
|
||||
@@ -43,12 +44,14 @@ images = [ 'mssql-appdeploy-init',
|
||||
'mssql-service-proxy',
|
||||
'mssql-app-service-proxy',
|
||||
'mssql-ssis-app-runtime',
|
||||
'mssql-monitor-telegraf']
|
||||
'mssql-monitor-telegraf',
|
||||
'mssql-security-support']
|
||||
|
||||
print("Execute docker login to source registry: " + SOURCE_DOCKER_REGISTRY)
|
||||
cmd = "docker login " + SOURCE_DOCKER_REGISTRY + " -u " + SOURCE_DOCKER_USERNAME + " -p " + SOURCE_DOCKER_PASSWORD
|
||||
execute_cmd(cmd)
|
||||
print("")
|
||||
# Use this only if your source is a private Docker registry
|
||||
# print("Execute docker login to source registry: " + SOURCE_DOCKER_REGISTRY)
|
||||
# cmd = "docker login " + SOURCE_DOCKER_REGISTRY + " -u " + SOURCE_DOCKER_USERNAME + " -p " + SOURCE_DOCKER_PASSWORD
|
||||
# execute_cmd(cmd)
|
||||
# print("")
|
||||
|
||||
|
||||
print("Pulling images from source repository: " + SOURCE_DOCKER_REGISTRY + "/" + SOURCE_DOCKER_REPOSITORY)
|
||||
|
||||
@@ -10,8 +10,8 @@ This folder contains the R samples.
|
||||
|
||||
[python](python)
|
||||
|
||||
This folder contains the R samples.
|
||||
This folder contains the Python samples.
|
||||
|
||||
[java](java)
|
||||
[java](https://github.com/microsoft/sql-server-language-extensions/tree/master/language-extensions/java/samples/regex)
|
||||
|
||||
This folder contains the Java samples.
|
||||
|
||||
+2
-2
@@ -25,7 +25,7 @@ SELECT TOP(80) PERCENT SIGN(q.clicks_in_category) AS book_category
|
||||
, q.clicks_in_7
|
||||
, q.clicks_in_8
|
||||
, q.clicks_in_9
|
||||
FROM web_clickstreams_book_clicks as q
|
||||
FROM web_clickstreams_hdfs_book_clicks as q
|
||||
';
|
||||
-- Training R script that uses rxLogit function from RevoScaleR package (Microsoft R Server) to generate model to predict book_category click(s).
|
||||
SET @train_script = N'
|
||||
@@ -91,7 +91,7 @@ SELECT TOP(@top_count_value) PERCENT SIGN(q.clicks_in_category) AS book_category
|
||||
, q.clicks_in_7
|
||||
, q.clicks_in_8
|
||||
, q.clicks_in_9
|
||||
FROM web_clickstreams_book_clicks as q
|
||||
FROM web_clickstreams_hdfs_book_clicks as q
|
||||
';
|
||||
|
||||
-- Scoring script that uses sklearn logistic regression model to predict book_category click(s)
|
||||
|
||||
@@ -4,13 +4,13 @@ SQL Server Big Data cluster bundles Spark and HDFS together with SQL server. Azu
|
||||
|
||||
## Contents
|
||||
|
||||
[PySpark Hello World](dataloading/hello_PySpark.ipynb)
|
||||
[PySpark Hello World](data-loading/hello_PySpark.ipynb)
|
||||
|
||||
[Scala Hello World ](dataloading/hello_Scala.ipynb)
|
||||
[Scala Hello World ](data-loading/hello_Scala.ipynb)
|
||||
|
||||
[SparkR Hello World ](dataloading/hello_sparkR.ipynb)
|
||||
[SparkR Hello World ](data-loading/hello_sparkR.ipynb)
|
||||
|
||||
[DataLoading - Transforming CSV to Parquet](dataloading/transform-csv-files.ipynb/)
|
||||
[Data Loading - Transforming CSV to Parquet](data-loading/transform-csv-files.ipynb/)
|
||||
|
||||
[Data Transfer - Spark to SQL using Spark JDBC connector](data-virtualization/spark_to_sql_jdbc.ipynb/)
|
||||
|
||||
@@ -20,7 +20,7 @@ SQL Server Big Data cluster bundles Spark and HDFS together with SQL server. Azu
|
||||
|
||||
[Install - Install 3rd party packages](config-install/installpackage_Spark.ipynb/)
|
||||
|
||||
[Restful-Access - Access Spark in BDC via restful Livy APIs](restful-api-accessn/accessing_spark_via_livy.ipynb/)
|
||||
[Restful-Access - Access Spark in BDC via restful Livy APIs](restful-api-access/accessing_spark_via_livy.ipynb/)
|
||||
|
||||
## Instructions on how to run in Azure Data Studio
|
||||
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
{
|
||||
"checks":[
|
||||
{
|
||||
"id": "SqlServer.Database.AutoCreateStats",
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"select": ["TraceFlag"],
|
||||
"enabled": false
|
||||
}
|
||||
]
|
||||
}
|
||||
{
|
||||
"checks":[
|
||||
{
|
||||
"id": "SqlServer.Database.AutoCreateStats",
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"select": ["TraceFlag"],
|
||||
"enabled": false
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@@ -1,60 +1,61 @@
|
||||
{
|
||||
"checks":[
|
||||
{
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"id": "Custom_SqlServer.Database.AutoUpdateStats",
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ],
|
||||
"displayName": "Auto-Update Statistics should be on",
|
||||
"description": "The query optimizer needs up-to-date and accurate statistics in order to generate good plans. In most cases, it's best to let SQL Server maintain the statistics. If you turn 'Auto Create Stats' and 'Auto Update Stats' off, then it is up to you to keep the statistics up-to-date somehow. Failure to do so will lead to poor query performance. Most applications should have these options ON.\n \n When the Auto Update Statistics setting is ON, the query optimizer updates statistics when they are used by a query and when they might be out-of-date. Statistics become out-of-date after insert, update, delete, or merge operations change the data distribution in the table or indexed view. The query optimizer determines when statistics might be out-of-date by counting the number of data modifications since the last statistics update and comparing the number of modifications to a threshold. The threshold is based on the number of rows in the table or indexed view. The query optimizer checks for out-of-date statistics before compiling a query and before executing a cached query plan. Before compiling a query, the query optimizer uses the columns, tables, and indexed views in the query predicate to determine which statistics might be out-of-date. Before executing a cached query plan, the Database Engine verifies that the query plan references up-to-date statistics. The AUTO_UPDATE_STATISTICS option applies to statistics created for indexes, single-columns in query predicates, and statistics that are created by using the CREATE STATISTICS statement. This option also applies to filtered statistics.",
|
||||
"message": "Turn Auto-Update Statistics option on to improve query performance.",
|
||||
"helpLink": "https://blogs.msdn.microsoft.com/buckwoody/2009/08/18/sql-server-best-practices-auto-create-and-auto-update-statistics-should-be-on-most-of-the-time/",
|
||||
"probes": [ "DatabaseConfiguration" ],
|
||||
"condition": "@is_auto_update_stats_on"
|
||||
},
|
||||
{
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)",
|
||||
"platform": "Windows",
|
||||
"name": { "not": "/^(master|msdb)$/" }
|
||||
},
|
||||
"id": "Custom_SqlServer.Database.QueryStoreOn",
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ],
|
||||
"displayName": "Query Store should be on",
|
||||
"description": "The SQL Server Query Store feature provides you with insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes. Query Store automatically captures a history of queries, plans, and runtime statistics, and retains these for your review. It separates data by time windows so you can see database usage patterns and understand when query plan changes happened on the server.",
|
||||
"message": "Turn Query Store option on to improve query performance troubleshooting.",
|
||||
"helpLink": "https://docs.microsoft.com/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store",
|
||||
"probes": [ "DatabaseConfiguration" ],
|
||||
"condition": "@is_query_store_on"
|
||||
}
|
||||
],
|
||||
"probes":{
|
||||
"DatabaseConfiguration": [
|
||||
{
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "(,12.0)",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": {
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, 0 AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": {
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, db.[is_query_store_on] AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
{
|
||||
"checks":[
|
||||
{
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"id": "Custom_SqlServer.Database.AutoUpdateStats",
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ],
|
||||
"displayName": "Auto-Update Statistics should be on",
|
||||
"description": "The query optimizer needs up-to-date and accurate statistics in order to generate good plans. In most cases, it's best to let SQL Server maintain the statistics. If you turn 'Auto Create Stats' and 'Auto Update Stats' off, then it is up to you to keep the statistics up-to-date somehow. Failure to do so will lead to poor query performance. Most applications should have these options ON.\n \n When the Auto Update Statistics setting is ON, the query optimizer updates statistics when they are used by a query and when they might be out-of-date. Statistics become out-of-date after insert, update, delete, or merge operations change the data distribution in the table or indexed view. The query optimizer determines when statistics might be out-of-date by counting the number of data modifications since the last statistics update and comparing the number of modifications to a threshold. The threshold is based on the number of rows in the table or indexed view. The query optimizer checks for out-of-date statistics before compiling a query and before executing a cached query plan. Before compiling a query, the query optimizer uses the columns, tables, and indexed views in the query predicate to determine which statistics might be out-of-date. Before executing a cached query plan, the Database Engine verifies that the query plan references up-to-date statistics. The AUTO_UPDATE_STATISTICS option applies to statistics created for indexes, single-columns in query predicates, and statistics that are created by using the CREATE STATISTICS statement. This option also applies to filtered statistics.",
|
||||
"message": "Turn Auto-Update Statistics option on to improve query performance.",
|
||||
"helpLink": "https://blogs.msdn.microsoft.com/buckwoody/2009/08/18/sql-server-best-practices-auto-create-and-auto-update-statistics-should-be-on-most-of-the-time/",
|
||||
"probes": [ "DatabaseConfiguration" ],
|
||||
"condition": "@is_auto_update_stats_on"
|
||||
},
|
||||
{
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)",
|
||||
"platform": "Windows",
|
||||
"name": { "not": "/^(master|msdb)$/" }
|
||||
},
|
||||
"id": "Custom_SqlServer.Database.QueryStoreOn",
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ],
|
||||
"displayName": "Query Store should be on",
|
||||
"description": "The SQL Server Query Store feature provides you with insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes. Query Store automatically captures a history of queries, plans, and runtime statistics, and retains these for your review. It separates data by time windows so you can see database usage patterns and understand when query plan changes happened on the server.",
|
||||
"message": "Turn Query Store option on to improve query performance troubleshooting.",
|
||||
"helpLink": "https://docs.microsoft.com/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store",
|
||||
"probes": [ "DatabaseConfiguration" ],
|
||||
"condition": "@is_query_store_on"
|
||||
}
|
||||
],
|
||||
"probes":{
|
||||
"DatabaseConfiguration": [
|
||||
{
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "(,12.0)",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": {
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, 0 AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)",
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": {
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, db.[is_query_store_on] AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,68 +1,73 @@
|
||||
# SQL Assessment API samples
|
||||
|
||||
Contains samples for customizing SQL Assessment API. Learn more about API and how run it with your own customization here <link to the SQL Assessment docs page>.
|
||||
|
||||
## DisablingBuiltInChecks_sample.json
|
||||
|
||||
Contains two parts. First shows how you can disable a specified check by its ID. The second disables all the checks with the "TraceFlag" tag.
|
||||
|
||||
## MakingCustomChecks_sample.json
|
||||
|
||||
Demonstrates how to make a custom rule set containing two checks. The sample contains two sections: `checks` and `probes`. `Checks` is for check (or rule) definitions. Usually, checks or rules are best practices or a company's internal policies that should be applied to SQL Server. Here's one of the checks from this sample with comments on each property:
|
||||
|
||||
```
|
||||
{
|
||||
"target": { //Object to describe which SQL Server object this check is applied.
|
||||
"type": "Database", //This check targets at Database object.
|
||||
"version": "[12.0,)", //Applies to SQL Server 2014 and higher.
|
||||
//Another example: "[12.0,13.0)" reads as "any SQL Server with version >= 12.0 and < 13.0.
|
||||
"platform": "Windows", //Applies to SQL Server on Windows.
|
||||
"name": { "not": "/^(master|msdb)$/" } //Applies to any database but master and msdb.
|
||||
},
|
||||
"id": "CustomCheck1", //Check ID.
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ], //Tags combine checks in different subsets.
|
||||
"displayName": "Query Store should be on", //Short name for check.
|
||||
"description": "The SQL Server Query Store feature provides you with insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes. /n Query Store automatically captures a history of queries, plans, and runtime statistics, and retains these for your review. It separates data by time windows so you can see database usage patterns and understand when query plan changes happened on the server.",
|
||||
//Some more detailed explanation of the best practice or policy.
|
||||
"message": "Turn Query Store option on to improve query performance troubleshooting.",
|
||||
//Usually, it's for recommendation what the user should do if the check fires up
|
||||
"helpLink": "https://docs.microsoft.com/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store",
|
||||
//Reference material
|
||||
"probes": [ "DatabaseConfiguration" ], //List of probes that are used to get the required data for this check.
|
||||
//Probes will be explained below.
|
||||
"condition": "@is_query_store_on" //Check will pass if condition is true. Otherwise, the check fires up.
|
||||
}
|
||||
```
|
||||
|
||||
`Probes`, in fact, describe how and where get required data to perform a check. For this, you can use T-SQL queries as well as methods from assemblies. The probe below uses a T-SQL query.
|
||||
```
|
||||
"probes":{
|
||||
"DatabaseConfiguration": [ //Probe name that is used to reference the probe from a check.
|
||||
//Probe can have a few implementations that will be used for different targets.
|
||||
//This probe has two implementations for different version of SQL Server.
|
||||
{
|
||||
"type": "SQL", //Probe uses a T-SQL query to get the required data
|
||||
"target": {
|
||||
"type": "Database", //Targets at database
|
||||
"version": "(,12.0)", //This implementation is for SQL Server before 2014
|
||||
"platform": "Windows" //Targets at SQL on Windows
|
||||
},
|
||||
"implementation": { //Implementation object with a T-SQL query.
|
||||
//sys.databases of SQL Server before 2014 doesn't have the field is_query_store_on so we replace it with 0.
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, 0 AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
},
|
||||
{ //Second implementation
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)", //This implementation is for SQL Server 2014 and up.
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": { //Query of the second implementation.
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, db.[is_query_store_on] AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
# SQL Assessment API samples
|
||||
|
||||
Contains samples for customizing SQL Assessment API. Learn more about the API on the [SQL Assessment API docs page](https://docs.microsoft.com/en-us/sql/sql-assessment-api/sql-assessment-api-overview).
|
||||
|
||||
## config.json
|
||||
|
||||
This is the default set of checks shipped with SQL Assessment API. Feel free to open issues to have us fix or add checks. Also, we're happy to see your pull requests to this file.
|
||||
|
||||
## DisablingBuiltInChecks_sample.json
|
||||
|
||||
Contains two parts. First shows how you can disable a specified check by its ID. The second disables all the checks with the "TraceFlag" tag.
|
||||
|
||||
## MakingCustomChecks_sample.json
|
||||
|
||||
Demonstrates how to make a custom rule set containing two checks. The sample contains two sections: `checks` and `probes`. `Checks` is for check (or rule) definitions. Usually, checks or rules are best practices or a company's internal policies that should be applied to SQL Server. Here's one of the checks from this sample with comments on each property:
|
||||
|
||||
```
|
||||
{
|
||||
"target": { //Object to describe which SQL Server object this check is applied.
|
||||
"type": "Database", //This check targets at Database object.
|
||||
"version": "[12.0,)", //Applies to SQL Server 2014 and higher.
|
||||
//Another example: "[12.0,13.0)" reads as "any SQL Server with version >= 12.0 and < 13.0.
|
||||
"platform": "Windows", //Applies to SQL Server on Windows.
|
||||
"name": { "not": "/^(master|msdb)$/" } //Applies to any database but master and msdb.
|
||||
},
|
||||
"id": "CustomCheck1", //Check ID.
|
||||
"tags": [ "InternalBestPracticeSet", "Performance" ], //Tags combine checks in different subsets.
|
||||
"displayName": "Query Store should be on", //Short name for check.
|
||||
"description": "The SQL Server Query Store feature provides you with insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes. /n Query Store automatically captures a history of queries, plans, and runtime statistics, and retains these for your review. It separates data by time windows so you can see database usage patterns and understand when query plan changes happened on the server.",
|
||||
//Some more detailed explanation of the best practice or policy.
|
||||
"message": "Turn Query Store option on to improve query performance troubleshooting.",
|
||||
//Usually, it's for recommendation what the user should do if the check fires up
|
||||
"helpLink": "https://docs.microsoft.com/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store",
|
||||
//Reference material
|
||||
"probes": [ "DatabaseConfiguration" ], //List of probes that are used to get the required data for this check.
|
||||
//Probes will be explained below.
|
||||
"condition": "@is_query_store_on" //Check will pass if condition is true. Otherwise, the check fires up.
|
||||
}
|
||||
```
|
||||
|
||||
`Probes` describe how and where get required data to perform a check. For this, you can use T-SQL queries as well as methods from assemblies. The probe below uses a T-SQL query.
|
||||
```
|
||||
"probes":{
|
||||
"DatabaseConfiguration": [ //Probe name that is used to reference the probe from a check.
|
||||
//Probe can have a few implementations that will be used for different targets.
|
||||
//This probe has two implementations for different version of SQL Server.
|
||||
{
|
||||
"type": "SQL", //Probe uses a T-SQL query to get the required data
|
||||
"target": {
|
||||
"type": "Database", //Targets at database
|
||||
"version": "(,12.0)", //This implementation is for SQL Server before 2014
|
||||
"platform": "Windows" //Targets at SQL on Windows
|
||||
},
|
||||
"implementation": { //Implementation object with a T-SQL query.
|
||||
//sys.databases of SQL Server before 2014 doesn't have the field is_query_store_on so we replace it with 0.
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, 0 AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
},
|
||||
{ //Second implementation
|
||||
"type": "SQL",
|
||||
"target": {
|
||||
"type": "Database",
|
||||
"version": "[12.0,)", //This implementation is for SQL Server 2014 and up.
|
||||
"platform": "Windows"
|
||||
},
|
||||
"implementation": { //Query of the second implementation.
|
||||
"query": "SELECT db.[is_auto_create_stats_on] AS is_auto_create_stats_on, db.[is_auto_update_stats_on] AS is_auto_update_stats_on, db.[is_query_store_on] AS is_query_store_on FROM sys.databases AS db WHERE db.[name]='@DatabaseName'"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user