github/sql-server-samples
1
0
Fork 0
mirror of https://github.com/Microsoft/sql-server-samples.git synced 2025-12-08 14:58:54 +00:00
Code Issues Releases Wiki Activity

Merge pull request #530 from jeroenterheerdt/master

Browse Source 
Updating readme for app-deploy samples
This commit is contained in:
Jeroen ter Heerdt
2019-02-27 16:59:37 -08:00
committed by GitHub
parent 98ad77b4ff a484fb1ddd
commit d97f978a15
3 changed files with 43 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/features/sql-big-data-cluster/app-deploy/README.md
+2 -2
View File
@@ -27,9 +27,9 @@ __[RollDice](RollDice/)__
This sample demonstrates the use of data frames
## MLeap
__[hello-mleap](hello-mleap/)__
__[mleap](mleap/)__
This sample demonstrates how you use a MLeap bundle ( a Spark model serialized in this format) and run it outside of Spark. The sample is based on the MLeap sample available here http://mleap-docs.combust.ml/mleap-serving/. We are using the MLeap Serving container that is published in Docker Hub. The MLeap Serving is deployed as container in SQL Server big data cluster as a container app with a web service that takes the Leap Frame as the input.
This sample demonstrates how you use a MLeap bundle (a Spark model serialized in this format) and run it outside of Spark. The sample is based on the MLeap sample available here http://mleap-docs.combust.ml/mleap-serving/. We are using the MLeap Serving container that is published in Docker Hub. The MLeap Serving is deployed as container in SQL Server big data cluster as a container app with a web service that takes the Leap Frame as the input.
## Sql Server Integration Services
samples/features/sql-big-data-cluster/app-deploy/SSIS/README.md
+19 -8
View File
@@ -1,5 +1,5 @@
## About
This is a sample SQL Server Integration Services (SSIS) app, which shows how to run a SSIS package as a scheduled service. This sample creates an app that is called each minute that executes an SSIS package. The SSIS package creates a backup of the `DWConfiguration` database on the master SQL instance to disk. Also, the package cleans any backup files for the `DWConfiguration` database that are older than one hour, making sure that maximum 60 backup files will be on disk at any moment.
This is a sample [SQL Server Integration Services (SSIS)](https://docs.microsoft.com/en-us/sql/integration-services/sql-server-integration-services?view=sql-server-2017) app, which shows how to run a SSIS package as a scheduled service. This sample creates an app that is called each minute that executes an SSIS package. The SSIS package creates a backup of the `DWConfiguration` database on the master SQL instance to disk. Also, the package cleans any backup files for the `DWConfiguration` database that are older than one hour, making sure that maximum 60 backup files will be on disk at any moment.
Refer to [installing mssqlctl](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions) document on setting up the mssqlctl and connecting to a Aris cluster.
@@ -16,34 +16,45 @@ Apart from regular settings, the `spec.yaml` file in this example specifies `opt
|options|Specifies any command line parameters passed to the execution of the SSIS package|
|schedule|Specifies when the job should run. This follows cron expressions. A value of '*/1 * * * *' means the job runs *every minute*.|
## How to run
# Pre-requisites
SQL Server big data cluster - CTP 2.3 or later
Clone or download this sample on your computer to a folder called `mleap` (note if you have downloaded it to a different folder then you'll have to modify the folder location appropriately in the information below).
### Change the `spec.yaml`
## Running the sample
### Connecting to SQL Server big data cluster
Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
```bash
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
```
### Changing the `spec.yaml`
Replace `[SA_PASSWORD]` in the `spec.yaml` file with the password for SQL user `sa`.
### Create the app:
### Deploying the application
```bash
# drop back-up-db.dtsx and spec.yaml in a folder, e.g. name back-up-db
# edit back-up-db.dtsx, replace the value after "Data Source" in the connection string to "service-master-pool;" if not alread. Then deploy it by:
mssqlctl app create --spec ./back-up-db
```
### Check the job:
### Testing the deployment
```bash
mssqlctl app list
```
Once the app is listed as `Ready` the job should run within a minute.
You can check if the backup is created by running:
```bash
kubectl -n test exec -it mssql-master-pool-0 -c mssql-server -- /bin/bash -c "ls /var/opt/mssql/data/*.DWConfigbak"
kubectl -n [your namespace] exec -it mssql-master-pool-0 -c mssql-server -- /bin/bash -c "ls /var/opt/mssql/data/*.DWConfigbak"
```
You should see a backup being created for every run of the job, with a maximum of 60 backups since the SSIS package cleans up backups older than one hour.
You can use any of the `.DWConfigbak` files to restore the database.
### Clean up:
### Clean up
```bash
# delete app
mssqlctl app delete --name back-up-db --version v1
# delete backup files
kubectl -n test exec -it mssql-master-pool-0 -c mssql-server -- /bin/bash -c "rm /var/opt/mssql/data/*.DWConfigbak"
kubectl -n [your namespace] exec -it mssql-master-pool-0 -c mssql-server -- /bin/bash -c "rm /var/opt/mssql/data/*.DWConfigbak"
```
samples/features/sql-big-data-cluster/app-deploy/mleap/README.md
+22 -22
View File
@@ -1,24 +1,26 @@
# How to deploy an MLeap app
This sample assumes you are familar with MLeap. MLeap provides simple interfaces to execute entire ML pipelines. You can get additional information about MLeap [here](http://mleap-docs.combust.ml/). When you build a model in Spark, typically for training and need score this outside of Spark environment you can serialize the model as an MLeap bundle and score this outside of Spark. This allows model portability. This example will demonstrate how a trained model serialized as MLeap bundle can be deployed as a RESTful webservice with a single line of code in SQL Server BDC 2019 and use as sample input in leap frame format to test it.
This sample assumes you are familiar with MLeap. MLeap provides simple interfaces to execute entire ML pipelines. For additional information on MLeap [see the documentation](http://mleap-docs.combust.ml/). MLeap allows you to take a trained Spark model and use it outside of Spark, for example for scoring. After you have created and trained your model in Spark, you can serialize the model as an MLeap bundle for use outside of Spark. This allows model portability and using the model for scoring outside of Spark. This example demonstrates how to serialize a trained model as MLeap bundle and how to deploy it a RESTful web service with a single line of code in SQL Server big data cluster. Also, this example shows how to use an sample input in MLeap frame format to test it.
# Pre-requisites
SQL Server Big Data Cluster - CTP 2.3 or later
Clone or download this sample on your computer to a folder called mleap (note if you have downloaded it to a different folder then you'l have to modify the folder location appropriately in the sample below)
SQL Server big data cluster - CTP 2.3 or later
Clone or download this sample on your computer to a folder called `mleap` (note if you have downloaded it to a different folder then you'll have to modify the folder location appropriately in the information below).
## Running the sample
## Deploying the Application
Login to the SQL Server Big Data Cluster using the command below by replacing the hostIf you are not familar you can refer to the document [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
### Connecting to SQL Server big data cluster
Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
```bash
mssqlctl login -e https://<ip-address-of-service-proxy-lb>:30777 -u <user-name> -p <password>
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
```
This example uses a Machine Learning Model that predicts the price per sq. ft given the location along wiht addtional paramters such as the type of dwelling. You can refer to the details and the example [here] (http://mleap-docs.combust.ml/mleap-serving/#load-model)
Deploy the app using the create command and pass the location of the spec file. Here the specification file is expected to be in the mleap folder. The specification file serves as the data required for deploying that app and it contains basic information about the app such as the name, version, inputs, outputs , replicas and poolsize you need for this app.
### Deploying the application
This example uses a Machine Learning Model that predicts the price per square foot of an Airbnb property based on various parameters such as the location and type of dwelling. [More details and information on the example are here](http://mleap-docs.combust.ml/mleap-serving/#load-model). The application you will be deploying as part of this sample is a Random Forest Model that was built in Spark and has been serialized as an MLeap bundle.
Deploy the app using the `create` command and pass the location of the spec file. In the example below, the spec file is expected to be in the `mleap` folder. The spec file serves as the data required for deploying the app and it contains basic information about the app such as the name, version, inputs, outputs, replicas and poolsize you need for this app.
Here is the spec file for this application
Here is the spec file for this application:
```yaml
name: mleap
@@ -29,28 +31,26 @@ replicas: 2
poolsize: 2
```
Deploy the application by running the following command, specifying the directory where your `spec.yaml` file is located.
```bash
> mssqlctl app create --spec ./mleap
mssqlctl app create --spec ./mleap
```
You can verify if the app has succesfully deployed by running the following command. The application you are deploying is Random Forest Model that was built in Spark and has been serialized in as an MLeap bundle.
### Testing the deployment
You can verify if the app has successfully deployed by running the following command:
```bash
> mssqlctl app list -n mleap -v v1
mssqlctl app list -n mleap -v v1
```
Once you see the app state as "Ready" you can proceed to the next step below.
Now that the app has been deployed you can test if the app works correctly by passing in a sample input that is available in the mleap folder. The deployed app is a RESTful webservice that is swagger compliant. For this sample we will show you how you can test this using the CLI.
The input parameter is a LeapFrame, a json file that describes the parameters and the values provided to the model for predicting the cost per square feet.
Note that the input paramter has a special character '@' to indicate that a json file is being passed. This command needs to be run within the mleap folder.
Now that the app has been deployed you can test if the app works correctly by passing in a sample input that is available in the mleap folder. The deployed app is a RESTful webservice that is [Swagger](swagger.io) compliant. For this sample we will show you how you can test this using the CLI.
To test the app, run the command below. The input parameter is a `LeapFrame`, a `json` file that describes the parameters and the values provided to the model for predicting the cost per square feet. Note that the input parameter has a special character '@' to indicate that a `json` file is being passed. This command needs to be run within the `mleap` folder.
```bash
> mssqlctl app run --name mleap --version v1 --inputs mleap-frame=@frame.json
mssqlctl app run --name mleap --version v1 --inputs mleap-frame=@frame.json
```
The result will be a json output that includes the prediction along with additional data.
The result will be a json output that includes the prediction along with additional data.
# Next Steps
You can learn how to train your model in Spark within SQL Server BDC and export to MLeap.[here] (https://docs.microsoft.com/en-us/sql/big-data-cluster/train-and-create-machinelearning-models-with-spark?view=sqlallproducts-allversions)
Please refer to [Train and Create machine learning models with Spark](https://docs.microsoft.com/en-us/sql/big-data-cluster/train-and-create-machinelearning-models-with-spark?view=sqlallproducts-allversions) on how to train your model in Spark within SQL Server big data clusters and export it to MLeap.