mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
Merge remote-tracking branch 'upstream/master'
This commit is contained in:
shivsood
@@ -0,0 +1,316 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Install Dependencies to deploy SQL Server 2019 Big Data Clusters\n",
|
||||
"----------------------------------------------------------------\n",
|
||||
"\n",
|
||||
"The following tools are most important for managing, connecting to, and\n",
|
||||
"querying the cluster:\n",
|
||||
"\n",
|
||||
"- **Azure CLI**\n",
|
||||
"- **kubectl**\n",
|
||||
"- **mssqlctl**\n",
|
||||
"\n",
|
||||
"**The cell below tries to install these dependencies for you. If you\n",
|
||||
"still experience any issues in executing this cell then please install\n",
|
||||
"from the following Installation links below.**\n",
|
||||
"\n",
|
||||
"<table>\n",
|
||||
"<colgroup>\n",
|
||||
"<col style=\"width: 27%\" />\n",
|
||||
"<col style=\"width: 24%\" />\n",
|
||||
"<col style=\"width: 24%\" />\n",
|
||||
"<col style=\"width: 24%\" />\n",
|
||||
"</colgroup>\n",
|
||||
"<thead>\n",
|
||||
"<tr class=\"header\">\n",
|
||||
"<th>Tool</th>\n",
|
||||
"<th>Required</th>\n",
|
||||
"<th>Description</th>\n",
|
||||
"<th>Installation</th>\n",
|
||||
"</tr>\n",
|
||||
"</thead>\n",
|
||||
"<tbody>\n",
|
||||
"<tr class=\"odd\">\n",
|
||||
"<td><strong>mssqlctl</strong></td>\n",
|
||||
"<td>Yes</td>\n",
|
||||
"<td>Command-line tool for installing and managing a big data cluster.</td>\n",
|
||||
"<td><a href=\"deploy-install-mssqlctl.md\">Install</a></td>\n",
|
||||
"</tr>\n",
|
||||
"<tr class=\"even\">\n",
|
||||
"<td><strong>kubectl</strong></td>\n",
|
||||
"<td>Yes</td>\n",
|
||||
"<td>Command-line tool for monitoring the underlying Kuberentes cluster (<a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/\">More info</a>).</td>\n",
|
||||
"<td><a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-powershell-from-psgallery\">Windows</a> | <a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-using-native-package-management\">Linux</a></td>\n",
|
||||
"</tr>\n",
|
||||
"<tr class=\"odd\">\n",
|
||||
"<td><strong>Azure CLI</strong></td>\n",
|
||||
"<td>For AKS</td>\n",
|
||||
"<td>Modern command-line interface for managing Azure services. Used with AKS big data cluster deployments (<a href=\"https://docs.microsoft.com/cli/azure/?view=azure-cli-latest\">More info</a>).</td>\n",
|
||||
"<td><a href=\"https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest\">Install</a></td>\n",
|
||||
"</tr>\n",
|
||||
"</tbody>\n",
|
||||
"</table>\n",
|
||||
"\n",
|
||||
"Steps\n",
|
||||
"-----\n",
|
||||
"\n",
|
||||
"### Provide overrides for any default installation parameters\n",
|
||||
"\n",
|
||||
"You don’t need to provide any installation parameters, we’ll provide a\n",
|
||||
"set of default which will provide a good experience.\n",
|
||||
"\n",
|
||||
"However, feel free to override any of the defaults provided below here:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"mssql_version=\"\"\n",
|
||||
"docker_registry=\"\"\n",
|
||||
"\n",
|
||||
"windows_azcli_url=\"\"\n",
|
||||
"windows_kubectl_url=\"\"\n",
|
||||
"\n",
|
||||
"azure_cli_use_force_install_option = None\n",
|
||||
"azure_cli_use_user_install_option = None\n",
|
||||
"\n",
|
||||
"mssqlctl_url=\"\" # url to download mssqlctl from (without ending filename)\n",
|
||||
"mssqlctl_use_force_install_option = None\n",
|
||||
"mssqlctl_use_user_install_option = None\n",
|
||||
"\n",
|
||||
"skip_mssqlctl_uninstall = None\n",
|
||||
"skip_mssqlctl_install = None"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Provide default installation parameters\n",
|
||||
"\n",
|
||||
"A default set of parameters to provide a good initial experience of SQL\n",
|
||||
"Server 2019 big data clusters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if mssql_version==\"\":\n",
|
||||
" mssql_version=\"ctp-2.5\"\n",
|
||||
"\n",
|
||||
"if windows_azcli_url==\"\":\n",
|
||||
" windows_azcli_url=\"https://aka.ms/installazurecliwindows\"\n",
|
||||
"\n",
|
||||
"if windows_kubectl_url==\"\":\n",
|
||||
" windows_kubectl_url=\"https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/windows/amd64/kubectl.exe\"\n",
|
||||
"\n",
|
||||
"if docker_repository==\"\":\n",
|
||||
" docker_repository=\"aris-p-release-candidate-gb\"\n",
|
||||
"\n",
|
||||
"if mssqlctl_url==\"\":\n",
|
||||
" mssqlctl_url=\"\"http://helsinki/browse/packages/python/{0}/mssqlctl/{1}\".format(docker_repository, 'requirements.txt')\n",
|
||||
"\n",
|
||||
"if azure_cli_use_force_install_option is None:\n",
|
||||
" azure_cli_use_force_install_option = False\n",
|
||||
"\n",
|
||||
"if azure_cli_use_user_install_option is None:\n",
|
||||
" azure_cli_use_user_install_option = True\n",
|
||||
"\n",
|
||||
"if mssqlctl_use_force_install_option is None:\n",
|
||||
" mssqlctl_use_force_install_option = False\n",
|
||||
"\n",
|
||||
"if mssqlctl_use_user_install_option is None:\n",
|
||||
" mssqlctl_use_user_install_option = True\n",
|
||||
"\n",
|
||||
"if skip_mssqlctl_uninstall is None:\n",
|
||||
" skip_mssqlctl_uninstall = True\n",
|
||||
"\n",
|
||||
"if skip_mssqlctl_install is None:\n",
|
||||
" skip_mssqlctl_install = False\n",
|
||||
"\n",
|
||||
"print('mssql_version = ' + mssql_version)\n",
|
||||
"print('docker_registry = ' + docker_registry)\n",
|
||||
"print('azure_cli_use_force_install_option = ' + str(azure_cli_use_force_install_option))\n",
|
||||
"print('azure_cli_use_user_install_option = ' + str(azure_cli_use_user_install_option))\n",
|
||||
"print('mssqlctl_url = ' + mssqlctl_url)\n",
|
||||
"print('mssqlctl_use_force_install_option = ' + str(mssqlctl_use_force_install_option))\n",
|
||||
"print('mssqlctl_use_user_install_option = ' + str(mssqlctl_use_user_install_option))\n",
|
||||
"print('skip_mssqlctl_uninstall = ' + str(skip_mssqlctl_uninstall))\n",
|
||||
"print('skip_mssqlctl_install = ' + str(skip_mssqlctl_install))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Azure CLI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"force_install_option='--force-install' if azure_cli_use_force_install_option else ''\n",
|
||||
"user_install_option='--user' if azure_cli_use_user_install_option else ''\n",
|
||||
"\n",
|
||||
"print(f'START: !{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\\n')\n",
|
||||
"\n",
|
||||
"!{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\n",
|
||||
"if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t!{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
"print(f'\\nSUCCESS: !{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Kubernetes CLI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import platform\n",
|
||||
"import requests\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if platform.system()==\"Darwin\":\n",
|
||||
" print(f'START: !brew update\\n')\n",
|
||||
"\n",
|
||||
" !brew update\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !brew update\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !brew update')\n",
|
||||
" print(f'START: !brew install kubernetes-cli\\n')\n",
|
||||
"\n",
|
||||
" !brew install kubernetes-cli\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !brew install kubernetes-cli\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !brew install kubernetes-cli')\n",
|
||||
"elif platform.system()==\"Windows\":\n",
|
||||
" print(f'START: !curl -LO {windows_kubectl_url}\\n')\n",
|
||||
"\n",
|
||||
" !curl -LO {windows_kubectl_url}\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !curl -LO {windows_kubectl_url}\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !curl -LO {windows_kubectl_url}')\n",
|
||||
"elif platform.system()==\"Linux\":\n",
|
||||
" print(f'START: !sudo apt-get update\\n')\n",
|
||||
"\n",
|
||||
" !sudo apt-get update\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !sudo apt-get update\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !sudo apt-get update')\n",
|
||||
" print(f'START: !sudo apt-get install -y kubectl\\n')\n",
|
||||
"\n",
|
||||
" !sudo apt-get install -y kubectl\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !sudo apt-get install -y kubectl\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !sudo apt-get install -y kubectl')\n",
|
||||
"else:\n",
|
||||
" raise SystemExit(\"Platform '{0}' is not recognized, must be 'Darwin', 'Windows' or 'Linux'\".format(platform.system()))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Uninstall MSSQLCTL CLI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if not skip_mssqlctl_uninstall:\n",
|
||||
" print(f'START: !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\\n')\n",
|
||||
"\n",
|
||||
" !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install MSSQLCTL CLI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"force_install_option='--force-install' if mssqlctl_use_force_install_option else ''\n",
|
||||
"user_install_option='--user' if mssqlctl_use_user_install_option else ''\n",
|
||||
"\n",
|
||||
"if not skip_mssqlctl_install:\n",
|
||||
" print(f'START: !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\\n')\n",
|
||||
"\n",
|
||||
" !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\n",
|
||||
" if _exit_code != 0:\n",
|
||||
" raise SystemExit('Shell command:\\n\\n\\t !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
|
||||
"\n",
|
||||
" print(f'\\nSUCCESS: !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki')"
|
||||
]
|
||||
}
|
||||
],
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5,
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"azdata": {
|
||||
"test": {
|
||||
"strategy": "Sequential",
|
||||
"dri": false,
|
||||
"ci": false,
|
||||
"gci": false
|
||||
},
|
||||
"publish": {
|
||||
"access": {
|
||||
"current": "Internal",
|
||||
"goal": "Public"
|
||||
},
|
||||
"state": "Draft"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,12 +9,12 @@ Installation instructions for SQL Server 2019 big data clusters can be found [he
|
||||
## Executing the sample scripts
|
||||
The scripts should be executed in a specific order to test the various features. Execute the scripts from each folder in below order:
|
||||
|
||||
1. __[spark](spark/)__
|
||||
1. __[spark/dataloading/transform-csv-files.sql](spark/dataloading/transform-csv-files.sql)__
|
||||
1. __[data-virtualization/storage-pool](data-virtualization/storage-pool)__
|
||||
1. __[data-virtualization/oracle](data-virtualization/oracle)__
|
||||
1. __[data-pool](data-pool/)__
|
||||
1. __[machine-learning/r](machine-learning/r)__
|
||||
1. __[machine-learning/python](machine-learning/python)__
|
||||
1. __[machine-learning/sql/r](machine-learning/sql/r)__
|
||||
1. __[machine-learning/sql/python](machine-learning/sql/python)__
|
||||
|
||||
## __[data-pool](data-pool/)__
|
||||
|
||||
|
||||
@@ -41,10 +41,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `roll-dice.R` files are located:
|
||||
```bash
|
||||
|
||||
@@ -32,10 +32,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Replace `[SA_PASSWORD]` in the `spec.yaml` file with the password for SQL user `sa`.
|
||||
4. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `back-up-db.dtsx` files are located:
|
||||
|
||||
@@ -42,10 +42,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `add.py` files are located:
|
||||
```bash
|
||||
@@ -116,7 +116,7 @@ To run this sample, you need the following prerequisites.
|
||||
|
||||
> *Optional*: If you want, you can open the URL for the `swagger` that was returned when you ran `mssqlctl app describe --name addpy --version [version]` in your browser. You will have to log in with the same credentials you used for `mssqlctl login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io). You will see that the web service exposes the `run` method.
|
||||
|
||||
You can use your favorite tool to call the `run` method (`https://[IP]:[PORT]/api/app/addpy/[version]/run`), passing in the parameters in the body of your POST request as json. In this example we will use [Postman](https://www.getpostman.com/). Before making the call, you will need to set the `Authorization` to `Bearer Token` and paste in the token you retrieved earlier. This will set a header on your request. See the screenshot below.
|
||||
You can use your favorite tool to call the `run` method (`https://[IP]:30778/api/app/addpy/[version]/run`), passing in the parameters in the body of your POST request as json. In this example we will use [Postman](https://www.getpostman.com/). Before making the call, you will need to set the `Authorization` to `Bearer Token` and paste in the token you retrieved earlier. This will set a header on your request. See the screenshot below.
|
||||
|
||||
Next, in the requests body, pass in the parameters to the app you are calling and set the `content-type` to `application/json`:
|
||||
|
||||
|
||||
@@ -41,10 +41,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `magic8ball.py` files are located:
|
||||
```bash
|
||||
|
||||
@@ -38,10 +38,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. This example uses a TensorFlow Machine Learning Model that uses public US Census data predict income. [More details and information on the example are here](https://docs.microsoft.com/en-us/sql/big-data-cluster/train-and-create-machinelearning-models-with-spark?view=sqlallproducts-allversions). The application you will be deploying as part of this sample is a Random Forest Model that was built in Spark and has been [serialized as an MLeap bundle](https://docs.microsoft.com/en-us/sql/big-data-cluster/export-model-with-spark-mleap?view=sqlallproducts-allversions).
|
||||
|
||||
|
||||
@@ -42,10 +42,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml`, `sentiment.rds` and `sentiment.R` files are located:
|
||||
```bash
|
||||
|
||||
@@ -42,10 +42,10 @@ To run this sample, you need the following prerequisites.
|
||||
## Run this sample
|
||||
|
||||
1. Clone or download this sample on your computer.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
|
||||
|
||||
```bash
|
||||
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
|
||||
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
|
||||
```
|
||||
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `sum_of_squares.R` files are located:
|
||||
```bash
|
||||
|
||||
@@ -36,7 +36,7 @@ if NOT EXIST tpcxbb_1gb.bak (
|
||||
|
||||
REM Copy the backup file, restore the database, create necessary objects and data file
|
||||
echo Copying sales database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
%DEBUG% kubectl cp tpcxbb_1gb.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
|
||||
|
||||
REM Download and copy the sample backup files
|
||||
if /i %AW_WWI_SAMPLES% EQU --install-extra-samples (
|
||||
@@ -45,28 +45,28 @@ if /i %AW_WWI_SAMPLES% EQU --install-extra-samples (
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorks2016_EXT.bak" -o AdventureWorks2016_EXT.bak
|
||||
)
|
||||
echo Copying AdventureWorks2016_EXT database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp AdventureWorks2016_EXT.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
%DEBUG% kubectl cp AdventureWorks2016_EXT.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
|
||||
|
||||
if NOT EXIST AdventureWorksDW2016_EXT.bak (
|
||||
echo Downloading AdventureWorksDW2016_EXT sample database backup file...
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/AdventureWorksDW2016_EXT.bak" -o AdventureWorksDW2016_EXT.bak
|
||||
)
|
||||
echo Copying AdventureWorksDW2016_EXT database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp AdventureWorksDW2016_EXT.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
%DEBUG% kubectl cp AdventureWorksDW2016_EXT.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
|
||||
|
||||
if NOT EXIST WideWorldImporters-Full.bak (
|
||||
echo Downloading WideWorldImporters sample database backup file...
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak" -o WideWorldImporters-Full.bak
|
||||
)
|
||||
echo Copying WideWorldImporters-Full database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp WideWorldImporters-Full.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
%DEBUG% kubectl cp WideWorldImporters-Full.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
|
||||
|
||||
if NOT EXIST WideWorldImportersDW-Full.bak (
|
||||
echo Downloading WideWorldImportersDW sample database backup file...
|
||||
%DEBUG% curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImportersDW-Full.bak" -o WideWorldImportersDW-Full.bak
|
||||
)
|
||||
echo Copying WideWorldImportersDW-Full database backup file to SQL Master instance...
|
||||
%DEBUG% kubectl cp WideWorldImportersDW-Full.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
|
||||
%DEBUG% kubectl cp WideWorldImportersDW-Full.bak %CLUSTER_NAMESPACE%/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || goto exit
|
||||
)
|
||||
|
||||
echo Configuring sample database(s)...
|
||||
@@ -74,7 +74,7 @@ echo Configuring sample database(s)...
|
||||
|
||||
REM remove files copied into the pod:
|
||||
echo Removing database backup files...
|
||||
kubectl exec mssql-master-pool-0 -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
kubectl exec mssql-master-pool-0 -n %CLUSTER_NAMESPACE% -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
|
||||
for %%F in (web_clickstreams inventory customer) do (
|
||||
if NOT EXIST %%F.csv (
|
||||
|
||||
@@ -44,7 +44,7 @@ then
|
||||
fi
|
||||
|
||||
echo Copying database backup file...
|
||||
$DEBUG kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1)
|
||||
$DEBUG kubectl cp tpcxbb_1gb.bak $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
# $DEBUG rm tpcxbb_1gb.bak
|
||||
|
||||
if [ $AW_WWI_SAMPLES == --install-extra-samples ]
|
||||
@@ -57,7 +57,7 @@ then
|
||||
$DEBUG curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/adventureworks/$file" -o $file
|
||||
fi
|
||||
echo Copying $file database backup file to SQL Master instance...
|
||||
$DEBUG kubectl cp $file mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1)
|
||||
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
done
|
||||
|
||||
|
||||
@@ -69,7 +69,7 @@ then
|
||||
$DEBUG curl -L -G "https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/$file" -o $file
|
||||
fi
|
||||
echo Copying $file database backup file to SQL Master instance...
|
||||
$DEBUG kubectl cp $file mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1)
|
||||
$DEBUG kubectl cp $file $CLUSTER_NAMESPACE/mssql-master-pool-0:/var/opt/mssql/data -c mssql-server || (echo $ERROR_MESSAGE && exit 1)
|
||||
done
|
||||
fi
|
||||
|
||||
@@ -80,7 +80,7 @@ $DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -I -b < "$S
|
||||
|
||||
# remove files copied into the pod:
|
||||
echo Removing database backup files...
|
||||
kubectl exec mssql-master-pool-0 -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
kubectl exec mssql-master-pool-0 -n $CLUSTER_NAMESPACE -c mssql-server -i -t -- bash -c "rm -rvf /var/opt/mssql/data/*.bak"
|
||||
|
||||
for table in web_clickstreams inventory customer
|
||||
do
|
||||
|
||||
@@ -80,14 +80,8 @@ BEGIN
|
||||
WITH (LOCATION = 'sqldatapool://service-mssql-controller:8080/datapools/default');
|
||||
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
BEGIN
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP2.3'
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-mssql-controller:8080');
|
||||
ELSE IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
END
|
||||
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for Data Pool inside a SQL big data cluster
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlDataPool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlDataPool
|
||||
WITH (LOCATION = 'sqldatapool://service-mssql-controller:8080/datapools/default');
|
||||
|
||||
-- Create external table in a data pool in SQL Server 2019 big data cluster.
|
||||
-- The SqlDataPool data source is a special data source that is available in
|
||||
-- any new database in SQL Master instance. This is used to reference the
|
||||
@@ -25,30 +31,13 @@ GO
|
||||
-- Insert results of a SELECT statement into the external table created on the data pool.
|
||||
-- Store summary results for quick access instead of going to the source tables always.
|
||||
--
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP2.3'
|
||||
BEGIN
|
||||
DECLARE @db_name SYSNAME = 'sales'
|
||||
DECLARE @schema_name SYSNAME = 'dbo'
|
||||
DECLARE @table_name SYSNAME = 'web_clickstream_clicks_data_pool'
|
||||
DECLARE @query NVARCHAR(MAX) = '
|
||||
SELECT wcs_user_sk, i_category_id, COUNT_BIG(*) as clicks
|
||||
FROM sales.dbo.web_clickstreams
|
||||
INNER JOIN sales.dbo.item it ON (wcs_item_sk = i_item_sk
|
||||
AND wcs_user_sk IS NOT NULL)
|
||||
GROUP BY wcs_user_sk, i_category_id
|
||||
HAVING COUNT_BIG(*) > 100;
|
||||
'
|
||||
EXEC model..sp_data_pool_table_insert_data @db_name, @schema_name, @table_name, @query
|
||||
END;
|
||||
|
||||
IF SERVERPROPERTY('ProductLevel') = 'CTP2.4'
|
||||
INSERT INTO web_clickstream_clicks_data_pool
|
||||
SELECT wcs_user_sk, i_category_id, COUNT_BIG(*) as clicks
|
||||
FROM sales.dbo.web_clickstreams_hdfs_parquet
|
||||
INNER JOIN sales.dbo.item it ON (wcs_item_sk = i_item_sk
|
||||
AND wcs_user_sk IS NOT NULL)
|
||||
GROUP BY wcs_user_sk, i_category_id
|
||||
HAVING COUNT_BIG(*) > 100;
|
||||
INSERT INTO web_clickstream_clicks_data_pool
|
||||
SELECT wcs_user_sk, i_category_id, COUNT_BIG(*) as clicks
|
||||
FROM sales.dbo.web_clickstreams_hdfs_parquet
|
||||
INNER JOIN sales.dbo.item it ON (wcs_item_sk = i_item_sk
|
||||
AND wcs_user_sk IS NOT NULL)
|
||||
GROUP BY wcs_user_sk, i_category_id
|
||||
HAVING COUNT_BIG(*) > 100;
|
||||
GO
|
||||
|
||||
-- Query data inserted into the data pool table
|
||||
|
||||
+15
@@ -1,6 +1,8 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Enable option to allow INSERT against external table defined on HADOOP data source
|
||||
--
|
||||
DECLARE @config_option nvarchar(100) = 'allow polybase export';
|
||||
IF NOT EXISTS(SELECT * FROM sys.configurations WHERE name = @config_option and value_in_use = 1)
|
||||
BEGIN
|
||||
@@ -9,6 +11,19 @@ BEGIN
|
||||
END;
|
||||
GO
|
||||
|
||||
-- Create data source for HDFS inside SQL big data cluster using the HADOOP type.
|
||||
-- The HADOOP data source type was introduced in SQL Server 2016 to query data in
|
||||
-- Hadoop clusters and relies on Java Hadoop client libraries and Map/Reduce for query
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
|
||||
);
|
||||
|
||||
-- Create file format for RCFILE with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'RCFILE')
|
||||
|
||||
+13
@@ -1,6 +1,19 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create data source for HDFS inside SQL big data cluster using the HADOOP type.
|
||||
-- The HADOOP data source type was introduced in SQL Server 2016 to query data in
|
||||
-- Hadoop clusters and relies on Java Hadoop client libraries and Map/Reduce for query
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
|
||||
);
|
||||
|
||||
-- Create file format for orc file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'orc_file')
|
||||
|
||||
+13
@@ -1,6 +1,19 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create data source for HDFS inside SQL big data cluster using the HADOOP type.
|
||||
-- The HADOOP data source type was introduced in SQL Server 2016 to query data in
|
||||
-- Hadoop clusters and relies on Java Hadoop client libraries and Map/Reduce for query
|
||||
-- execution.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'HadoopData')
|
||||
CREATE EXTERNAL DATA SOURCE HadoopData
|
||||
WITH(
|
||||
TYPE=HADOOP,
|
||||
LOCATION='hdfs://mssql-master-pool-0.service-master-pool:9000/',
|
||||
RESOURCE_MANAGER_LOCATION='mssql-master-pool-0.service-master-pool:8032'
|
||||
);
|
||||
|
||||
-- Create file format for orc file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'orc_file')
|
||||
|
||||
+6
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for HDFS inside SQ: big data cluster.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
|
||||
-- Create file format for CSV separated file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file')
|
||||
|
||||
+6
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for HDFS inside SQ: big data cluster.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
|
||||
-- Create file format for parquet file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file')
|
||||
|
||||
+6
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for HDFS inside SQ: big data cluster.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
|
||||
-- Create file format for tab separated file with appropriate properties.
|
||||
--
|
||||
CREATE EXTERNAL FILE FORMAT tsv_file
|
||||
|
||||
+6
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for HDFS inside SQ: big data cluster.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
|
||||
-- Create file format for CSV file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'csv_file')
|
||||
|
||||
+6
@@ -1,6 +1,12 @@
|
||||
USE sales
|
||||
GO
|
||||
|
||||
-- Create external data source for HDFS inside SQ: big data cluster.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePool')
|
||||
CREATE EXTERNAL DATA SOURCE SqlStoragePool
|
||||
WITH (LOCATION = 'sqlhdfs://service-master-pool:50070');
|
||||
|
||||
-- Create file format for parquet file with appropriate properties.
|
||||
--
|
||||
IF NOT EXISTS(SELECT * FROM sys.external_file_formats WHERE name = 'parquet_file')
|
||||
|
||||
@@ -10,3 +10,7 @@ Use the scripts in the **kubeadm** folder to deploy Kubernetes over multiple Lin
|
||||
## Deploy a SQL Server big data cluster on Azure Kubernetes Service (AKS)
|
||||
|
||||
Using the sample Python script in **aks** folder, you will deploy a Kubernetes cluster in Azure using AKS and a SQL Server big data cluster using on top of it.
|
||||
|
||||
## Push SQL Server big data cluster images to your own private Docker repository
|
||||
|
||||
Using the sample Python script in **offline** folder, you will push the necessary images required for the deployment to your own repository.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Push SQL Server big data cluster Docker images to your own private Docker repository
|
||||
|
||||
Big data clusters must have access to a Docker repository from which to pull container images. If you need to deploy to an environment that can't access the registry provided by Microsoft, you need to first push necessary images to your own private repository that the environment can access. This repository is then used as the target for a new deployment.
|
||||
|
||||
Using this sample Python script, you can pull all images from the Microsoft repository to your local Docker environment, and then push them to your own private repo.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine 1.8+ on any supported Linux distribution or Docker for Mac/Windows. For more information, see [Install Docker](https://docs.docker.com/engine/installation/).
|
||||
- Running the script will require: Python minimum version 3.0
|
||||
|
||||
## Instructions
|
||||
|
||||
Run the script using:
|
||||
```
|
||||
python push-bdc-images-to-custom-private-repo.py
|
||||
```
|
||||
|
||||
>**Note**
|
||||
>
|
||||
>If you have both python3 and python2 on your client machine and in the path, you will have to run the command using python3:
|
||||
>```
|
||||
>python3 push-bdc-images-to-custom-private-repo.py
|
||||
>```
|
||||
|
||||
When prompted, provide your input for:
|
||||
- Docker registry, repository and credentials to access Microsoft private registry where the images will be pulled from (source)
|
||||
- Docker registry, repository and credentials to access your private registry where the images will be pushed to (target)
|
||||
|
||||
## Deploy with from your private repository
|
||||
|
||||
To deploy from your private repository, use the steps described in the [deployment guide](deployment-guidance.md), but customize the following environment variables to match your private Docker repository.
|
||||
|
||||
- **DOCKER_REGISTRY**
|
||||
- **DOCKER_REPOSITORY**
|
||||
- **DOCKER_USERNAME**
|
||||
- **DOCKER_PASSWORD**
|
||||
- **DOCKER_EMAIL**
|
||||
- **DOCKER_IMAGE_TAG**
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
# requires installation of Docker: https://docs.docker.com/install/
|
||||
|
||||
from subprocess import check_output, CalledProcessError, STDOUT, Popen, PIPE
|
||||
import os
|
||||
import getpass
|
||||
|
||||
def execute_cmd (cmd):
|
||||
if os.name=="nt":
|
||||
process = Popen(cmd.split(),stdin=PIPE, shell=True)
|
||||
else:
|
||||
process = Popen(cmd.split(),stdin=PIPE)
|
||||
stdout, stderr = process.communicate()
|
||||
if (stderr is not None):
|
||||
raise Exception(stderr)
|
||||
|
||||
SOURCE_DOCKER_REGISTRY = input("Provide Docker registry source - press ENTER for using `private-repo.microsoft.com`:") or "private-repo.microsoft.com"
|
||||
SOURCE_DOCKER_REPOSITORY = input("Provide Docker repository source - press ENTER for using `mssql-private-preview`:") or "mssql-private-preview"
|
||||
SOURCE_DOCKER_USERNAME = input("Provide Docker username for the source registry:")
|
||||
SOURCE_DOCKER_PASSWORD=getpass.getpass("Provide Docker password for the source registry:")
|
||||
SOURCE_DOCKER_TAG = input("Provide Docker tag for the images at the source: ") or "latest"
|
||||
|
||||
TARGET_DOCKER_REGISTRY = input("Provide Docker registry target:")
|
||||
TARGET_DOCKER_REPOSITORY = input("Provide Docker repository target:")
|
||||
TARGET_DOCKER_USERNAME = input("Provide Docker username for the target registry:")
|
||||
TARGET_DOCKER_PASSWORD = getpass.getpass("Provide Docker password for the target registry:")
|
||||
TARGET_DOCKER_TAG = input("Provide Docker tag for the images at the target: ") or "latest"
|
||||
|
||||
images = [ 'mssql-appdeploy-init',
|
||||
'mssql-monitor-fluentbit',
|
||||
'mssql-monitor-collectd',
|
||||
'mssql-server-data',
|
||||
'mssql-hadoop',
|
||||
'mssql-java',
|
||||
'mssql-mlservices-pythonserver',
|
||||
'mssql-mlservices-rserver',
|
||||
'mssql-monitor-elasticsearch',
|
||||
'mssql-monitor-influxdb',
|
||||
'mssql-security-knox',
|
||||
'mssql-mlserver-r-runtime',
|
||||
'mssql-mlserver-py-runtime',
|
||||
'mssql-controller',
|
||||
'mssql-portal',
|
||||
'mssql-server-controller',
|
||||
'mssql-monitor-grafana',
|
||||
'mssql-monitor-kibana',
|
||||
'mssql-service-proxy',
|
||||
'mssql-app-service-proxy',
|
||||
'mssql-ssis-app-runtime',
|
||||
'mssql-monitor-telegraf']
|
||||
|
||||
print("Execute docker login to source registry: " + SOURCE_DOCKER_REGISTRY)
|
||||
cmd = "docker login " + SOURCE_DOCKER_REGISTRY + " -u " + SOURCE_DOCKER_USERNAME + " -p " + SOURCE_DOCKER_PASSWORD
|
||||
execute_cmd(cmd)
|
||||
print("")
|
||||
|
||||
|
||||
print("Pulling images from source repository: " + SOURCE_DOCKER_REGISTRY + "/" + SOURCE_DOCKER_REPOSITORY)
|
||||
cmd = ""
|
||||
for image in images:
|
||||
cmd += "docker pull " + SOURCE_DOCKER_REGISTRY + "/" + SOURCE_DOCKER_REPOSITORY + "/" + image + ":" + SOURCE_DOCKER_TAG + " & "
|
||||
cmd = cmd[:len(cmd)-3]
|
||||
execute_cmd(cmd)
|
||||
|
||||
print("Execute docker login to target registry:" + TARGET_DOCKER_REGISTRY)
|
||||
cmd = "docker login " + TARGET_DOCKER_REGISTRY + " -u " + TARGET_DOCKER_USERNAME + " -p " + TARGET_DOCKER_PASSWORD
|
||||
execute_cmd(cmd)
|
||||
print("")
|
||||
|
||||
print("Tagging local images...")
|
||||
cmd = ""
|
||||
for image in images:
|
||||
cmd += "docker tag " + SOURCE_DOCKER_REGISTRY + "/" + SOURCE_DOCKER_REPOSITORY + "/" + image + ":" + SOURCE_DOCKER_TAG + " " + TARGET_DOCKER_REGISTRY + "/" + TARGET_DOCKER_REPOSITORY + "/" + image + ":" + TARGET_DOCKER_TAG + " & "
|
||||
cmd = cmd[:len(cmd)-3]
|
||||
execute_cmd(cmd)
|
||||
|
||||
print("Push images to target Docker repository: " + TARGET_DOCKER_REGISTRY + "/" + TARGET_DOCKER_REPOSITORY)
|
||||
cmd = ""
|
||||
for image in images:
|
||||
cmd += "docker push " + TARGET_DOCKER_REGISTRY + "/" + TARGET_DOCKER_REPOSITORY + "/" + image + ":" + TARGET_DOCKER_TAG + " & "
|
||||
cmd = cmd[:len(cmd)-3]
|
||||
execute_cmd(cmd)
|
||||
|
||||
print("Images are now pushed to the target repository.")
|
||||
cmd = "docker images"
|
||||
execute_cmd(cmd)
|
||||
|
||||
+213
-547
@@ -1,549 +1,215 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Powerplant Output Prediction\n",
|
||||
"- This notebook is based on the power plant output prediction example presented in H2O’s [blog post on H2O AutoML in Spark](https://www.h2o.ai/blog/h2os-automl-in-spark/).\n",
|
||||
"- Run this notebook in Azure Data Studio connected to a SQL Server 2019 Big Data Cluster by following the instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/notebooks-guidance?view=sqlallproducts-allversions)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Spark Configuration\n",
|
||||
"- We can control the Spark Driver and Executor memory, cores, and number of executors per pod using the “%%configure” cell magic\n",
|
||||
"- Additional configuration settings are listed at the end of this notebook\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"Current session configs: <tt>{'executorMemory': '4g', 'driverMemory': '4g', 'executorCores': 2, 'driverCores': 2, 'numExecutors': 2, 'kind': 'pyspark3'}</tt><br>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"name": "pyspark3kernel",
|
||||
"display_name": "PySpark3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "pyspark3",
|
||||
"mimetype": "text/x-python",
|
||||
"codemirror_mode": {
|
||||
"name": "python",
|
||||
"version": 3
|
||||
},
|
||||
"pygments_lexer": "python3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"No active sessions."
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%%configure -f\n",
|
||||
"{\n",
|
||||
" \"executorMemory\": \"4g\",\n",
|
||||
" \"driverMemory\": \"4g\",\n",
|
||||
" \"executorCores\": 2,\n",
|
||||
" \"driverCores\": 2,\n",
|
||||
" \"numExecutors\": 2\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Install H2O\n",
|
||||
"- This cell downloads the h2o_pysparkling_2.3 python package and installs it on the pod where the Spark driver is currently running, if it is not already installed. Propagating the software to additional pods is handled automatically once we launch H2O.\n",
|
||||
"- For an enterprise scenario where we cannot reach out to the PyPi repository on the Internet, pip3 can be pointed to a local copy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Starting Spark application\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table>\n",
|
||||
"<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>24</td><td>application_1543381571657_0025</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.193.17.116:30443/gateway/default/yarn/proxy/application_1543381571657_0025/\">Link</a></td><td><a target=\"_blank\" href=\"http://mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local:8042/node/containerlogs/container_1543381571657_0025_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"SparkSession available as 'spark'.\n",
|
||||
"Collecting h2o_pysparkling_2.3\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): tabulate in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): six in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): colorama>=0.3.8 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): pyspark<=2.3.2,>=2.3.0 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): idna<2.8,>=2.5 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): py4j==0.10.7 in /usr/local/lib/python3.5/dist-packages (from pyspark<=2.3.2,>=2.3.0->h2o_pysparkling_2.3)\n",
|
||||
"Installing collected packages: h2o-pysparkling-2.3\n",
|
||||
"Successfully installed h2o-pysparkling-2.3-2.3.18\n",
|
||||
"You are using pip version 8.1.1, however version 18.1 is available.\n",
|
||||
"You should consider upgrading via the 'pip install --upgrade pip' command."
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import subprocess\n",
|
||||
"\n",
|
||||
"# Install H2O PySparkling\n",
|
||||
"stdout = subprocess.check_output(\n",
|
||||
" \"pip3 install h2o_pysparkling_2.3\",\n",
|
||||
" stderr=subprocess.STDOUT,\n",
|
||||
" shell=True).decode(\"utf-8\")\n",
|
||||
"print(stdout)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Download and copy data to HDFS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"ls: `/tmp/powerplant_output.csv': No such file or directory\n",
|
||||
"--2018-12-06 19:29:35-- https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/powerplant_output.csv\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 308777 (302K) [text/plain]\n",
|
||||
"Saving to: 'powerplant_output.csv'\n",
|
||||
"\n",
|
||||
" 0K .......... .......... .......... .......... .......... 16% 574K 0s\n",
|
||||
" 50K .......... .......... .......... .......... .......... 33% 1.14M 0s\n",
|
||||
" 100K .......... .......... .......... .......... .......... 49% 30.9M 0s\n",
|
||||
" 150K .......... .......... .......... .......... .......... 66% 1.16M 0s\n",
|
||||
" 200K .......... .......... .......... .......... .......... 82% 42.7M 0s\n",
|
||||
" 250K .......... .......... .......... .......... .......... 99% 77.2M 0s\n",
|
||||
" 300K . 100% 2937G=0.2s\n",
|
||||
"\n",
|
||||
"2018-12-06 19:29:36 (1.68 MB/s) - 'powerplant_output.csv' saved [308777/308777]"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"dataFileName = \"powerplant_output.csv\"\n",
|
||||
"dataFileUrl = \"https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/\" + dataFileName\n",
|
||||
"\n",
|
||||
"# Download data file and copy to HDFS, if not already there\n",
|
||||
"cmd = 'hdfs dfs -ls /tmp/' + dataFileName + ' || ' \\\n",
|
||||
" '(wget ' + dataFileUrl + ' && ' \\\n",
|
||||
" 'hdfs dfs -copyFromLocal ' + dataFileName + ' /tmp && ' \\\n",
|
||||
" 'rm ' + dataFileName + ')'\n",
|
||||
"\n",
|
||||
"stdout = subprocess.check_output(\n",
|
||||
" cmd,\n",
|
||||
" stderr=subprocess.STDOUT,\n",
|
||||
" shell=True).decode(\"utf-8\")\n",
|
||||
"print(stdout)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Start H2O engine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Connecting to H2O server at http://10.244.0.66:54323... successful.\n",
|
||||
"-------------------------- ---------------------------------------------------\n",
|
||||
"H2O cluster uptime: 13 secs\n",
|
||||
"H2O cluster timezone: Etc/UTC\n",
|
||||
"H2O data parsing timezone: UTC\n",
|
||||
"H2O cluster version: 3.22.0.2\n",
|
||||
"H2O cluster version age: 14 days, 12 hours and 24 minutes\n",
|
||||
"H2O cluster name: sparkling-water-root_application_1543381571657_0021\n",
|
||||
"H2O cluster total nodes: 2\n",
|
||||
"H2O cluster free memory: 6.928 Gb\n",
|
||||
"H2O cluster total cores: 32\n",
|
||||
"H2O cluster allowed cores: 4\n",
|
||||
"H2O cluster status: accepting new members, healthy\n",
|
||||
"H2O connection url: http://10.244.0.66:54323\n",
|
||||
"H2O connection proxy:\n",
|
||||
"H2O internal security: False\n",
|
||||
"H2O API Extensions: XGBoost, Algos, AutoML, Core V3, Core V4\n",
|
||||
"Python version: 3.5.2 final\n",
|
||||
"-------------------------- ---------------------------------------------------\n",
|
||||
"\n",
|
||||
"Sparkling Water Context:\n",
|
||||
" * H2O name: sparkling-water-root_application_1543381571657_0021\n",
|
||||
" * cluster size: 2\n",
|
||||
" * list of used nodes:\n",
|
||||
" (executorId, host, port)\n",
|
||||
" ------------------------\n",
|
||||
" (1,mssql-storage-pool-default-1.service-storage-pool-default.test.svc.cluster.local,54321)\n",
|
||||
" (2,mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local,54321)\n",
|
||||
" ------------------------\n",
|
||||
"\n",
|
||||
" Open H2O Flow in browser: http://10.244.0.66:54323 (CMD + click in Mac OSX)\n",
|
||||
"\n",
|
||||
" \n",
|
||||
" * Yarn App ID of Spark application: application_1543381571657_0021"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pysparkling import H2OContext\n",
|
||||
"\n",
|
||||
"hc = H2OContext.getOrCreate(spark)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"mssql-storage-pool-default-0"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Print the hostname of the pod where the driver is running\n",
|
||||
"stdout = subprocess.check_output(\n",
|
||||
" \"hostname\",\n",
|
||||
" stderr=subprocess.STDOUT,\n",
|
||||
" shell=True).decode(\"utf-8\")\n",
|
||||
"print(stdout)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Read and split data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"powerplant_df = spark.read.option(\"inferSchema\", \"true\").csv(\"/tmp/powerplant_output.csv\", header=True)\n",
|
||||
"\n",
|
||||
"splits = powerplant_df.randomSplit([0.8, 0.2], seed=1)\n",
|
||||
"train = splits[0]\n",
|
||||
"for_predictions = splits[1]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Training and Prediction\n",
|
||||
"- Fit AutoML model on training data\n",
|
||||
"- Generate predictions on \"for_predictions\" data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"|TemperatureCelcius|ExhaustVacuumHg|AmbientPressureMillibar|RelativeHumidity|HourlyEnergyOutputMW| prediction_output|\n",
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"| 10.01| 41.17| 1018.78| 86.84| 479.4|[477.07336140008584]|\n",
|
||||
"| 10.02| 39.66| 1016.34| 79.98| 480.05| [476.9418514217447]|\n",
|
||||
"| 10.03| 43.13| 1014.85| 70.09| 482.16| [476.1059605448468]|\n",
|
||||
"| 10.04| 41.62| 1013.36| 95.17| 463.87| [470.5313939263834]|\n",
|
||||
"| 10.05| 41.58| 1021.35| 95.19| 469.03| [469.2805814291723]|\n",
|
||||
"| 10.06| 34.69| 1027.9| 71.73| 477.68| [477.8063480544951]|\n",
|
||||
"| 10.08| 37.92| 1010.47| 66.37| 474.63| [475.4396167315097]|\n",
|
||||
"| 10.08| 41.16| 1023.14| 96.03| 469.17|[471.84298416925895]|\n",
|
||||
"| 10.09| 41.01| 1019.89| 96.55| 471.15| [471.4895037099167]|\n",
|
||||
"| 10.1| 41.4| 1024.29| 85.94| 474.28|[477.99005354475656]|\n",
|
||||
"| 10.11| 39.35| 1015.19| 90.74| 479.83| [478.0068968374228]|\n",
|
||||
"| 10.11| 39.72| 1019.1| 69.68| 476.8|[474.19764922516293]|\n",
|
||||
"| 10.11| 42.49| 1010.22| 82.11| 483.56|[477.02467192323263]|\n",
|
||||
"| 10.12| 41.55| 1005.78| 62.34| 475.46|[475.25159731408183]|\n",
|
||||
"| 10.12| 41.78| 1013.43| 73.47| 477.67| [475.3766641350098]|\n",
|
||||
"| 10.13| 39.18| 1024.09| 85.48| 479.42|[478.11614912148826]|\n",
|
||||
"| 10.15| 39.22| 1020.09| 68.75| 474.87| [477.1365182805492]|\n",
|
||||
"| 10.15| 41.46| 1019.78| 83.56| 481.31|[479.42546663321224]|\n",
|
||||
"| 10.15| 43.41| 1018.4| 82.07| 473.43|[476.34953390951085]|\n",
|
||||
"| 10.16| 39.3| 1019.71| 81.21| 480.74|[476.95324226094823]|\n",
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"only showing top 20 rows"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pyspark.ml.feature import SQLTransformer\n",
|
||||
"from pysparkling.ml import H2OAutoML\n",
|
||||
"from pyspark.ml import Pipeline\n",
|
||||
"\n",
|
||||
"temperatureTransformer = SQLTransformer(statement=\"SELECT * FROM __THIS__ WHERE TemperatureCelcius > 10\")\n",
|
||||
"\n",
|
||||
"automlEstimator = H2OAutoML(maxModels=2, predictionCol=\"HourlyEnergyOutputMW\", seed=1)\n",
|
||||
"\n",
|
||||
"pipeline = Pipeline(stages=[temperatureTransformer, automlEstimator])\n",
|
||||
"\n",
|
||||
"# Fit AutoML model\n",
|
||||
"model = pipeline.fit(train)\n",
|
||||
"\n",
|
||||
"# Generate predictions using fitted model\n",
|
||||
"predicted = model.transform(for_predictions)\n",
|
||||
"\n",
|
||||
"predicted.show()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Display the leaderboard metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n",
|
||||
"|model_id |mean_residual_deviance|rmse |mse |mae |rmsle |\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n",
|
||||
"|StackedEnsemble_BestOfFamily_AutoML_20181206_193040|11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n",
|
||||
"|StackedEnsemble_AllModels_AutoML_20181206_193040 |11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n",
|
||||
"|DRF_1_AutoML_20181206_193040 |11.349494687812056 |3.3689011098297406|11.349494687812056|2.5426288374345605|0.007472705853530634|\n",
|
||||
"|XRT_1_AutoML_20181206_193040 |11.464865035526516 |3.3859806608317355|11.464865035526516|2.545269555169042 |0.007510102104757881|\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"automlEstimator.leaderboard().show(truncate=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Evaluate predictions on held-out data\n",
|
||||
"- As expected, we find that the mean absolute error (mae) on the for_predictions data is similar to the leaderboard mae"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Mean absolute error: 2.3167231443313843"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pyspark.sql.functions import *\n",
|
||||
"from pyspark.ml.evaluation import RegressionEvaluator\n",
|
||||
"\n",
|
||||
"scores = predicted.select(predicted['HourlyEnergyOutputMW'], predicted['prediction_output']['value'].alias('prediction'))\n",
|
||||
"\n",
|
||||
"evaluator = RegressionEvaluator(predictionCol=\"prediction\",\n",
|
||||
" labelCol=\"HourlyEnergyOutputMW\",\n",
|
||||
" metricName=\"mae\")\n",
|
||||
"\n",
|
||||
"mae = evaluator.evaluate(scores)\n",
|
||||
"\n",
|
||||
"print(\"Mean absolute error:\", mae)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Configuration settings for scaling to larger data\n",
|
||||
"\n",
|
||||
"## Number and size of nodes in our Kubernetes cluster\n",
|
||||
"We can control the number and size of nodes in our Kubernetes cluster via the node-vm-size and node-count switches in our `aks create` command:\n",
|
||||
"\n",
|
||||
"`az aks create --name mycluster --resource-group myrg --generate-ssh-keys --node-vm-size Standard_DS14_v2 --node-count 3 --kubernetes-version 1.10.9`\n",
|
||||
"\n",
|
||||
"More information is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-on-aks?view=sqlallproducts-allversions#create-a-kubernetes-cluster).\n",
|
||||
"\n",
|
||||
"## Number of Spark pods\n",
|
||||
"We can control the number of Spark pods via the CLUSTER_STORAGE_POOL_REPLICAS environment variable used by `mssqlctl create cluster`:\n",
|
||||
"\n",
|
||||
"SET CLUSTER_STORAGE_POOL_REPLICAS=2\n",
|
||||
"\n",
|
||||
"## YARN scheduler memory and cores\n",
|
||||
"We can control the YARN scheduler memory and cores via the following environment variable used by `mssqlctl create cluster`:\n",
|
||||
"\n",
|
||||
"- YARN_SCHEDULER_MAX_MEMORY\n",
|
||||
"- YARN_SCHEDULER_MAX_VCORES\n",
|
||||
"- YARN_NODEMANAGER_RESOURCE_MEMORY\n",
|
||||
"- YARN_NODEMANAGER_RESOURCE_VCORES\n",
|
||||
"\n",
|
||||
"Further information regarding mssqlctl environtment variables is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions#define-environment-variables).\n",
|
||||
"\n",
|
||||
"## Livy timeout\n",
|
||||
"The Livy timeout sets a limit on the runtime of a cell in a PySpark3 Jupyter notebook. In SQL Server 2019 Big Data CTP 2.1, the Livy timeout defaults to 1 hour. In CTP 2.2, it defaults to 24 days. One can modify this as follows:\n",
|
||||
"\n",
|
||||
"- Log into the mssql-master-pool-0 pod using this command (requires permission to run kubectl):\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"kubectl exec -it mssql-master-pool-0 -n <your-cluster-name> -- /bin/bash\n",
|
||||
"```\n",
|
||||
"- To set the Livy timeout to 24 days, run the following command or edit /livy/conf/livy.conf accordingly:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"echo 'livy.server.session.timeout = 24d' | cat >> /livy/conf/livy.conf \n",
|
||||
"```\n",
|
||||
"- Then restart the Livy server by running the following command:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"supervisorctl restart livy\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Monitoring and Diagnostics\n",
|
||||
"## YARN UI\n",
|
||||
"\n",
|
||||
"Access from the \"View Yarn History\" button in Azure Data Studio (ADS) or at `https://<knox-gateway>:30443/gateway/default/yarn`\n",
|
||||
"\n",
|
||||
"## Spark UI\n",
|
||||
"\n",
|
||||
"Access from the \"Spark UI\" link that appears after running the first cell in a notebook in a (Py)Spark kernel in ADS or by clicking the ApplicationMaster link of a running application in the YARN UI\n",
|
||||
"\n",
|
||||
"## Spark History\n",
|
||||
"\n",
|
||||
"Access from the \"View Spark History\" button in ADS or at `https://<knox-gateway>:30443/gateway/default/sparkhistory`\n",
|
||||
"\n",
|
||||
"## H2O Flow UI\n",
|
||||
"- The command `H2OContext.getOrCreate(spark)` outputs the IP address and port number for connection to H2O’s Flow UI, for example:\n",
|
||||
"\n",
|
||||
" `H2O connection url: http://10.244.0.16:54325`\n",
|
||||
"\n",
|
||||
"- This connection can be forwarded to one’s workstation using this command (requires permission to run kubectl):\n",
|
||||
"\n",
|
||||
" `kubectl -n test port-forward <pod-running-driver> <port>`\n",
|
||||
"\n",
|
||||
" Here is an example:\n",
|
||||
"\n",
|
||||
" `kubectl -n test port-forward mssql-storage-pool-default-0 54325`\n",
|
||||
"\n",
|
||||
" The port number is the number after the colon in the H2O connection URL.\n",
|
||||
" \n",
|
||||
" To determine `<pod-running-driver>`, run this command in the PySpark3 kernel in the ADS notebook:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
" # Print the hostname of the pod where the driver is running\n",
|
||||
" stdout = subprocess.check_output( \n",
|
||||
" \"hostname\", \n",
|
||||
" stderr=subprocess.STDOUT, \n",
|
||||
" shell=True).decode(\"utf-8\") \n",
|
||||
" print(stdout) \n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"- After setting up port forwarding, the Flow UI can be accessed at `http://localhost:<port>`; for example, `http://localhost:54325`"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "PySpark3",
|
||||
"language": "",
|
||||
"name": "pyspark3kernel"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "python",
|
||||
"version": 3
|
||||
},
|
||||
"mimetype": "text/x-python",
|
||||
"name": "pyspark3",
|
||||
"pygments_lexer": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
"nbformat_minor": 2,
|
||||
"nbformat": 4,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Powerplant Output Prediction\n- This notebook is based on the power plant output prediction example presented in H2O’s [blog post on H2O AutoML in Spark](https://www.h2o.ai/blog/h2os-automl-in-spark/).\n- Run this notebook in Azure Data Studio connected to a SQL Server 2019 Big Data Cluster by following the instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/notebooks-guidance?view=sqlallproducts-allversions).",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "## Spark Configuration\n- We can control the Spark Driver and Executor memory, cores, and number of executors per pod using the “%%configure” cell magic\n- Additional configuration settings are listed at the end of this notebook\n",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "%%configure -f\n{\n \"executorMemory\": \"4g\",\n \"driverMemory\": \"4g\",\n \"executorCores\": 2,\n \"driverCores\": 2,\n \"numExecutors\": 2\n}",
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "Current session configs: <tt>{'executorMemory': '4g', 'driverMemory': '4g', 'executorCores': 2, 'driverCores': 2, 'numExecutors': 2, 'kind': 'pyspark3'}</tt><br>",
|
||||
"text/plain": "<IPython.core.display.HTML object>"
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "No active sessions.",
|
||||
"text/plain": "<IPython.core.display.HTML object>"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Install H2O\n- This cell downloads the h2o_pysparkling_* python package and installs it on the pod where the Spark driver is currently running, if it is not already installed. Propagating the software to additional pods is handled automatically once we launch H2O.\n- _**Be sure to match the version of pysparkling to the version of Spark on your cluster:**_\n - For clusters running Spark 2.3 (SQL Server 2019 CTP 2.3 and earlier), use \"pip3 install h2o_pysparkling_2.3\"\n - For clusters running Spark 2.4 (SQL Server 2019 CTP 2.4 and later), use \"pip3 install h2o_pysparkling_2.4\"\n- For an enterprise scenario where we cannot reach out to the PyPi repository on the Internet, pip3 can be pointed to a local copy.",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "import subprocess\n\n# Install H2O PySparkling\nstdout = subprocess.check_output(\n \"pip3 install h2o_pysparkling_2.3\",\n stderr=subprocess.STDOUT,\n shell=True).decode(\"utf-8\")\nprint(stdout)\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Starting Spark application\n"
|
||||
},
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>24</td><td>application_1543381571657_0025</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.193.17.116:30443/gateway/default/yarn/proxy/application_1543381571657_0025/\">Link</a></td><td><a target=\"_blank\" href=\"http://mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local:8042/node/containerlogs/container_1543381571657_0025_01_000001/root\">Link</a></td><td>✔</td></tr></table>",
|
||||
"text/plain": "<IPython.core.display.HTML object>"
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "SparkSession available as 'spark'.\nCollecting h2o_pysparkling_2.3\nRequirement already satisfied (use --upgrade to upgrade): tabulate in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): six in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): colorama>=0.3.8 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): pyspark<=2.3.2,>=2.3.0 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): idna<2.8,>=2.5 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\nRequirement already satisfied (use --upgrade to upgrade): py4j==0.10.7 in /usr/local/lib/python3.5/dist-packages (from pyspark<=2.3.2,>=2.3.0->h2o_pysparkling_2.3)\nInstalling collected packages: h2o-pysparkling-2.3\nSuccessfully installed h2o-pysparkling-2.3-2.3.18\nYou are using pip version 8.1.1, however version 18.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command."
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Download and copy data to HDFS",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "dataFileName = \"powerplant_output.csv\"\ndataFileUrl = \"https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/\" + dataFileName\n\n# Download data file and copy to HDFS, if not already there\ncmd = 'hdfs dfs -ls /tmp/' + dataFileName + ' || ' \\\n '(wget ' + dataFileUrl + ' && ' \\\n 'hdfs dfs -copyFromLocal ' + dataFileName + ' /tmp && ' \\\n 'rm ' + dataFileName + ')'\n\nstdout = subprocess.check_output(\n cmd,\n stderr=subprocess.STDOUT,\n shell=True).decode(\"utf-8\")\nprint(stdout)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "ls: `/tmp/powerplant_output.csv': No such file or directory\n--2018-12-06 19:29:35-- https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/powerplant_output.csv\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 308777 (302K) [text/plain]\nSaving to: 'powerplant_output.csv'\n\n 0K .......... .......... .......... .......... .......... 16% 574K 0s\n 50K .......... .......... .......... .......... .......... 33% 1.14M 0s\n 100K .......... .......... .......... .......... .......... 49% 30.9M 0s\n 150K .......... .......... .......... .......... .......... 66% 1.16M 0s\n 200K .......... .......... .......... .......... .......... 82% 42.7M 0s\n 250K .......... .......... .......... .......... .......... 99% 77.2M 0s\n 300K . 100% 2937G=0.2s\n\n2018-12-06 19:29:36 (1.68 MB/s) - 'powerplant_output.csv' saved [308777/308777]"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Start H2O engine",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "from pysparkling import H2OContext\n\nhc = H2OContext.getOrCreate(spark)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Connecting to H2O server at http://10.244.0.66:54323... successful.\n-------------------------- ---------------------------------------------------\nH2O cluster uptime: 13 secs\nH2O cluster timezone: Etc/UTC\nH2O data parsing timezone: UTC\nH2O cluster version: 3.22.0.2\nH2O cluster version age: 14 days, 12 hours and 24 minutes\nH2O cluster name: sparkling-water-root_application_1543381571657_0021\nH2O cluster total nodes: 2\nH2O cluster free memory: 6.928 Gb\nH2O cluster total cores: 32\nH2O cluster allowed cores: 4\nH2O cluster status: accepting new members, healthy\nH2O connection url: http://10.244.0.66:54323\nH2O connection proxy:\nH2O internal security: False\nH2O API Extensions: XGBoost, Algos, AutoML, Core V3, Core V4\nPython version: 3.5.2 final\n-------------------------- ---------------------------------------------------\n\nSparkling Water Context:\n * H2O name: sparkling-water-root_application_1543381571657_0021\n * cluster size: 2\n * list of used nodes:\n (executorId, host, port)\n ------------------------\n (1,mssql-storage-pool-default-1.service-storage-pool-default.test.svc.cluster.local,54321)\n (2,mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local,54321)\n ------------------------\n\n Open H2O Flow in browser: http://10.244.0.66:54323 (CMD + click in Mac OSX)\n\n \n * Yarn App ID of Spark application: application_1543381571657_0021"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "# Print the hostname of the pod where the driver is running\nstdout = subprocess.check_output(\n \"hostname\",\n stderr=subprocess.STDOUT,\n shell=True).decode(\"utf-8\")\nprint(stdout)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "mssql-storage-pool-default-0"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Read and split data",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "powerplant_df = spark.read.option(\"inferSchema\", \"true\").csv(\"/tmp/powerplant_output.csv\", header=True)\n\nsplits = powerplant_df.randomSplit([0.8, 0.2], seed=1)\ntrain = splits[0]\nfor_predictions = splits[1]\n",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Training and Prediction\n- Fit AutoML model on training data\n- Generate predictions on \"for_predictions\" data",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "from pyspark.ml.feature import SQLTransformer\nfrom pysparkling.ml import H2OAutoML\nfrom pyspark.ml import Pipeline\n\ntemperatureTransformer = SQLTransformer(statement=\"SELECT * FROM __THIS__ WHERE TemperatureCelcius > 10\")\n\nautomlEstimator = H2OAutoML(maxModels=2, predictionCol=\"HourlyEnergyOutputMW\", seed=1)\n\npipeline = Pipeline(stages=[temperatureTransformer, automlEstimator])\n\n# Fit AutoML model\nmodel = pipeline.fit(train)\n\n# Generate predictions using fitted model\npredicted = model.transform(for_predictions)\n\npredicted.show()\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n|TemperatureCelcius|ExhaustVacuumHg|AmbientPressureMillibar|RelativeHumidity|HourlyEnergyOutputMW| prediction_output|\n+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n| 10.01| 41.17| 1018.78| 86.84| 479.4|[477.07336140008584]|\n| 10.02| 39.66| 1016.34| 79.98| 480.05| [476.9418514217447]|\n| 10.03| 43.13| 1014.85| 70.09| 482.16| [476.1059605448468]|\n| 10.04| 41.62| 1013.36| 95.17| 463.87| [470.5313939263834]|\n| 10.05| 41.58| 1021.35| 95.19| 469.03| [469.2805814291723]|\n| 10.06| 34.69| 1027.9| 71.73| 477.68| [477.8063480544951]|\n| 10.08| 37.92| 1010.47| 66.37| 474.63| [475.4396167315097]|\n| 10.08| 41.16| 1023.14| 96.03| 469.17|[471.84298416925895]|\n| 10.09| 41.01| 1019.89| 96.55| 471.15| [471.4895037099167]|\n| 10.1| 41.4| 1024.29| 85.94| 474.28|[477.99005354475656]|\n| 10.11| 39.35| 1015.19| 90.74| 479.83| [478.0068968374228]|\n| 10.11| 39.72| 1019.1| 69.68| 476.8|[474.19764922516293]|\n| 10.11| 42.49| 1010.22| 82.11| 483.56|[477.02467192323263]|\n| 10.12| 41.55| 1005.78| 62.34| 475.46|[475.25159731408183]|\n| 10.12| 41.78| 1013.43| 73.47| 477.67| [475.3766641350098]|\n| 10.13| 39.18| 1024.09| 85.48| 479.42|[478.11614912148826]|\n| 10.15| 39.22| 1020.09| 68.75| 474.87| [477.1365182805492]|\n| 10.15| 41.46| 1019.78| 83.56| 481.31|[479.42546663321224]|\n| 10.15| 43.41| 1018.4| 82.07| 473.43|[476.34953390951085]|\n| 10.16| 39.3| 1019.71| 81.21| 480.74|[476.95324226094823]|\n+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\nonly showing top 20 rows"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Display the leaderboard metrics",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "automlEstimator.leaderboard().show(truncate=False)",
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n|model_id |mean_residual_deviance|rmse |mse |mae |rmsle |\n+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n|StackedEnsemble_BestOfFamily_AutoML_20181206_193040|11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n|StackedEnsemble_AllModels_AutoML_20181206_193040 |11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n|DRF_1_AutoML_20181206_193040 |11.349494687812056 |3.3689011098297406|11.349494687812056|2.5426288374345605|0.007472705853530634|\n|XRT_1_AutoML_20181206_193040 |11.464865035526516 |3.3859806608317355|11.464865035526516|2.545269555169042 |0.007510102104757881|\n+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Evaluate predictions on held-out data\n- As expected, we find that the mean absolute error (mae) on the for_predictions data is similar to the leaderboard mae",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "from pyspark.sql.functions import *\nfrom pyspark.ml.evaluation import RegressionEvaluator\n\nscores = predicted.select(predicted['HourlyEnergyOutputMW'], predicted['prediction_output']['value'].alias('prediction'))\n\nevaluator = RegressionEvaluator(predictionCol=\"prediction\",\n labelCol=\"HourlyEnergyOutputMW\",\n metricName=\"mae\")\n\nmae = evaluator.evaluate(scores)\n\nprint(\"Mean absolute error:\", mae)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Mean absolute error: 2.3167231443313843"
|
||||
}
|
||||
],
|
||||
"execution_count": 1
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Configuration settings for scaling to larger data\n\n## Number and size of nodes in our Kubernetes cluster\nWe can control the number and size of nodes in our Kubernetes cluster via the node-vm-size and node-count switches in our `aks create` command:\n\n`az aks create --name mycluster --resource-group myrg --generate-ssh-keys --node-vm-size Standard_DS14_v2 --node-count 3 --kubernetes-version 1.10.9`\n\nMore information is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-on-aks?view=sqlallproducts-allversions#create-a-kubernetes-cluster).\n\n## Number of Spark pods\nWe can control the number of Spark pods via the CLUSTER_STORAGE_POOL_REPLICAS environment variable used by `mssqlctl create cluster`:\n\nSET CLUSTER_STORAGE_POOL_REPLICAS=2\n\n## YARN scheduler memory and cores\nWe can control the YARN scheduler memory and cores via the following environment variable used by `mssqlctl create cluster`:\n\n- YARN_SCHEDULER_MAX_MEMORY\n- YARN_SCHEDULER_MAX_VCORES\n- YARN_NODEMANAGER_RESOURCE_MEMORY\n- YARN_NODEMANAGER_RESOURCE_VCORES\n\nFurther information regarding mssqlctl environtment variables is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions#define-environment-variables).\n\n## Livy timeout\nThe Livy timeout sets a limit on the runtime of a cell in a PySpark3 Jupyter notebook. In SQL Server 2019 Big Data CTP 2.1, the Livy timeout defaults to 1 hour. In CTP 2.2, it defaults to 24 days. One can modify this as follows:\n\n- Log into the mssql-master-pool-0 pod using this command (requires permission to run kubectl):\n\n```\nkubectl exec -it mssql-master-pool-0 -n <your-cluster-name> -- /bin/bash\n```\n- To set the Livy timeout to 24 days, run the following command or edit /livy/conf/livy.conf accordingly:\n\n```\necho 'livy.server.session.timeout = 24d' | cat >> /livy/conf/livy.conf \n```\n- Then restart the Livy server by running the following command:\n\n```\nsupervisorctl restart livy\n```",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "# Monitoring and Diagnostics\n## YARN UI\n\nAccess from the \"View Yarn History\" button in Azure Data Studio (ADS) or at `https://<knox-gateway>:30443/gateway/default/yarn`\n\n## Spark UI\n\nAccess from the \"Spark UI\" link that appears after running the first cell in a notebook in a (Py)Spark kernel in ADS or by clicking the ApplicationMaster link of a running application in the YARN UI\n\n## Spark History\n\nAccess from the \"View Spark History\" button in ADS or at `https://<knox-gateway>:30443/gateway/default/sparkhistory`\n\n## H2O Flow UI\n- The command `H2OContext.getOrCreate(spark)` outputs the IP address and port number for connection to H2O’s Flow UI, for example:\n\n `H2O connection url: http://10.244.0.16:54325`\n\n- This connection can be forwarded to one’s workstation using this command (requires permission to run kubectl):\n\n `kubectl -n test port-forward <pod-running-driver> <port>`\n\n Here is an example:\n\n `kubectl -n test port-forward mssql-storage-pool-default-0 54325`\n\n The port number is the number after the colon in the H2O connection URL.\n \n To determine `<pod-running-driver>`, run this command in the PySpark3 kernel in the ADS notebook:\n\n```python\n # Print the hostname of the pod where the driver is running\n stdout = subprocess.check_output( \n \"hostname\", \n stderr=subprocess.STDOUT, \n shell=True).decode(\"utf-8\") \n print(stdout) \n```\n\n- After setting up port forwarding, the Flow UI can be accessed at `http://localhost:<port>`; for example, `http://localhost:54325`",
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -9,6 +9,7 @@ The new built-in notebooks in Azure Data Studio enables data scientists and data
|
||||
1. Right-click on the server name, select **Manage**, switch to **SQL Server Big Data Cluster** tab, and use open Notebook
|
||||
|
||||
## __[dataloading](dataloading/)__
|
||||
<<<<<<< HEAD
|
||||
|
||||
This folder contains samples that show how to load data using Spark.
|
||||
|
||||
@@ -21,11 +22,23 @@ This folder contains samples that show how to load data using Spark.
|
||||
[DataLoading - Transforming CSV to Parquet](dataloading/transform-csv-files.ipynb/)
|
||||
|
||||
[Data Transfer - Spark to SQL using JDBC ](spark_to_sql/spark_to_sql_jdbc.ipynb/)
|
||||
=======
|
||||
|
||||
This folder contains samples that show how to load data using Spark.
|
||||
|
||||
[dataloading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
|
||||
>>>>>>> upstream/master
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Download and save the notebook file [dataloading/transnform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/) locally.
|
||||
|
||||
<<<<<<< HEAD
|
||||
2. Open the notebook in Azure Data Studio, wait for the “Kernel” and the target context (“Attach to”) to be populated. Set the “Kernel” to **PySpark3** and **Attach to** needs to be the IP address of your big data cluster endpoint.
|
||||
|
||||
3. Run each cell in the Notebook sequentially.
|
||||
3. Run each cell in the Notebook sequentially.
|
||||
=======
|
||||
1. Open the notebook in Azure Data Studio, wait for the “Kernel” and the target context (“Attach to”) to be populated. Set the “Kernel” to **PySpark3** and **Attach to** needs to be the IP address of your big data cluster endpoint.
|
||||
|
||||
1. Run each cell in the Notebook sequentially.
|
||||
>>>>>>> upstream/master
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user