mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
update notebook and add readme
This commit is contained in:
Mario Inchiosa
@@ -0,0 +1,19 @@
|
||||
# Automated Machine Learning using H2O in SQL Server 2019 Big Data Cluster
|
||||
|
||||
## Machine Learning using Spark
|
||||
|
||||
The new built-in notebooks in Azure Data Studio enable data scientists and data engineers to run Python, R, or Scala code against the cluster. This is a great way to explore the data and build machine learning models. Notebooks facilitate collaboration between teammates working on a shared data set.
|
||||
|
||||
This sample uses the automated machine learning capabilities of the third party H2O package running in Spark in a SQL Server 2019 Big Data Cluster to build a machine learning model that predicts powerplant output.
|
||||
|
||||
### Instructions
|
||||
|
||||
1. From Azure Data Studio, connect to the SQL Server Big Data Cluster endpoint. Information about how you connect from Azure Data Studio can be found [here](https://docs.microsoft.com/en-us/sql/azure-data-studio/sql-server-2019-extension?view=sql-server-ver15).
|
||||
|
||||
1. Download and save the notebook file [h2o-automl-powerplant.ipynb](h2o-automl-powerplant.ipynb/) locally.
|
||||
|
||||
1. Open the notebook file in Azure Data Studio (right click on the SQL Server big data cluster server name-> **Manage**-> Open Notebook).
|
||||
|
||||
1. Wait for the “Kernel” and the target context (“Attach to”) to be populated. Set the “Kernel” to **PySpark3** and “Attach to” needs to be the IP address of your big data cluster endpoint.
|
||||
|
||||
1. Run each cell in the Notebook sequentially using Azure Data Studio. Expect the first cell to take 20 sec to finish. Other cells downloading and installing H2O, downloading data, and performing automated machine learning may take several minutes to finish.
|
||||
+275
-154
@@ -1,5 +1,75 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Powerplant Output Prediction\n",
|
||||
"- This notebook is based on the power plant output prediction example presented in H2O’s [blog post on H2O AutoML in Spark](https://www.h2o.ai/blog/h2os-automl-in-spark/).\n",
|
||||
"- Run this notebook in Azure Data Studio connected to a SQL Server 2019 Big Data Cluster by following the instructions [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/notebooks-guidance?view=sqlallproducts-allversions)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Spark Configuration\n",
|
||||
"- We can control the Spark Driver and Executor memory, cores, and number of executors per pod using the “%%configure” cell magic\n",
|
||||
"- Additional configuration settings are listed at the end of this notebook\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"Current session configs: <tt>{'executorMemory': '4g', 'driverMemory': '4g', 'executorCores': 2, 'driverCores': 2, 'numExecutors': 2, 'kind': 'pyspark3'}</tt><br>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"No active sessions."
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%%configure -f\n",
|
||||
"{\n",
|
||||
" \"executorMemory\": \"4g\",\n",
|
||||
" \"driverMemory\": \"4g\",\n",
|
||||
" \"executorCores\": 2,\n",
|
||||
" \"driverCores\": 2,\n",
|
||||
" \"numExecutors\": 2\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Install H2O\n",
|
||||
"- This cell downloads the h2o_pysparkling_2.3 python package and installs it on the pod where the Spark driver is currently running, if it is not already installed. Propagating the software to additional pods is handled automatically once we launch H2O.\n",
|
||||
"- For an enterprise scenario where we cannot reach out to the PyPi repository on the Internet, pip3 can be pointed to a local copy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@@ -18,7 +88,7 @@
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table>\n",
|
||||
"<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>1</td><td>application_1543381571657_0002</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.193.17.116:30443/gateway/default/yarn/proxy/application_1543381571657_0002/\">Link</a></td><td><a target=\"_blank\" href=\"http://mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local:8042/node/containerlogs/container_1543381571657_0002_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
"<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>24</td><td>application_1543381571657_0025</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.193.17.116:30443/gateway/default/yarn/proxy/application_1543381571657_0025/\">Link</a></td><td><a target=\"_blank\" href=\"http://mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local:8042/node/containerlogs/container_1543381571657_0025_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
@@ -31,61 +101,21 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"SparkSession available as 'spark'.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"\n",
|
||||
"start_time = time.time()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"SparkSession available as 'spark'.\n",
|
||||
"Collecting h2o_pysparkling_2.3\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/e2/2d/43ea2377f68a072161c0dc05a90d722e9b391bac6bb28ac0f54fb1bdafbf/h2o_pysparkling_2.3-2.3.18.tar.gz (51.2MB)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Collecting tabulate (from h2o_pysparkling_2.3)\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/12/c2/11d6845db5edf1295bc08b2f488cf5937806586afe42936c3f34c097ebdc/tabulate-0.8.2.tar.gz (45kB)\n",
|
||||
"Collecting future (from h2o_pysparkling_2.3)\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)\n",
|
||||
"Collecting colorama>=0.3.8 (from h2o_pysparkling_2.3)\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): tabulate in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): six in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Collecting pyspark<=2.3.2,>=2.3.0 (from h2o_pysparkling_2.3)\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/5e/cb/d8ff49ba885e2c88b8cf2967edd84235ffa9ac301bffef657dfa5605a112/pyspark-2.3.2.tar.gz (211.9MB)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): colorama>=0.3.8 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): pyspark<=2.3.2,>=2.3.0 in /usr/local/lib/python3.5/dist-packages (from h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): idna<2.8,>=2.5 in /usr/local/lib/python3.5/dist-packages (from requests->h2o_pysparkling_2.3)\n",
|
||||
"Collecting py4j==0.10.7 (from pyspark<=2.3.2,>=2.3.0->h2o_pysparkling_2.3)\n",
|
||||
" Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)\n",
|
||||
"Building wheels for collected packages: h2o-pysparkling-2.3, tabulate, future, pyspark\n",
|
||||
" Running setup.py bdist_wheel for h2o-pysparkling-2.3: started\n",
|
||||
" Running setup.py bdist_wheel for h2o-pysparkling-2.3: finished with status 'done'\n",
|
||||
" Stored in directory: /home/.cache/pip/wheels/0c/cd/e6/65d3a3c52525dc1ea3d13ad17e0935172db2f0dcd4a08322ba\n",
|
||||
" Running setup.py bdist_wheel for tabulate: started\n",
|
||||
" Running setup.py bdist_wheel for tabulate: finished with status 'done'\n",
|
||||
" Stored in directory: /home/.cache/pip/wheels/2a/85/33/2f6da85d5f10614cbe5a625eab3b3aebfdf43e7b857f25f829\n",
|
||||
" Running setup.py bdist_wheel for future: started\n",
|
||||
" Running setup.py bdist_wheel for future: finished with status 'done'\n",
|
||||
" Stored in directory: /home/.cache/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e\n",
|
||||
" Running setup.py bdist_wheel for pyspark: started\n",
|
||||
" Running setup.py bdist_wheel for pyspark: finished with status 'done'\n",
|
||||
" Stored in directory: /home/.cache/pip/wheels/be/7d/34/cd3cfbc75d8b6b6ae0658e5425348560b86d187fe3e53832cc\n",
|
||||
"Successfully built h2o-pysparkling-2.3 tabulate future pyspark\n",
|
||||
"Installing collected packages: tabulate, future, colorama, py4j, pyspark, h2o-pysparkling-2.3\n",
|
||||
"Successfully installed colorama-0.4.1 future-0.17.1 h2o-pysparkling-2.3-2.3.18 py4j-0.10.7 pyspark-2.3.2 tabulate-0.8.2\n",
|
||||
"Requirement already satisfied (use --upgrade to upgrade): py4j==0.10.7 in /usr/local/lib/python3.5/dist-packages (from pyspark<=2.3.2,>=2.3.0->h2o_pysparkling_2.3)\n",
|
||||
"Installing collected packages: h2o-pysparkling-2.3\n",
|
||||
"Successfully installed h2o-pysparkling-2.3-2.3.18\n",
|
||||
"You are using pip version 8.1.1, however version 18.1 is available.\n",
|
||||
"You should consider upgrading via the 'pip install --upgrade pip' command."
|
||||
]
|
||||
@@ -102,6 +132,13 @@
|
||||
"print(stdout)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Download and copy data to HDFS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@@ -114,22 +151,22 @@
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"ls: `/tmp/powerplant_output.csv': No such file or directory\n",
|
||||
"--2018-11-28 18:21:03-- https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/powerplant_output.csv\n",
|
||||
"--2018-12-06 19:29:35-- https://raw.githubusercontent.com/h2oai/h2o-tutorials/master/h2o-world-2017/automl/data/powerplant_output.csv\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 308777 (302K) [text/plain]\n",
|
||||
"Saving to: 'powerplant_output.csv'\n",
|
||||
"\n",
|
||||
" 0K .......... .......... .......... .......... .......... 16% 293K 1s\n",
|
||||
" 50K .......... .......... .......... .......... .......... 33% 568K 1s\n",
|
||||
" 100K .......... .......... .......... .......... .......... 49% 294K 0s\n",
|
||||
" 150K .......... .......... .......... .......... .......... 66% 228K 0s\n",
|
||||
" 200K .......... .......... .......... .......... .......... 82% 193K 0s\n",
|
||||
" 250K .......... .......... .......... .......... .......... 99% 231K 0s\n",
|
||||
" 300K . 100% 2937G=1.1s\n",
|
||||
" 0K .......... .......... .......... .......... .......... 16% 574K 0s\n",
|
||||
" 50K .......... .......... .......... .......... .......... 33% 1.14M 0s\n",
|
||||
" 100K .......... .......... .......... .......... .......... 49% 30.9M 0s\n",
|
||||
" 150K .......... .......... .......... .......... .......... 66% 1.16M 0s\n",
|
||||
" 200K .......... .......... .......... .......... .......... 82% 42.7M 0s\n",
|
||||
" 250K .......... .......... .......... .......... .......... 99% 77.2M 0s\n",
|
||||
" 300K . 100% 2937G=0.2s\n",
|
||||
"\n",
|
||||
"2018-11-28 18:21:05 (268 KB/s) - 'powerplant_output.csv' saved [308777/308777]"
|
||||
"2018-12-06 19:29:36 (1.68 MB/s) - 'powerplant_output.csv' saved [308777/308777]"
|
||||
]
|
||||
}
|
||||
],
|
||||
@@ -150,6 +187,13 @@
|
||||
"print(stdout)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Start H2O engine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@@ -163,16 +207,16 @@
|
||||
"text": [
|
||||
"Connecting to H2O server at http://10.244.0.66:54323... successful.\n",
|
||||
"-------------------------- ---------------------------------------------------\n",
|
||||
"H2O cluster uptime: 12 secs\n",
|
||||
"H2O cluster uptime: 13 secs\n",
|
||||
"H2O cluster timezone: Etc/UTC\n",
|
||||
"H2O data parsing timezone: UTC\n",
|
||||
"H2O cluster version: 3.22.0.2\n",
|
||||
"H2O cluster version age: 6 days\n",
|
||||
"H2O cluster name: sparkling-water-root_application_1543381571657_0002\n",
|
||||
"H2O cluster total nodes: 3\n",
|
||||
"H2O cluster free memory: 3.698 Gb\n",
|
||||
"H2O cluster total cores: 48\n",
|
||||
"H2O cluster allowed cores: 3\n",
|
||||
"H2O cluster version age: 14 days, 12 hours and 24 minutes\n",
|
||||
"H2O cluster name: sparkling-water-root_application_1543381571657_0021\n",
|
||||
"H2O cluster total nodes: 2\n",
|
||||
"H2O cluster free memory: 6.928 Gb\n",
|
||||
"H2O cluster total cores: 32\n",
|
||||
"H2O cluster allowed cores: 4\n",
|
||||
"H2O cluster status: accepting new members, healthy\n",
|
||||
"H2O connection url: http://10.244.0.66:54323\n",
|
||||
"H2O connection proxy:\n",
|
||||
@@ -182,20 +226,19 @@
|
||||
"-------------------------- ---------------------------------------------------\n",
|
||||
"\n",
|
||||
"Sparkling Water Context:\n",
|
||||
" * H2O name: sparkling-water-root_application_1543381571657_0002\n",
|
||||
" * cluster size: 3\n",
|
||||
" * H2O name: sparkling-water-root_application_1543381571657_0021\n",
|
||||
" * cluster size: 2\n",
|
||||
" * list of used nodes:\n",
|
||||
" (executorId, host, port)\n",
|
||||
" ------------------------\n",
|
||||
" (1,mssql-storage-pool-default-1.service-storage-pool-default.test.svc.cluster.local,54321)\n",
|
||||
" (2,mssql-storage-pool-default-0.service-storage-pool-default.test.svc.cluster.local,54321)\n",
|
||||
" (3,mssql-storage-pool-default-1.service-storage-pool-default.test.svc.cluster.local,54323)\n",
|
||||
" ------------------------\n",
|
||||
"\n",
|
||||
" Open H2O Flow in browser: http://10.244.0.66:54323 (CMD + click in Mac OSX)\n",
|
||||
"\n",
|
||||
" \n",
|
||||
" * Yarn App ID of Spark application: application_1543381571657_0002"
|
||||
" * Yarn App ID of Spark application: application_1543381571657_0021"
|
||||
]
|
||||
}
|
||||
],
|
||||
@@ -221,7 +264,7 @@
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Print the hostname\n",
|
||||
"# Print the hostname of the pod where the driver is running\n",
|
||||
"stdout = subprocess.check_output(\n",
|
||||
" \"hostname\",\n",
|
||||
" stderr=subprocess.STDOUT,\n",
|
||||
@@ -229,6 +272,37 @@
|
||||
"print(stdout)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Read and split data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"powerplant_df = spark.read.option(\"inferSchema\", \"true\").csv(\"/tmp/powerplant_output.csv\", header=True)\n",
|
||||
"\n",
|
||||
"splits = powerplant_df.randomSplit([0.8, 0.2], seed=1)\n",
|
||||
"train = splits[0]\n",
|
||||
"for_predictions = splits[1]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Training and Prediction\n",
|
||||
"- Fit AutoML model on training data\n",
|
||||
"- Generate predictions on \"for_predictions\" data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@@ -243,54 +317,58 @@
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"|TemperatureCelcius|ExhaustVacuumHg|AmbientPressureMillibar|RelativeHumidity|HourlyEnergyOutputMW| prediction_output|\n",
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"| 10.01| 41.17| 1018.78| 86.84| 479.4|[478.98443473089264]|\n",
|
||||
"| 10.02| 39.66| 1016.34| 79.98| 480.05| [478.4744401166399]|\n",
|
||||
"| 10.03| 43.13| 1014.85| 70.09| 482.16|[475.98085258410845]|\n",
|
||||
"| 10.04| 41.62| 1013.36| 95.17| 463.87|[468.36092209033416]|\n",
|
||||
"| 10.05| 41.58| 1021.35| 95.19| 469.03|[468.67240194913506]|\n",
|
||||
"| 10.06| 34.69| 1027.9| 71.73| 477.68|[477.79641646128107]|\n",
|
||||
"| 10.08| 37.92| 1010.47| 66.37| 474.63| [475.2033972206058]|\n",
|
||||
"| 10.08| 41.16| 1023.14| 96.03| 469.17| [470.3570481951197]|\n",
|
||||
"| 10.09| 41.01| 1019.89| 96.55| 471.15| [469.6710775300654]|\n",
|
||||
"| 10.1| 41.4| 1024.29| 85.94| 474.28|[477.42433033233783]|\n",
|
||||
"| 10.11| 39.35| 1015.19| 90.74| 479.83| [477.792144625552]|\n",
|
||||
"| 10.11| 39.72| 1019.1| 69.68| 476.8|[473.80793646542656]|\n",
|
||||
"| 10.11| 42.49| 1010.22| 82.11| 483.56| [476.5521742794438]|\n",
|
||||
"| 10.12| 41.55| 1005.78| 62.34| 475.46| [475.9253245215578]|\n",
|
||||
"| 10.12| 41.78| 1013.43| 73.47| 477.67| [475.0193582197604]|\n",
|
||||
"| 10.13| 39.18| 1024.09| 85.48| 479.42| [477.9988789711212]|\n",
|
||||
"| 10.15| 39.22| 1020.09| 68.75| 474.87| [476.4446239317696]|\n",
|
||||
"| 10.15| 41.46| 1019.78| 83.56| 481.31| [479.5577986068803]|\n",
|
||||
"| 10.15| 43.41| 1018.4| 82.07| 473.43|[476.77545876514284]|\n",
|
||||
"| 10.16| 39.3| 1019.71| 81.21| 480.74| [477.2729727910962]|\n",
|
||||
"| 10.01| 41.17| 1018.78| 86.84| 479.4|[477.07336140008584]|\n",
|
||||
"| 10.02| 39.66| 1016.34| 79.98| 480.05| [476.9418514217447]|\n",
|
||||
"| 10.03| 43.13| 1014.85| 70.09| 482.16| [476.1059605448468]|\n",
|
||||
"| 10.04| 41.62| 1013.36| 95.17| 463.87| [470.5313939263834]|\n",
|
||||
"| 10.05| 41.58| 1021.35| 95.19| 469.03| [469.2805814291723]|\n",
|
||||
"| 10.06| 34.69| 1027.9| 71.73| 477.68| [477.8063480544951]|\n",
|
||||
"| 10.08| 37.92| 1010.47| 66.37| 474.63| [475.4396167315097]|\n",
|
||||
"| 10.08| 41.16| 1023.14| 96.03| 469.17|[471.84298416925895]|\n",
|
||||
"| 10.09| 41.01| 1019.89| 96.55| 471.15| [471.4895037099167]|\n",
|
||||
"| 10.1| 41.4| 1024.29| 85.94| 474.28|[477.99005354475656]|\n",
|
||||
"| 10.11| 39.35| 1015.19| 90.74| 479.83| [478.0068968374228]|\n",
|
||||
"| 10.11| 39.72| 1019.1| 69.68| 476.8|[474.19764922516293]|\n",
|
||||
"| 10.11| 42.49| 1010.22| 82.11| 483.56|[477.02467192323263]|\n",
|
||||
"| 10.12| 41.55| 1005.78| 62.34| 475.46|[475.25159731408183]|\n",
|
||||
"| 10.12| 41.78| 1013.43| 73.47| 477.67| [475.3766641350098]|\n",
|
||||
"| 10.13| 39.18| 1024.09| 85.48| 479.42|[478.11614912148826]|\n",
|
||||
"| 10.15| 39.22| 1020.09| 68.75| 474.87| [477.1365182805492]|\n",
|
||||
"| 10.15| 41.46| 1019.78| 83.56| 481.31|[479.42546663321224]|\n",
|
||||
"| 10.15| 43.41| 1018.4| 82.07| 473.43|[476.34953390951085]|\n",
|
||||
"| 10.16| 39.3| 1019.71| 81.21| 480.74|[476.95324226094823]|\n",
|
||||
"+------------------+---------------+-----------------------+----------------+--------------------+--------------------+\n",
|
||||
"only showing top 20 rows"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pyspark.ml.feature import SQLTransformer\n",
|
||||
"from pysparkling.ml import H2OAutoML\n",
|
||||
"from pyspark.ml import Pipeline\n",
|
||||
"from pyspark.ml.feature import SQLTransformer\n",
|
||||
"\n",
|
||||
"powerplant_df = spark.read.option(\"inferSchema\", \"true\").csv(\"/tmp/powerplant_output.csv\", header=True)\n",
|
||||
"\n",
|
||||
"splits = powerplant_df.randomSplit([0.8, 0.2], seed=1)\n",
|
||||
"train = splits[0]\n",
|
||||
"for_predictions = splits[1]\n",
|
||||
"\n",
|
||||
"temperatureTransformer = SQLTransformer(statement=\"SELECT * FROM __THIS__ WHERE TemperatureCelcius > 10\")\n",
|
||||
"\n",
|
||||
"automlEstimator = H2OAutoML(maxModels=2, predictionCol=\"HourlyEnergyOutputMW\",\n",
|
||||
" ratio=0.9, seed=1)\n",
|
||||
"automlEstimator = H2OAutoML(maxModels=2, predictionCol=\"HourlyEnergyOutputMW\", seed=1)\n",
|
||||
"\n",
|
||||
"pipeline = Pipeline(stages=[temperatureTransformer, automlEstimator])\n",
|
||||
"\n",
|
||||
"# Fit AutoML model\n",
|
||||
"model = pipeline.fit(train)\n",
|
||||
"\n",
|
||||
"# Generate predictions using fitted model\n",
|
||||
"predicted = model.transform(for_predictions)\n",
|
||||
"\n",
|
||||
"predicted.show()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Display the leaderboard metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@@ -302,14 +380,14 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+---------------------+\n",
|
||||
"|model_id |mean_residual_deviance|rmse |mse |mae |rmsle |\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+---------------------+\n",
|
||||
"|StackedEnsemble_BestOfFamily_AutoML_20181128_003911|11.00711037433692 |3.317696546451607 |11.00711037433692 |2.460978645004018 |0.0073362304526039795|\n",
|
||||
"|StackedEnsemble_AllModels_AutoML_20181128_003911 |11.00711037433692 |3.317696546451607 |11.00711037433692 |2.460978645004018 |0.0073362304526039795|\n",
|
||||
"|XRT_1_AutoML_20181128_003911 |11.18040320029184 |3.3437109923394757|11.18040320029184 |2.5010207855031896|0.007393792671247257 |\n",
|
||||
"|DRF_1_AutoML_20181128_003911 |11.245178401907951 |3.3533831278140513|11.245178401907951|2.4970917585784322|0.007413255513108019 |\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+---------------------+"
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n",
|
||||
"|model_id |mean_residual_deviance|rmse |mse |mae |rmsle |\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+\n",
|
||||
"|StackedEnsemble_BestOfFamily_AutoML_20181206_193040|11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n",
|
||||
"|StackedEnsemble_AllModels_AutoML_20181206_193040 |11.204658852035337 |3.3473360829225585|11.204658852035337|2.509389117612746 |0.007425043374216511|\n",
|
||||
"|DRF_1_AutoML_20181206_193040 |11.349494687812056 |3.3689011098297406|11.349494687812056|2.5426288374345605|0.007472705853530634|\n",
|
||||
"|XRT_1_AutoML_20181206_193040 |11.464865035526516 |3.3859806608317355|11.464865035526516|2.545269555169042 |0.007510102104757881|\n",
|
||||
"+---------------------------------------------------+----------------------+------------------+------------------+------------------+--------------------+"
|
||||
]
|
||||
}
|
||||
],
|
||||
@@ -318,16 +396,11 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [],
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"from pyspark.sql.functions import *\n",
|
||||
"\n",
|
||||
"scores = predicted.select(predicted['HourlyEnergyOutputMW'], predicted['prediction_output']['value'].alias('prediction'))\n"
|
||||
"# Evaluate predictions on held-out data\n",
|
||||
"- As expected, we find that the mean absolute error (mae) on the for_predictions data is similar to the leaderboard mae"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -341,13 +414,16 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Mean absolute error: 2.4058999799874976"
|
||||
"Mean absolute error: 2.3167231443313843"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pyspark.sql.functions import *\n",
|
||||
"from pyspark.ml.evaluation import RegressionEvaluator\n",
|
||||
"\n",
|
||||
"scores = predicted.select(predicted['HourlyEnergyOutputMW'], predicted['prediction_output']['value'].alias('prediction'))\n",
|
||||
"\n",
|
||||
"evaluator = RegressionEvaluator(predictionCol=\"prediction\",\n",
|
||||
" labelCol=\"HourlyEnergyOutputMW\",\n",
|
||||
" metricName=\"mae\")\n",
|
||||
@@ -358,50 +434,95 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"r squared: 0.9516895664346249"
|
||||
]
|
||||
}
|
||||
],
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"from pyspark.ml.evaluation import RegressionEvaluator\n",
|
||||
"# Configuration settings for scaling to larger data\n",
|
||||
"\n",
|
||||
"evaluator = RegressionEvaluator(predictionCol=\"prediction\",\n",
|
||||
" labelCol=\"HourlyEnergyOutputMW\",\n",
|
||||
" metricName=\"r2\")\n",
|
||||
"## Number and size of nodes in our Kubernetes cluster\n",
|
||||
"We can control the number and size of nodes in our Kubernetes cluster via the node-vm-size and node-count switches in our `aks create` command:\n",
|
||||
"\n",
|
||||
"r2 = evaluator.evaluate(scores)\n",
|
||||
"`az aks create --name mycluster --resource-group myrg --generate-ssh-keys --node-vm-size Standard_DS14_v2 --node-count 3 --kubernetes-version 1.10.9`\n",
|
||||
"\n",
|
||||
"print(\"r squared:\", r2)"
|
||||
"For information is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-on-aks?view=sqlallproducts-allversions#create-a-kubernetes-cluster).\n",
|
||||
"\n",
|
||||
"## Number of Spark pods\n",
|
||||
"We can control the number of Spark pods via the CLUSTER_STORAGE_POOL_REPLICAS environment variable used by `mssqlctl create cluster`:\n",
|
||||
"\n",
|
||||
"SET CLUSTER_STORAGE_POOL_REPLICAS=2\n",
|
||||
"\n",
|
||||
"## YARN scheduler memory and cores\n",
|
||||
"We can control the YARN scheduler memory and cores via the following environment variable used by `mssqlctl create cluster`:\n",
|
||||
"\n",
|
||||
"- YARN_SCHEDULER_MAX_MEMORY\n",
|
||||
"- YARN_SCHEDULER_MAX_VCORES\n",
|
||||
"- YARN_NODEMANAGER_RESOURCE_MEMORY\n",
|
||||
"- YARN_NODEMANAGER_RESOURCE_VCORES\n",
|
||||
"\n",
|
||||
"Further information regarding mssqlctl environtment variables is available [here](https://docs.microsoft.com/en-us/sql/big-data-cluster/deployment-guidance?view=sqlallproducts-allversions#define-environment-variables).\n",
|
||||
"\n",
|
||||
"## Livy timeout\n",
|
||||
"The Livy timeout sets a limit on the runtime of a cell in a PySpark3 Jupyter notebook. In SQL Server 2019 Big Data CTP 2.1, the Livy timeout defaults to 1 hour. In CTP 2.2, it defaults to 24 days. One can modify this as follows:\n",
|
||||
"\n",
|
||||
"- Log into the mssql-master-pool-0 pod using this command (requires permission to run kubectl):\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"kubectl exec -it mssql-master-pool-0 -n <your-cluster-name> -- /bin/bash\n",
|
||||
"```\n",
|
||||
"- To set the Livy timeout to 24 days, run the following command or edit /livy/conf/livy.conf accordingly:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"echo 'livy.server.session.timeout = 24d' | cat >> /livy/conf/livy.conf \n",
|
||||
"```\n",
|
||||
"- Then restart the Livy server by running the following command:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"supervisorctl restart livy\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"language": "python"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Elapsed time: 199.5 seconds."
|
||||
]
|
||||
}
|
||||
],
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"elapsed_time = time.time() - start_time\n",
|
||||
"print(\"Elapsed time: {:.1f} seconds.\".format(elapsed_time))"
|
||||
"# Monitoring and Diagnostics\n",
|
||||
"## YARN UI\n",
|
||||
"\n",
|
||||
"Access from the \"View Yarn History\" button in Azure Data Studio (ADS) or at `https://<knox-gateway>:30443/gateway/default/yarn`\n",
|
||||
"\n",
|
||||
"## Spark UI\n",
|
||||
"\n",
|
||||
"Access from the \"Spark UI\" link that appears after running the first cell in a notebook in a (Py)Spark kernel in ADS or by clicking the ApplicationMaster link of a running application in the YARN UI\n",
|
||||
"\n",
|
||||
"## Spark History\n",
|
||||
"\n",
|
||||
"Access from the \"View Spark History\" button in ADS or at `https://<knox-gateway>:30443/gateway/default/sparkhistory`\n",
|
||||
"\n",
|
||||
"## H2O Flow UI\n",
|
||||
"- The command `H2OContext.getOrCreate(spark)` outputs the IP address and port number for connection to H2O’s Flow UI, for example:\n",
|
||||
"\n",
|
||||
" `H2O connection url: http://10.244.0.16:54325`\n",
|
||||
"\n",
|
||||
"- This connection can be forwarded to one’s workstation using this command (requires permission to run kubectl):\n",
|
||||
"\n",
|
||||
" `kubectl -n test port-forward <pod-running-driver> <port>`\n",
|
||||
"\n",
|
||||
" Here is an example:\n",
|
||||
"\n",
|
||||
" `kubectl -n test port-forward mssql-storage-pool-default-0 54325`\n",
|
||||
"\n",
|
||||
" The port number is the number after the colon in the H2O connection URL.\n",
|
||||
" \n",
|
||||
" To determine `<pod-running-driver>`, run this command in the PySpark3 kernel in the ADS notebook:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
" # Print the hostname of the pod where the driver is running\n",
|
||||
" stdout = subprocess.check_output( \n",
|
||||
" \"hostname\", \n",
|
||||
" stderr=subprocess.STDOUT, \n",
|
||||
" shell=True).decode(\"utf-8\") \n",
|
||||
" print(stdout) \n",
|
||||
"```"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user