mirror of
https://github.com/Microsoft/sql-server-samples.git
synced 2025-12-08 14:58:54 +00:00
update the jupyter notebook
This commit is contained in:
Lixin Gong
@@ -1,11 +1,13 @@
|
||||
# MLeap on SQL Server Big Data cluster
|
||||
This folder shows how we can build a model with [Spark ML](https://spark.apache.org/docs/latest/ml-guide.html), export the model to [MLeap](https://github.com/combust/mleap), and score the model in SQL Server with its [Java Language Extension](https://docs.microsoft.com/en-us/sql/language-extensions/language-extensions-overview?view=sqlallproducts-allversions)
|
||||
This folder shows how we can build a model with [Spark ML](https://spark.apache.org/docs/latest/ml-guide.html), export the model to [MLeap](mleap-docs.combust.ml/), and score the model in SQL Server with its [Java Language Extension](https://docs.microsoft.com/en-us/sql/language-extensions/language-extensions-overview?view=sqlallproducts-allversions)
|
||||
|
||||
## Model training with Spark ML
|
||||
In this sample code, AdultCensusIncome.csv is used to build a Spark ML pipeline model. We can [download the dataset from internet](mleap_sql_test/setup.sh#L11) and [put it on HDFS on a SQL BDC cluster](mleap_sql_test/setup.sh#L12) so that it can be accessed by Spark.
|
||||
|
||||
The data is first [read into Spark](mleap_sql_test/mleap_pyspark.py#L25) and [split into training and testing datasets](mleap_sql_test/mleap_pyspark.py#L64). We then [train a pipeline mode with the training data](mleap_sql_test/mleap_pyspark.py#L87) and [export the model to a mleap bundle](mleap_sql_test/mleap_pyspark.py#L204).
|
||||
|
||||
An equivalent Jupyter notebook is also included [here](train_score_export_ml_models_with_spark.ipynb) if it is preferred over pure Python code.
|
||||
|
||||
## Model scoring with SQL Server
|
||||
Now that we have the Spark ML pipeline model in a common serialization [MLeap bundle](http://mleap-docs.combust.ml/core-concepts/mleap-bundles.html) format, we can score the model in Java without the presence of Spark.
|
||||
|
||||
|
||||
+39
-21
@@ -24,7 +24,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "## Step 1 - Explore your data\r\n### Load the data\r\nFor this example we'll use **AdultCensusIncome** data from [here]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ). From your Azure Data Studio connect to the HDFS/Spark gateway and create a directory called spark_data under HDFS. \r\nDownload [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine and upload to HDFS.Upload AdultCensusIncome.csv to the folder we created.\r\n\r\n### Exploratory Analysis\r\n- Baisc exploration on the data\r\n- Labels & Features\r\n1. **Label** - This refers to predicted value. This is represented as a column in the data. Label is **income** \r\n2. **Features** - This refers to the characteristics that are used to predict. **age** and **hours_per_week**\r\n\r\nNote : In reality features are chosen by applying some correlations techniques to understand what best characterize the Label we are predicting.\r\n\r\n### The Model we will build\r\nIn AdultCensusIncome.csv contains several columsn like Income range, age, hours-per-week, education, occupation etc. We'll build a model that can predict income range would be >50K or <50K.\r\n",
|
||||
"source": "## Step 1 - Explore your data\r\n### Load the data\r\nFor this example we'll use **AdultCensusIncome** data from [here]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ). From your Azure Data Studio connect to the HDFS/Spark gateway and create a directory called spark_data under HDFS. \r\nDownload [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine and upload to HDFS.Upload AdultCensusIncome.csv to the folder we created.\r\n\r\n### Exploratory Analysis\r\n- Baisc exploration on the data\r\n- Labels & Features\r\n1. **Label** - This refers to predicted value. This is represented as a column in the data. Label is **income** \r\n2. **Features** - This refers to the characteristics that are used to predict. **age**, **hours_per_week**, and **education**\r\n\r\nNote : In reality features are chosen by applying some correlations techniques to understand what best characterize the Label we are predicting.\r\n\r\n### The Model we will build\r\nIn AdultCensusIncome.csv contains several columsn like Income range, age, hours-per-week, education, occupation etc. We'll build a model that can predict income range would be >50K or <50K.\r\n",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
@@ -33,35 +33,53 @@
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Number of rows: 32561, Number of coulumns : 15\nroot\n |-- age: integer (nullable = true)\n |-- workclass: string (nullable = true)\n |-- fnlwgt: integer (nullable = true)\n |-- education: string (nullable = true)\n |-- education-num: integer (nullable = true)\n |-- marital-status: string (nullable = true)\n |-- occupation: string (nullable = true)\n |-- relationship: string (nullable = true)\n |-- race: string (nullable = true)\n |-- sex: string (nullable = true)\n |-- capital-gain: integer (nullable = true)\n |-- capital-loss: integer (nullable = true)\n |-- hours-per-week: integer (nullable = true)\n |-- native-country: string (nullable = true)\n |-- income: string (nullable = true)\n\nroot\n |-- age: integer (nullable = true)\n |-- workclass: string (nullable = true)\n |-- fnlwgt: integer (nullable = true)\n |-- education: string (nullable = true)\n |-- education_num: integer (nullable = true)\n |-- marital_status: string (nullable = true)\n |-- occupation: string (nullable = true)\n |-- relationship: string (nullable = true)\n |-- race: string (nullable = true)\n |-- sex: string (nullable = true)\n |-- capital_gain: integer (nullable = true)\n |-- capital_loss: integer (nullable = true)\n |-- hours_per_week: integer (nullable = true)\n |-- native_country: string (nullable = true)\n |-- income: string (nullable = true)"
|
||||
"text": "Starting Spark application\n",
|
||||
"output_type": "stream"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": "<IPython.core.display.HTML object>",
|
||||
"text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>19</td><td>application_1559313998190_0085</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://10.193.17.7:30443/gateway/default/yarn/proxy/application_1559313998190_0085/\">Link</a></td><td><a target=\"_blank\" href=\"https://10.193.17.7:30443/gateway/default/yarn/container/container_1559313998190_0085_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "SparkSession available as 'spark'.\n",
|
||||
"output_type": "stream"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"text": "Number of rows: 32561, Number of coulumns : 15\nroot\n |-- age: integer (nullable = true)\n |-- workclass: string (nullable = true)\n |-- fnlwgt: integer (nullable = true)\n |-- education: string (nullable = true)\n |-- education-num: integer (nullable = true)\n |-- marital-status: string (nullable = true)\n |-- occupation: string (nullable = true)\n |-- relationship: string (nullable = true)\n |-- race: string (nullable = true)\n |-- sex: string (nullable = true)\n |-- capital-gain: integer (nullable = true)\n |-- capital-loss: integer (nullable = true)\n |-- hours-per-week: integer (nullable = true)\n |-- native-country: string (nullable = true)\n |-- income: string (nullable = true)\n\nroot\n |-- age: integer (nullable = true)\n |-- workclass: string (nullable = true)\n |-- fnlwgt: integer (nullable = true)\n |-- education: string (nullable = true)\n |-- education_num: integer (nullable = true)\n |-- marital_status: string (nullable = true)\n |-- occupation: string (nullable = true)\n |-- relationship: string (nullable = true)\n |-- race: string (nullable = true)\n |-- sex: string (nullable = true)\n |-- capital_gain: integer (nullable = true)\n |-- capital_loss: integer (nullable = true)\n |-- hours_per_week: integer (nullable = true)\n |-- native_country: string (nullable = true)\n |-- income: string (nullable = true)",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 3
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "#Basic data exploration\r\n\r\n##1. Sub set the data and print some important columns\r\nprint(\"Select few columns to see the data\")\r\ndata_all.select(['income','age','hours_per_week']).show(10)\r\n\r\n## Find the number of distict values\r\nprint(\"Number of distinct values for income\")\r\nds_sub = data_all.select('income').distinct()\r\nds_sub.show()\r\n\r\n##Add a numberic column(income_code) derived from income column\r\nprint(\"Added numeric column(income_code) derived from income column\")\r\nfrom pyspark.sql.functions import expr\r\n\r\ndf_new = data_all.withColumn(\"income_code\", expr(\"case \\\r\n when income like '%<=50K%' then 0 \\\r\n when income like '%>50K%' then 1 \\\r\n else 2 end \"))\r\n\r\ndf_new.select(['income','age','hours_per_week','income_code']).show(10)\r\n\r\n##Summary statistical operations on dataframe\r\nprint(\"Print a statistical summary of a few columns\")\r\ndf_new.select(['income','age','hours_per_week','income_code']).describe().show()\r\n\r\nprint(\"Calculate Co variance between a few columns to understand features to use\")\r\nmycov = df_new.stat.cov('income_code','hours_per_week')\r\nprint(\"Covariance between income and hours_per_week is\", round(mycov,1))\r\n\r\nmycov = df_new.stat.cov('income_code','age')\r\nprint(\"Covariance between income and age is\", round(mycov,1))\r\n\r\n",
|
||||
"source": "#Basic data exploration\r\n\r\n##1. Sub set the data and print some important columns\r\nprint(\"Select few columns to see the data\")\r\ndata_all.select(['income','age','hours_per_week', 'education']).show(10)\r\n\r\n## Find the number of distict values\r\nprint(\"Number of distinct values for income\")\r\nds_sub = data_all.select('income').distinct()\r\nds_sub.show()\r\n\r\n##Add a numberic column(income_code) derived from income column\r\nprint(\"Added numeric column(income_code) derived from income column\")\r\nfrom pyspark.sql.functions import expr\r\n\r\ndf_new = data_all.withColumn(\"income_code\", expr(\"case \\\r\n when income like '%<=50K%' then 0 \\\r\n when income like '%>50K%' then 1 \\\r\n else 2 end \"))\r\n\r\ndf_new.select(['income', 'age', 'hours_per_week', 'education', 'income_code']).show(10)\r\n\r\n##Summary statistical operations on dataframe\r\nprint(\"Print a statistical summary of a few columns\")\r\ndf_new.select(['income','age','hours_per_week', 'education','income_code']).describe().show()\r\n\r\nprint(\"Calculate Co variance between a few columns to understand features to use\")\r\nmycov = df_new.stat.cov('income_code','hours_per_week')\r\nprint(\"Covariance between income and hours_per_week is\", round(mycov,1))\r\n\r\nmycov = df_new.stat.cov('income_code','age')\r\nprint(\"Covariance between income and age is\", round(mycov,1))\r\n\r\n",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Select few columns to see the data\n+------+---+--------------+\n|income|age|hours_per_week|\n+------+---+--------------+\n| <=50K| 39| 40|\n| <=50K| 50| 13|\n| <=50K| 38| 40|\n| <=50K| 53| 40|\n| <=50K| 28| 40|\n| <=50K| 37| 40|\n| <=50K| 49| 16|\n| >50K| 52| 45|\n| >50K| 31| 50|\n| >50K| 42| 40|\n+------+---+--------------+\nonly showing top 10 rows\n\nNumber of distinct values for income\n+------+\n|income|\n+------+\n| <=50K|\n| >50K|\n+------+\n\nAdded numeric column(income_code) derived from income column\n+------+---+--------------+-----------+\n|income|age|hours_per_week|income_code|\n+------+---+--------------+-----------+\n| <=50K| 39| 40| 0|\n| <=50K| 50| 13| 0|\n| <=50K| 38| 40| 0|\n| <=50K| 53| 40| 0|\n| <=50K| 28| 40| 0|\n| <=50K| 37| 40| 0|\n| <=50K| 49| 16| 0|\n| >50K| 52| 45| 1|\n| >50K| 31| 50| 1|\n| >50K| 42| 40| 1|\n+------+---+--------------+-----------+\nonly showing top 10 rows\n\nPrint a statistical summary of a few columns\n+-------+------+------------------+------------------+-------------------+\n|summary|income| age| hours_per_week| income_code|\n+-------+------+------------------+------------------+-------------------+\n| count| 32561| 32561| 32561| 32561|\n| mean| null| 38.58164675532078|40.437455852092995| 0.2408095574460244|\n| stddev| null|13.640432553581356|12.347428681731838|0.42758148856469247|\n| min| <=50K| 17| 1| 0|\n| max| >50K| 90| 99| 1|\n+-------+------+------------------+------------------+-------------------+\n\nCalculate Co variance between a few columns to understand features to use\nCovariance between income and hours_per_week is 1.2\nCovariance between income and age is 1.4"
|
||||
"text": "Select few columns to see the data\n+------+---+--------------+---------+\n|income|age|hours_per_week|education|\n+------+---+--------------+---------+\n| <=50K| 39| 40|Bachelors|\n| <=50K| 50| 13|Bachelors|\n| <=50K| 38| 40| HS-grad|\n| <=50K| 53| 40| 11th|\n| <=50K| 28| 40|Bachelors|\n| <=50K| 37| 40| Masters|\n| <=50K| 49| 16| 9th|\n| >50K| 52| 45| HS-grad|\n| >50K| 31| 50| Masters|\n| >50K| 42| 40|Bachelors|\n+------+---+--------------+---------+\nonly showing top 10 rows\n\nNumber of distinct values for income\n+------+\n|income|\n+------+\n| <=50K|\n| >50K|\n+------+\n\nAdded numeric column(income_code) derived from income column\n+------+---+--------------+---------+-----------+\n|income|age|hours_per_week|education|income_code|\n+------+---+--------------+---------+-----------+\n| <=50K| 39| 40|Bachelors| 0|\n| <=50K| 50| 13|Bachelors| 0|\n| <=50K| 38| 40| HS-grad| 0|\n| <=50K| 53| 40| 11th| 0|\n| <=50K| 28| 40|Bachelors| 0|\n| <=50K| 37| 40| Masters| 0|\n| <=50K| 49| 16| 9th| 0|\n| >50K| 52| 45| HS-grad| 1|\n| >50K| 31| 50| Masters| 1|\n| >50K| 42| 40|Bachelors| 1|\n+------+---+--------------+---------+-----------+\nonly showing top 10 rows\n\nPrint a statistical summary of a few columns\n+-------+------+------------------+------------------+------------+-------------------+\n|summary|income| age| hours_per_week| education| income_code|\n+-------+------+------------------+------------------+------------+-------------------+\n| count| 32561| 32561| 32561| 32561| 32561|\n| mean| null| 38.58164675532078|40.437455852092995| null| 0.2408095574460244|\n| stddev| null|13.640432553581356|12.347428681731838| null|0.42758148856469247|\n| min| <=50K| 17| 1| 10th| 0|\n| max| >50K| 90| 99|Some-college| 1|\n+-------+------+------------------+------------------+------------+-------------------+\n\nCalculate Co variance between a few columns to understand features to use\nCovariance between income and hours_per_week is 1.2\nCovariance between income and age is 1.4",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 4
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "# Choose feature columns and the label column.\r\nlabel = \"income\"\r\nxvars = [\"age\", \"hours_per_week\"] #all numeric\r\n\r\nprint(\"label = {}\".format(label))\r\nprint(\"features = {}\".format(xvars))\r\n\r\n#Check label counts to check data bias\r\nprint(\"Count of rows that are <=50K\", data_all[data_all.income==\"<=50K\"].count())\r\nprint(\"Count of rows that are >50K\", data_all[data_all.income==\">50K\"].count())\r\n\r\n\r\nselect_cols = xvars\r\nselect_cols.append(label)\r\ndata = data_all.select(select_cols)",
|
||||
"source": "# Choose feature columns and the label column.\r\nlabel = \"income\"\r\nxvars = [\"age\", \"hours_per_week\", 'education'] #numeric and string\r\n\r\nprint(\"label = {}\".format(label))\r\nprint(\"features = {}\".format(xvars))\r\n\r\n#Check label counts to check data bias\r\nprint(\"Count of rows that are <=50K\", data_all[data_all.income==\"<=50K\"].count())\r\nprint(\"Count of rows that are >50K\", data_all[data_all.income==\">50K\"].count())\r\n\r\n\r\nselect_cols = xvars\r\nselect_cols.append(label)\r\ndata = data_all.select(select_cols)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "label = income\nfeatures = ['age', 'hours_per_week']\nCount of rows that are <=50K 24720\nCount of rows that are >50K 7841"
|
||||
"text": "label = income\nfeatures = ['age', 'hours_per_week', 'education']\nCount of rows that are <=50K 24720\nCount of rows that are >50K 7841",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 5
|
||||
@@ -77,27 +95,27 @@
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "train (24469, 3)\ntest (8092, 3)\ntrain and test datasets saved to /spark_ml/AdultCensusIncomeTrain and /spark_ml/AdultCensusIncomeTest"
|
||||
"text": "train (24469, 4)\ntest (8092, 4)\ntrain and test datasets saved to /spark_ml/AdultCensusIncomeTrain and /spark_ml/AdultCensusIncomeTest",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 6
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": "## Step 3 - Train a model\r\n[Spark ML pipeline] ( https://spark.apache.org/docs/2.3.1/ml-pipeline.html ) allow to sequence all steps as a workflow and make it easier to experiment with various algorithms and their parameters. The following code first constructs the stages and then puts these stages together in Ml pipeline. LogisticRegression is used to create the model.\r\n\r\n",
|
||||
"source": "## Step 3 - Train a model\r\n[Spark ML pipelines](https://spark.apache.org/docs/latest/ml-pipeline.html) allow to sequence all steps as a workflow and make it easier to experiment with various algorithms and their parameters. The following code first constructs the stages and then puts these stages together in Ml pipeline. LogisticRegression is used to create the model.",
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": "from pyspark.ml import Pipeline, PipelineModel\r\nfrom pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler\r\nfrom pyspark.ml.classification import LogisticRegression\r\n\r\nreg = 0.1\r\nprint(\"Using LogisticRegression model with Regularization Rate of {}.\".format(reg))\r\n\r\n# create a new Logistic Regression model.\r\nlr = LogisticRegression(regParam=reg)\r\n\r\ndtypes = dict(train.dtypes)\r\ndtypes.pop(label)\r\n\r\nsi_xvars = []\r\nohe_xvars = []\r\nfeatureCols = []\r\nfor idx,key in enumerate(dtypes):\r\n if dtypes[key] == \"string\":\r\n featureCol = \"-\".join([key, \"encoded\"])\r\n featureCols.append(featureCol)\r\n \r\n tmpCol = \"-\".join([key, \"tmp\"])\r\n si_xvars.append(StringIndexer(inputCol=key, outputCol=tmpCol, handleInvalid=\"skip\")) #, handleInvalid=\"keep\"\r\n ohe_xvars.append(OneHotEncoder(inputCol=tmpCol, outputCol=featureCol))\r\n else:\r\n featureCols.append(key)\r\n\r\n# string-index the label column into a column named \"label\"\r\nsi_label = StringIndexer(inputCol=label, outputCol='label')\r\n\r\n# assemble the encoded feature columns in to a column named \"features\"\r\nassembler = VectorAssembler(inputCols=featureCols, outputCol=\"features\")\r\n\r\n\r\nstages = []\r\nstages.extend(si_xvars)\r\nstages.extend(ohe_xvars)\r\nstages.append(si_label)\r\nstages.append(assembler)\r\nstages.append(lr)\r\npipe = Pipeline(stages=stages)\r\nprint(\"Pipeline Created\")\r\n\r\nmodel = pipe.fit(train)\r\nprint(\"Model Trained\")\r\nprint(\"Model is \", model)\r\nprint(\"Model Stages\", model.stages)",
|
||||
"source": "from pyspark.ml import Pipeline, PipelineModel\r\nfrom pyspark.ml.feature import OneHotEncoderEstimator, StringIndexer, VectorAssembler\r\nfrom pyspark.ml.classification import LogisticRegression\r\n\r\nreg = 0.1\r\nprint(\"Using LogisticRegression model with Regularization Rate of {}.\".format(reg))\r\n\r\n# create a new Logistic Regression model.\r\nlr = LogisticRegression(regParam=reg)\r\n\r\ndtypes = dict(train.dtypes)\r\ndtypes.pop(label)\r\n\r\nsi_xvars = []\r\nohe_xvars = []\r\nfeatureCols = []\r\nfor idx,key in enumerate(dtypes):\r\n if dtypes[key] == \"string\":\r\n featureCol = \"-\".join([key, \"encoded\"])\r\n featureCols.append(featureCol)\r\n \r\n tmpCol = \"-\".join([key, \"tmp\"])\r\n si_xvars.append(StringIndexer(inputCol=key, outputCol=tmpCol, handleInvalid=\"skip\")) #, handleInvalid=\"keep\"\r\n ohe_xvars.append(OneHotEncoderEstimator(inputCols=[tmpCol], outputCols=[featureCol]))\r\n else:\r\n featureCols.append(key)\r\n\r\n# string-index the label column into a column named \"label\"\r\nsi_label = StringIndexer(inputCol=label, outputCol='label')\r\n\r\n# assemble the encoded feature columns in to a column named \"features\"\r\nassembler = VectorAssembler(inputCols=featureCols, outputCol=\"features\")\r\n\r\n\r\nstages = []\r\nstages.extend(si_xvars)\r\nstages.extend(ohe_xvars)\r\nstages.append(si_label)\r\nstages.append(assembler)\r\nstages.append(lr)\r\npipe = Pipeline(stages=stages)\r\nprint(\"Pipeline Created\")\r\n\r\nmodel = pipe.fit(train)\r\nprint(\"Model Trained\")\r\nprint(\"Model is \", model)\r\nprint(\"Model Stages\", model.stages)",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Using LogisticRegression model with Regularization Rate of 0.1.\nPipeline Created\nModel Trained\nModel is PipelineModel_e5284bc61285\nModel Stages [StringIndexer_1ecf86c8d2ae, VectorAssembler_450ee37e6955, LogisticRegressionModel: uid = LogisticRegression_deb52c17940d, numClasses = 2, numFeatures = 2]"
|
||||
"text": "Using LogisticRegression model with Regularization Rate of 0.1.\nPipeline Created\nModel Trained\nModel is PipelineModel_8c7a4fdc6110\nModel Stages [StringIndexer_7d244350f55d, OneHotEncoderEstimator_559780c5ce92, StringIndexer_53280e6349e6, VectorAssembler_1051507100cb, LogisticRegressionModel: uid = LogisticRegression_5c0eda4eab78, numClasses = 2, numFeatures = 17]",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 7
|
||||
@@ -113,9 +131,9 @@
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "Area under ROC: 0.7363559303440261\nArea Under PR: 0.39475773290351296\n+------+-----+----------+\n|income|label|prediction|\n+------+-----+----------+\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n+------+-----+----------+\nonly showing top 20 rows"
|
||||
"text": "Area under ROC: 0.7964496884726682\nArea Under PR: 0.5358180243123482\n+------+-----+----------+\n|income|label|prediction|\n+------+-----+----------+\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n| <=50K| 0.0| 1.0|\n| >50K| 1.0| 1.0|\n+------+-----+----------+\nonly showing top 20 rows",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 8
|
||||
@@ -131,9 +149,9 @@
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "saved model to /spark_ml/AdultCensus.mml\nSuccessfully loaded from /spark_ml/AdultCensus.mml"
|
||||
"text": "saved model to /spark_ml/AdultCensus.mml\nSuccessfully loaded from /spark_ml/AdultCensus.mml",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 9
|
||||
@@ -149,9 +167,9 @@
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": "persist the mleap bundle from local to hdfs"
|
||||
"text": "persist the mleap bundle from local to hdfs",
|
||||
"output_type": "stream"
|
||||
}
|
||||
],
|
||||
"execution_count": 10
|
||||
|
||||
Reference in New Issue
Block a user