<?xml version="1.0" encoding="utf-8" standalone="yes" ?>

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">

<channel>

<title>spatialworks</title>

<link>https://rapidsurveys.io/</link>

<atom:link href="https://rapidsurveys.io/index.xml" rel="self" type="application/rss+xml" />

<description>spatialworks</description>

<generator>Hugo -- gohugo.io</generator><language>en-gb</language><lastBuildDate>Thu, 25 Jun 2020 00:00:00 +0000</lastBuildDate>

<item>

<title>Build a model</title>

<link>https://rapidsurveys.io/start/models/</link>

<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>

<guid>https://rapidsurveys.io/start/models/</guid>

<description>&lt;h2 id=&#34;intro&#34;&gt;Introduction&lt;/h2&gt;

&lt;p&gt;How do you create a statistical model using tidymodels? In this article, we will walk you through the steps. We start with data for modeling, learn how to specify and train models with different engines using the

&lt;a href=&#34;https://tidymodels.github.io/parsnip/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;parsnip package&lt;/a&gt;, and understand why these functions are designed this way.&lt;/p&gt;

&lt;p&gt;To use code in this article, you will need to install the following packages: readr, rstanarm, and tidymodels.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(tidymodels) &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# for the parsnip package, along with the rest of tidymodels&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Helper packages&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(readr) &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# for importing data&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;data&#34;&gt;The Sea Urchins Data&lt;/h2&gt;

&lt;p&gt;Let&amp;rsquo;s use the data from

&lt;a href=&#34;https://link.springer.com/article/10.1007/BF00349318&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Constable (1993)&lt;/a&gt; to explore how three different feeding regimes affect the size of sea urchins over time. The initial size of the sea urchins at the beginning of the experiment probably affects how big they grow as they are fed.&lt;/p&gt;

&lt;p&gt;To start, let&amp;rsquo;s read our urchins data into R, which we&amp;rsquo;ll do by providing

&lt;a href=&#34;https://readr.tidyverse.org/reference/read_delim.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;code&gt;readr::read_csv()&lt;/code&gt;&lt;/a&gt; with a url where our CSV data is located (&amp;ldquo;&lt;a href=&#34;https://tidymodels.org/start/models/urchins.csv&#34;&gt;https://tidymodels.org/start/models/urchins.csv&lt;/a&gt;&amp;rdquo;):&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;urchins &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Data were assembled for a tutorial &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# at https://www.flutterbys.com.au/stats/tut/tut7.5a.html&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;read_csv&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;https://tidymodels.org/start/models/urchins.csv&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Change the names to be a little more verbose&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;setNames&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;c&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;food_regime&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;initial_volume&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;width&amp;#34;&lt;/span&gt;)) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Factors are very helpful for modeling, so we convert one column&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mutate&lt;/span&gt;(food_regime &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;factor&lt;/span&gt;(food_regime, levels &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;c&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Initial&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Low&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;High&amp;#34;&lt;/span&gt;)))

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Parsed with column specification:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; cols(&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; TREAT = col_character(),&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; IV = col_double(),&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; SUTW = col_double()&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; )&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Let&amp;rsquo;s take a quick look at the data:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;urchins

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 72 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; food_regime initial_volume width&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;fct&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 Initial 3.5 0.01 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 Initial 5 0.02 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 Initial 8 0.061&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 Initial 10 0.051&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 Initial 13 0.041&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 Initial 13 0.061&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 7 Initial 15 0.041&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 8 Initial 15 0.071&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 9 Initial 16 0.092&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 10 Initial 17 0.051&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # … with 62 more rows&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The urchins data is a

&lt;a href=&#34;https://tibble.tidyverse.org/index.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;tibble&lt;/a&gt;. If you are new to tibbles, the best place to start is the

&lt;a href=&#34;https://r4ds.had.co.nz/tibbles.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;tibbles chapter&lt;/a&gt; in &lt;em&gt;R for Data Science&lt;/em&gt;. For each of the 72 urchins, we know their:&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;experimental feeding regime group (&lt;code&gt;food_regime&lt;/code&gt;: either &lt;code&gt;Initial&lt;/code&gt;, &lt;code&gt;Low&lt;/code&gt;, or &lt;code&gt;High&lt;/code&gt;),&lt;/li&gt;

&lt;li&gt;size in milliliters at the start of the experiment (&lt;code&gt;initial_volume&lt;/code&gt;), and&lt;/li&gt;

&lt;li&gt;suture width at the end of the experiment (&lt;code&gt;width&lt;/code&gt;).&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;As a first step in modeling, it&amp;rsquo;s always a good idea to plot the data:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(urchins,

&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; initial_volume,

y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; width,

group &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; food_regime,

col &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; food_regime)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_smooth&lt;/span&gt;(method &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; lm, se &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;FALSE&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;scale_color_viridis_d&lt;/span&gt;(option &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;plasma&amp;#34;&lt;/span&gt;, end &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.7&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; `geom_smooth()` using formula &amp;#39;y ~ x&amp;#39;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/urchin-plot-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;p&gt;We can see that urchins that were larger in volume at the start of the experiment tended to have wider sutures at the end, but the slopes of the lines look different so this effect may depend on the feeding regime condition.&lt;/p&gt;

&lt;h2 id=&#34;build-model&#34;&gt;Build and fit a model&lt;/h2&gt;

&lt;p&gt;A standard two-way analysis of variance (

&lt;a href=&#34;https://www.itl.nist.gov/div898/handbook/prc/section4/prc43.htm&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;ANOVA&lt;/a&gt;) model makes sense for this dataset because we have both a continuous predictor and a categorical predictor. Since the slopes appear to be different for at least two of the feeding regimes, let&amp;rsquo;s build a model that allows for two-way interactions. Specifying an R formula with our variables in this way:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;width &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; initial_volume &lt;span style=&#34;color:#666&#34;&gt;*&lt;/span&gt; food_regime

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;allows our regression model depending on initial volume to have separate slopes and intercepts for each food regime.&lt;/p&gt;

&lt;p&gt;For this kind of model, ordinary least squares is a good initial approach. With tidymodels, we start by specifying the &lt;em&gt;functional form&lt;/em&gt; of the model that we want using the

&lt;a href=&#34;https://tidymodels.github.io/parsnip/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;parsnip package&lt;/a&gt;. Since there is a numeric outcome and the model should be linear with slopes and intercepts, the model type is

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/linear_reg.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&amp;ldquo;linear regression&amp;rdquo;&lt;/a&gt;. We can declare this with:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;linear_reg&lt;/span&gt;()

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Linear Regression Model Specification (regression)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That is pretty underwhelming since, on its own, it doesn&amp;rsquo;t really do much. However, now that the type of model has been specified, a method for &lt;em&gt;fitting&lt;/em&gt; or training the model can be stated using the &lt;strong&gt;engine&lt;/strong&gt;. The engine value is often a mash-up of the software that can be used to fit or train the model as well as the estimation method. For example, to use ordinary least squares, we can set the engine to be &lt;code&gt;lm&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;linear_reg&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;lm&amp;#34;&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Linear Regression Model Specification (regression)&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Computational engine: lm&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/linear_reg.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;documentation page for &lt;code&gt;linear_reg()&lt;/code&gt;&lt;/a&gt; lists the possible engines. We&amp;rsquo;ll save this model object as &lt;code&gt;lm_mod&lt;/code&gt;.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;lm_mod &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;linear_reg&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;lm&amp;#34;&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;From here, the model can be estimated or trained using the

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/fit.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;code&gt;fit()&lt;/code&gt;&lt;/a&gt; function:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;lm_fit &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

lm_mod &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(width &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; initial_volume &lt;span style=&#34;color:#666&#34;&gt;*&lt;/span&gt; food_regime, data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; urchins)

lm_fit

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 3ms &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; stats::lm(formula = formula, data = data)&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Coefficients:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; (Intercept) initial_volume &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 0.0331216 0.0015546 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; food_regimeLow food_regimeHigh &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 0.0197824 0.0214111 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; initial_volume:food_regimeLow initial_volume:food_regimeHigh &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; -0.0012594 0.0005254&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Perhaps our analysis requires a description of the model parameter estimates and their statistical properties. Although the &lt;code&gt;summary()&lt;/code&gt; function for &lt;code&gt;lm&lt;/code&gt; objects can provide that, it gives the results back in an unwieldy format. Many models have a &lt;code&gt;tidy()&lt;/code&gt; method that provides the summary results in a more predictable and useful format (e.g. a data frame with standard column names):&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;tidy&lt;/span&gt;(lm_fit)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 6 x 5&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; term estimate std.error statistic p.value&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 (Intercept) 0.0331 0.00962 3.44 0.00100 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 initial_volume 0.00155 0.000398 3.91 0.000222&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 food_regimeLow 0.0198 0.0130 1.52 0.133 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 food_regimeHigh 0.0214 0.0145 1.47 0.145 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 initial_volume:food_regimeLow -0.00126 0.000510 -2.47 0.0162 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 initial_volume:food_regimeHigh 0.000525 0.000702 0.748 0.457&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;predict-model&#34;&gt;Use a model to predict&lt;/h2&gt;

&lt;p&gt;This fitted object &lt;code&gt;lm_fit&lt;/code&gt; has the &lt;code&gt;lm&lt;/code&gt; model output built-in, which you can access with &lt;code&gt;lm_fit$fit&lt;/code&gt;, but there are some benefits to using the fitted parsnip model object when it comes to predicting.&lt;/p&gt;

&lt;p&gt;Suppose that, for a publication, it would be particularly interesting to make a plot of the mean body size for urchins that started the experiment with an initial volume of 20ml. To create such a graph, we start with some new example data that we will make predictions for, to show in our graph:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;new_points &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;expand.grid&lt;/span&gt;(initial_volume &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;20&lt;/span&gt;,

food_regime &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;c&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Initial&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Low&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;High&amp;#34;&lt;/span&gt;))

new_points

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; initial_volume food_regime&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 20 Initial&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 20 Low&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 20 High&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;To get our predicted results, we can use the &lt;code&gt;predict()&lt;/code&gt; function to find the mean values at 20ml.&lt;/p&gt;

&lt;p&gt;It is also important to communicate the variability, so we also need to find the predicted confidence intervals. If we had used &lt;code&gt;lm()&lt;/code&gt; to fit the model directly, a few minutes of reading the

&lt;a href=&#34;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/predict.lm.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;documentation page&lt;/a&gt; for &lt;code&gt;predict.lm()&lt;/code&gt; would explain how to do this. However, if we decide to use a different model to estimate urchin size (&lt;em&gt;spoiler:&lt;/em&gt; we will!), it is likely that a completely different syntax would be required.&lt;/p&gt;

&lt;p&gt;Instead, with tidymodels, the types of predicted values are standardized so that we can use the same syntax to get these values.&lt;/p&gt;

&lt;p&gt;First, let&amp;rsquo;s generate the mean body width values:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;mean_pred &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(lm_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; new_points)

mean_pred

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 3 x 1&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .pred&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 0.0642&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 0.0588&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 0.0961&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;When making predictions, the tidymodels convention is to always produce a tibble of results with standardized column names. This makes it easy to combine the original data and the predictions in a usable format:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;conf_int_pred &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(lm_fit,

new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; new_points,

type &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;conf_int&amp;#34;&lt;/span&gt;)

conf_int_pred

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 3 x 2&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .pred_lower .pred_upper&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 0.0555 0.0729&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 0.0499 0.0678&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 0.0870 0.105&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Now combine: &lt;/span&gt;

plot_data &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

new_points &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(mean_pred) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(conf_int_pred)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# and plot:&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(plot_data, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; food_regime)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_errorbar&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(ymin &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred_lower,

ymax &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred_upper),

width &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.2&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;labs&lt;/span&gt;(y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;urchin size&amp;#34;&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/lm-all-pred-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;new-engine&#34;&gt;Model with a different engine&lt;/h2&gt;

&lt;p&gt;Every one on your team is happy with that plot &lt;em&gt;except&lt;/em&gt; that one person who just read their first book on

&lt;a href=&#34;https://bayesian.org/what-is-bayesian-analysis/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Bayesian analysis&lt;/a&gt;. They are interested in knowing if the results would be different if the model were estimated using a Bayesian approach. In such an analysis, a

&lt;a href=&#34;https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;em&gt;prior distribution&lt;/em&gt;&lt;/a&gt; needs to be declared for each model parameter that represents the possible values of the parameters (before being exposed to the observed data). After some discussion, the group agrees that the priors should be bell-shaped but, since no one has any idea what the range of values should be, to take a conservative approach and make the priors &lt;em&gt;wide&lt;/em&gt; using a Cauchy distribution (which is the same as a t-distribution with a single degree of freedom).&lt;/p&gt;

&lt;p&gt;The

&lt;a href=&#34;https://mc-stan.org/rstanarm/articles/priors.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;documentation&lt;/a&gt; on the rstanarm package shows us that the &lt;code&gt;stan_glm()&lt;/code&gt; function can be used to estimate this model, and that the function arguments that need to be specified are called &lt;code&gt;prior&lt;/code&gt; and &lt;code&gt;prior_intercept&lt;/code&gt;. It turns out that &lt;code&gt;linear_reg()&lt;/code&gt; has a

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/linear_reg.html#details&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;code&gt;stan&lt;/code&gt; engine&lt;/a&gt;. Since these prior distribution arguments are specific to the Stan software, they are passed as arguments to

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/set_engine.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;code&gt;parsnip::set_engine()&lt;/code&gt;&lt;/a&gt;. After that, the same exact &lt;code&gt;fit()&lt;/code&gt; call is used:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# set the prior distribution&lt;/span&gt;

prior_dist &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; rstanarm&lt;span style=&#34;color:#666&#34;&gt;::&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;student_t&lt;/span&gt;(df &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;)

&lt;span style=&#34;color:#00f&#34;&gt;set.seed&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;123&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# make the parsnip model&lt;/span&gt;

bayes_mod &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;linear_reg&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;stan&amp;#34;&lt;/span&gt;,

prior_intercept &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; prior_dist,

prior &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; prior_dist)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# train the model&lt;/span&gt;

bayes_fit &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

bayes_mod &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(width &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; initial_volume &lt;span style=&#34;color:#666&#34;&gt;*&lt;/span&gt; food_regime, data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; urchins)

&lt;span style=&#34;color:#00f&#34;&gt;print&lt;/span&gt;(bayes_fit, digits &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;5&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 1.5s &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; stan_glm&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; family: gaussian [identity]&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; formula: width ~ initial_volume * food_regime&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; observations: 72&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; predictors: 6&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ------&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Median MAD_SD &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; (Intercept) 0.03452 0.00883&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; initial_volume 0.00150 0.00037&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; food_regimeLow 0.01805 0.01221&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; food_regimeHigh 0.01934 0.01367&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; initial_volume:food_regimeLow -0.00119 0.00047&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; initial_volume:food_regimeHigh 0.00061 0.00065&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Auxiliary parameter(s):&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Median MAD_SD &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; sigma 0.02121 0.00186&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ------&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; * For help interpreting the printed output see ?print.stanreg&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; * For info on the priors used see ?prior_summary.stanreg&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This kind of Bayesian analysis (like many models) involves randomly generated numbers in its fitting procedure. We can use &lt;code&gt;set.seed()&lt;/code&gt; to ensure that the same (pseudo-)random numbers are generated each time we run this code. The number &lt;code&gt;123&lt;/code&gt; isn&amp;rsquo;t special or related to our data; it is just a &amp;ldquo;seed&amp;rdquo; used to choose random numbers.&lt;/p&gt;

&lt;p&gt;To update the parameter table, the &lt;code&gt;tidy()&lt;/code&gt; method is once again used:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;tidy&lt;/span&gt;(bayes_fit, intervals &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;TRUE&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 6 x 5&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; term estimate std.error lower upper&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 (Intercept) 0.0345 0.00883 0.0200 0.0490 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 initial_volume 0.00150 0.000369 0.000895 0.00212 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 food_regimeLow 0.0181 0.0122 -0.00181 0.0380 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 food_regimeHigh 0.0193 0.0137 -0.00317 0.0420 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 initial_volume:food_regimeLow -0.00119 0.000472 -0.00199 -0.000413&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 initial_volume:food_regimeHigh 0.000610 0.000651 -0.000490 0.00170&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A goal of the tidymodels packages is that the &lt;strong&gt;interfaces to common tasks are standardized&lt;/strong&gt; (as seen in the &lt;code&gt;tidy()&lt;/code&gt; results above). The same is true for getting predictions; we can use the same code even though the underlying packages use very different syntax:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;bayes_plot_data &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

new_points &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(bayes_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; new_points)) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(bayes_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; new_points, type &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;conf_int&amp;#34;&lt;/span&gt;))

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(bayes_plot_data, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; food_regime)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_errorbar&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(ymin &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred_lower, ymax &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred_upper), width &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.2&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;labs&lt;/span&gt;(y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;urchin size&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;ggtitle&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Bayesian model with t(1) prior distribution&amp;#34;&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/stan-pred-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;p&gt;This isn&amp;rsquo;t very different from the non-Bayesian results (except in interpretation).&lt;/p&gt;

&lt;div class=&#34;note&#34;&gt;The &lt;a href=&#34;https://parsnip.tidymodels.org/&#34;&gt;parsnip&lt;/a&gt; package can work with many model types, engines, and arguments. Check out &lt;a href=&#34;https://rapidsurveys.io/find/parsnip/&#34;&gt;tidymodels.org/find/parsnip&lt;/a&gt; to see what is available.&lt;/div&gt;

&lt;h2 id=&#34;why&#34;&gt;Why does it work that way?&lt;/h2&gt;

&lt;p&gt;The extra step of defining the model using a function like &lt;code&gt;linear_reg()&lt;/code&gt; might seem superfluous since a call to &lt;code&gt;lm()&lt;/code&gt; is much more succinct. However, the problem with standard modeling functions is that they don&amp;rsquo;t separate what you want to do from the execution. For example, the process of executing a formula has to happen repeatedly across model calls even when the formula does not change; we can&amp;rsquo;t recycle those computations.&lt;/p&gt;

&lt;p&gt;Also, using the tidymodels framework, we can do some interesting things by incrementally creating a model (instead of using single function call).

&lt;a href=&#34;https://rapidsurveys.io/start/tuning/&#34;&gt;Model tuning&lt;/a&gt; with tidymodels uses the specification of the model to declare what parts of the model should be tuned. That would be very difficult to do if &lt;code&gt;linear_reg()&lt;/code&gt; immediately fit the model.&lt;/p&gt;

&lt;p&gt;If you are familiar with the tidyverse, you may have noticed that our modeling code uses the magrittr pipe (&lt;code&gt;%&amp;gt;%&lt;/code&gt;). With dplyr and other tidyverse packages, the pipe works well because all of the functions take the &lt;em&gt;data&lt;/em&gt; as the first argument. For example:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;urchins &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;group_by&lt;/span&gt;(food_regime) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;summarize&lt;/span&gt;(med_vol &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;median&lt;/span&gt;(initial_volume))

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 3 x 2&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; food_regime med_vol&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;fct&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 Initial 20.5&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 Low 19.2&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 High 15&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;whereas the modeling code uses the pipe to pass around the &lt;em&gt;model object&lt;/em&gt;:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;bayes_mod &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(width &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; initial_volume &lt;span style=&#34;color:#666&#34;&gt;*&lt;/span&gt; food_regime, data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; urchins)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This may seem jarring if you have used dplyr a lot, but it is extremely similar to how ggplot2 operates:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(urchins,

&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(initial_volume, width)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# returns a ggplot object &lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_jitter&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# same&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_smooth&lt;/span&gt;(method &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; lm, se &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;FALSE&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# same &lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;labs&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Volume&amp;#34;&lt;/span&gt;, y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Width&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# etc&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;session-info&#34;&gt;Session information&lt;/h2&gt;

&lt;pre&gt;&lt;code&gt;#&amp;gt; ─ Session info ───────────────────────────────────────────────────────────────

#&amp;gt; setting value

#&amp;gt; version R version 4.0.0 (2020-04-24)

#&amp;gt; os macOS Mojave 10.14.6

#&amp;gt; system x86_64, darwin17.0

#&amp;gt; ui X11

#&amp;gt; language (EN)

#&amp;gt; collate en_US.UTF-8

#&amp;gt; ctype en_US.UTF-8

#&amp;gt; tz America/New_York

#&amp;gt; date 2020-05-19

#&amp;gt;

#&amp;gt; ─ Packages ───────────────────────────────────────────────────────────────────

#&amp;gt; package * version date lib source

#&amp;gt; broom * 0.5.6 2020-04-20 [1] CRAN (R 4.0.0)

#&amp;gt; dials * 0.0.6 2020-04-03 [1] CRAN (R 4.0.0)

#&amp;gt; dplyr * 0.8.5 2020-03-07 [1] CRAN (R 4.0.0)

#&amp;gt; ggplot2 * 3.3.0 2020-03-05 [1] CRAN (R 4.0.0)

#&amp;gt; infer * 0.5.1 2019-11-19 [1] CRAN (R 4.0.0)

#&amp;gt; parsnip * 0.1.1 2020-05-06 [1] CRAN (R 4.0.0)

#&amp;gt; purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.0.0)

#&amp;gt; readr * 1.3.1 2018-12-21 [1] CRAN (R 4.0.0)

#&amp;gt; recipes * 0.1.12 2020-05-01 [1] CRAN (R 4.0.0)

#&amp;gt; rlang 0.4.6 2020-05-02 [1] CRAN (R 4.0.0)

#&amp;gt; rsample * 0.0.6 2020-03-31 [1] CRAN (R 4.0.0)

#&amp;gt; rstanarm * 2.19.3 2020-02-11 [1] CRAN (R 4.0.0)

#&amp;gt; tibble * 3.0.1 2020-04-20 [1] CRAN (R 4.0.0)

#&amp;gt; tidymodels * 0.1.0 2020-02-16 [1] CRAN (R 4.0.0)

#&amp;gt; tune * 0.1.0 2020-04-02 [1] CRAN (R 4.0.0)

#&amp;gt; workflows * 0.1.1 2020-03-17 [1] CRAN (R 4.0.0)

#&amp;gt; yardstick * 0.0.6 2020-03-17 [1] CRAN (R 4.0.0)

#&amp;gt;

#&amp;gt; [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

&lt;/code&gt;&lt;/pre&gt;</description>

</item>

<item>

<title>Regression models two ways</title>

<link>https://rapidsurveys.io/learn/models/parsnip-ranger-glmnet/</link>

<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>

<guid>https://rapidsurveys.io/learn/models/parsnip-ranger-glmnet/</guid>

<description>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;

&lt;p&gt;To use the code in this article, you will need to install the following packages: AmesHousing, glmnet, randomForest, ranger, and tidymodels.&lt;/p&gt;

&lt;p&gt;We can create regression models with the tidymodels package

&lt;a href=&#34;https://tidymodels.github.io/parsnip/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;parsnip&lt;/a&gt; to predict continuous or numeric quantities. Here, let&amp;rsquo;s first fit a random forest model, which does &lt;em&gt;not&lt;/em&gt; require all numeric input (see discussion

&lt;a href=&#34;https://bookdown.org/max/FES/categorical-trees.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;here&lt;/a&gt;) and discuss how to use &lt;code&gt;fit()&lt;/code&gt; and &lt;code&gt;fit_xy()&lt;/code&gt;, as well as &lt;em&gt;data descriptors&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Second, let&amp;rsquo;s fit a regularized linear regression model to demonstrate how to move between different types of models using parsnip.&lt;/p&gt;

&lt;h2 id=&#34;the-ames-housing-data&#34;&gt;The Ames housing data&lt;/h2&gt;

&lt;p&gt;We&amp;rsquo;ll use the Ames housing data set to demonstrate how to create regression models using parsnip. First, set up the data set and create a simple training/test set split:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(AmesHousing)

ames &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;make_ames&lt;/span&gt;()

&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(tidymodels)

&lt;span style=&#34;color:#00f&#34;&gt;set.seed&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;4595&lt;/span&gt;)

data_split &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;initial_split&lt;/span&gt;(ames, strata &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Sale_Price&amp;#34;&lt;/span&gt;, p &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0.75&lt;/span&gt;)

ames_train &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;training&lt;/span&gt;(data_split)

ames_test &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;testing&lt;/span&gt;(data_split)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The use of the test set here is &lt;em&gt;only for illustration&lt;/em&gt;; normally in a data analysis these data would be saved to the very end after many models have been evaluated.&lt;/p&gt;

&lt;h2 id=&#34;random-forest&#34;&gt;Random forest&lt;/h2&gt;

&lt;p&gt;We&amp;rsquo;ll start by fitting a random forest model to a small set of parameters. Let&amp;rsquo;s create a model with the predictors &lt;code&gt;Longitude&lt;/code&gt;, &lt;code&gt;Latitude&lt;/code&gt;, &lt;code&gt;Lot_Area&lt;/code&gt;, &lt;code&gt;Neighborhood&lt;/code&gt;, and &lt;code&gt;Year_Sold&lt;/code&gt;. A simple random forest model can be specified via:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;rf_defaults &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;rand_forest&lt;/span&gt;(mode &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;regression&amp;#34;&lt;/span&gt;)

rf_defaults

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Random Forest Model Specification (regression)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The model will be fit with the ranger package by default. Since we didn&amp;rsquo;t add any extra arguments to &lt;code&gt;fit&lt;/code&gt;, &lt;em&gt;many&lt;/em&gt; of the arguments will be set to their defaults from the function &lt;code&gt;ranger::ranger()&lt;/code&gt;. The help pages for the model function describe the default parameters and you can also use the &lt;code&gt;translate()&lt;/code&gt; function to check out such details.&lt;/p&gt;

&lt;p&gt;The parsnip package provides two different interfaces to fit a model:&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;the formula interface (&lt;code&gt;fit()&lt;/code&gt;), and&lt;/li&gt;

&lt;li&gt;the non-formula interface (&lt;code&gt;fit_xy()&lt;/code&gt;).&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;Let&amp;rsquo;s start with the non-formula interface:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;preds &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;c&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Longitude&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Latitude&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Lot_Area&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Neighborhood&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Year_Sold&amp;#34;&lt;/span&gt;)

rf_xy_fit &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

rf_defaults &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;ranger&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit_xy&lt;/span&gt;(

x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train[, preds],

y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;log10&lt;/span&gt;(ames_train&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;Sale_Price)

)

rf_xy_fit

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 952ms &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Ranger result&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ranger::ranger(formula = formula, data = data, num.threads = 1, verbose = FALSE, seed = sample.int(10^5, 1)) &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Type: Regression &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of trees: 500 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Sample size: 2199 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of independent variables: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Mtry: 2 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Target node size: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Variable importance mode: none &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Splitrule: variance &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; OOB prediction error (MSE): 0.00844 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; R squared (OOB): 0.736&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The non-formula interface doesn&amp;rsquo;t do anything to the predictors before passing them to the underlying model function. This particular model does &lt;em&gt;not&lt;/em&gt; require indicator variables (sometimes called &amp;ldquo;dummy variables&amp;rdquo;) to be created prior to fitting the model. Note that the output shows &amp;ldquo;Number of independent variables: 5&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;For regression models, we can use the basic &lt;code&gt;predict()&lt;/code&gt; method, which returns a tibble with a column named &lt;code&gt;.pred&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;test_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

ames_test &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;select&lt;/span&gt;(Sale_Price) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mutate&lt;/span&gt;(Sale_Price &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;log10&lt;/span&gt;(Sale_Price)) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(rf_xy_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_test[, preds])

)

test_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;slice&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;5&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 5 x 2&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Sale_Price .pred&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 5.33 5.22&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 5.02 5.21&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 5.27 5.25&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 5.60 5.51&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 5.28 5.24&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# summarize performance&lt;/span&gt;

test_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;metrics&lt;/span&gt;(truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Sale_Price, estimate &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 3 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .metric .estimator .estimate&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 rmse standard 0.0914&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 rsq standard 0.717 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 mae standard 0.0662&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Note that:&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;If the model required indicator variables, we would have to create them manually prior to using &lt;code&gt;fit()&lt;/code&gt; (perhaps using the recipes package).&lt;/li&gt;

&lt;li&gt;We had to manually log the outcome prior to modeling.&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;Now, for illustration, let&amp;rsquo;s use the formula method using some new parameter values:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;rand_forest&lt;/span&gt;(mode &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;regression&amp;#34;&lt;/span&gt;, mtry &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;3&lt;/span&gt;, trees &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1000&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;ranger&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;log10&lt;/span&gt;(Sale_Price) &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; Longitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Latitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Lot_Area &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Neighborhood &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Year_Sold,

data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train

)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 2.6s &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Ranger result&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ranger::ranger(formula = formula, data = data, mtry = ~3, num.trees = ~1000, num.threads = 1, verbose = FALSE, seed = sample.int(10^5, 1)) &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Type: Regression &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of trees: 1000 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Sample size: 2199 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of independent variables: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Mtry: 3 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Target node size: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Variable importance mode: none &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Splitrule: variance &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; OOB prediction error (MSE): 0.00848 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; R squared (OOB): 0.735&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Suppose that we would like to use the randomForest package instead of ranger. To do so, the only part of the syntax that needs to change is the &lt;code&gt;set_engine()&lt;/code&gt; argument:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;rand_forest&lt;/span&gt;(mode &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;regression&amp;#34;&lt;/span&gt;, mtry &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;3&lt;/span&gt;, trees &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1000&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;randomForest&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;log10&lt;/span&gt;(Sale_Price) &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; Longitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Latitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Lot_Area &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Neighborhood &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Year_Sold,

data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train

)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 2.1s &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; randomForest(x = as.data.frame(x), y = y, ntree = ~1000, mtry = ~3) &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Type of random forest: regression&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of trees: 1000&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; No. of variables tried at each split: 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Mean of squared residuals: 0.013&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; % Var explained: 59.4&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Look at the formula code that was printed out; one function uses the argument name &lt;code&gt;ntree&lt;/code&gt; and the other uses &lt;code&gt;num.trees&lt;/code&gt;. The parsnip models don&amp;rsquo;t require you to know the specific names of the main arguments.&lt;/p&gt;

&lt;p&gt;Now suppose that we want to modify the value of &lt;code&gt;mtry&lt;/code&gt; based on the number of predictors in the data. Usually, a good default value is &lt;code&gt;floor(sqrt(num_predictors))&lt;/code&gt; but a pure bagging model requires an &lt;code&gt;mtry&lt;/code&gt; value equal to the total number of parameters. There may be cases where you may not know how many predictors are going to be present when the model will be fit (perhaps due to the generation of indicator variables or a variable filter) so this might be difficult to know exactly ahead of time when you write your code.&lt;/p&gt;

&lt;p&gt;When the model it being fit by parsnip,

&lt;a href=&#34;https://tidymodels.github.io/parsnip/reference/descriptors.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&lt;em&gt;data descriptors&lt;/em&gt;&lt;/a&gt; are made available. These attempt to let you know what you will have available when the model is fit. When a model object is created (say using &lt;code&gt;rand_forest()&lt;/code&gt;), the values of the arguments that you give it are &lt;em&gt;immediately evaluated&lt;/em&gt; unless you delay them. To delay the evaluation of any argument, you can used &lt;code&gt;rlang::expr()&lt;/code&gt; to make an expression.&lt;/p&gt;

&lt;p&gt;Two relevant data descriptors for our example model are:&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;&lt;code&gt;.preds()&lt;/code&gt;: the number of predictor &lt;em&gt;variables&lt;/em&gt; in the data set that are associated with the predictors &lt;strong&gt;prior to dummy variable creation&lt;/strong&gt;.&lt;/li&gt;

&lt;li&gt;&lt;code&gt;.cols()&lt;/code&gt;: the number of predictor &lt;em&gt;columns&lt;/em&gt; after dummy variables (or other encodings) are created.&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;Since ranger won&amp;rsquo;t create indicator values, &lt;code&gt;.preds()&lt;/code&gt; would be appropriate for &lt;code&gt;mtry&lt;/code&gt; for a bagging model.&lt;/p&gt;

&lt;p&gt;For example, let&amp;rsquo;s use an expression with the &lt;code&gt;.preds()&lt;/code&gt; descriptor to fit a bagging model:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;rand_forest&lt;/span&gt;(mode &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;regression&amp;#34;&lt;/span&gt;, mtry &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;.preds&lt;/span&gt;(), trees &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1000&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;ranger&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;log10&lt;/span&gt;(Sale_Price) &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; Longitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Latitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Lot_Area &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Neighborhood &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Year_Sold,

data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train

)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 3.6s &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Ranger result&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ranger::ranger(formula = formula, data = data, mtry = ~.preds(), num.trees = ~1000, num.threads = 1, verbose = FALSE, seed = sample.int(10^5, 1)) &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Type: Regression &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of trees: 1000 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Sample size: 2199 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Number of independent variables: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Mtry: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Target node size: 5 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Variable importance mode: none &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Splitrule: variance &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; OOB prediction error (MSE): 0.00869 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; R squared (OOB): 0.728&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;regularized-regression&#34;&gt;Regularized regression&lt;/h2&gt;

&lt;p&gt;A linear model might work for this data set as well. We can use the &lt;code&gt;linear_reg()&lt;/code&gt; parsnip model. There are two engines that can perform regularization/penalization, the glmnet and sparklyr packages. Let&amp;rsquo;s use the former here. The glmnet package only implements a non-formula method, but parsnip will allow either one to be used.&lt;/p&gt;

&lt;p&gt;When regularization is used, the predictors should first be centered and scaled before being passed to the model. The formula method won&amp;rsquo;t do that automatically so we will need to do this ourselves. We&amp;rsquo;ll use the

&lt;a href=&#34;https://tidymodels.github.io/recipes/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;recipes&lt;/a&gt; package for these steps.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;norm_recipe &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;recipe&lt;/span&gt;(

Sale_Price &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; Longitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Latitude &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Lot_Area &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Neighborhood &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; Year_Sold,

data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train

) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_other&lt;/span&gt;(Neighborhood) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_dummy&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;all_nominal&lt;/span&gt;()) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_center&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;()) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_scale&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;()) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_log&lt;/span&gt;(Sale_Price, base &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;10&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# estimate the means and standard deviations&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;prep&lt;/span&gt;(training &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_train, retain &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;TRUE&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Now let&amp;#39;s fit the model using the processed version of the data&lt;/span&gt;

glmn_fit &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;linear_reg&lt;/span&gt;(penalty &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0.001&lt;/span&gt;, mixture &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0.5&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;glmnet&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(Sale_Price &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; ., data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;juice&lt;/span&gt;(norm_recipe))

glmn_fit

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 13ms &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Call: glmnet::glmnet(x = as.matrix(x), y = y, family = &amp;#34;gaussian&amp;#34;, alpha = ~0.5) &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Df %Dev Lambda&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 0 0.000 0.1370&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 1 0.019 0.1250&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 1 0.036 0.1140&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 1 0.050 0.1040&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 2 0.068 0.0946&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 4 0.093 0.0862&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 7 5 0.125 0.0785&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 8 5 0.153 0.0716&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 9 7 0.184 0.0652&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 10 7 0.214 0.0594&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 11 7 0.240 0.0541&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 12 8 0.262 0.0493&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 13 8 0.286 0.0449&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 14 8 0.306 0.0409&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 15 8 0.323 0.0373&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 16 8 0.338 0.0340&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 17 8 0.350 0.0310&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 18 8 0.361 0.0282&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 19 9 0.370 0.0257&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 20 9 0.379 0.0234&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 21 9 0.386 0.0213&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 22 9 0.392 0.0195&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 23 9 0.397 0.0177&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 24 9 0.401 0.0161&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 25 9 0.405 0.0147&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 26 9 0.408 0.0134&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 27 10 0.410 0.0122&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 28 11 0.413 0.0111&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 29 11 0.415 0.0101&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 30 11 0.417 0.0092&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 31 12 0.418 0.0084&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 32 12 0.420 0.0077&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 33 12 0.421 0.0070&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 34 12 0.422 0.0064&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 35 12 0.423 0.0058&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 36 12 0.423 0.0053&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 37 12 0.424 0.0048&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 38 12 0.425 0.0044&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 39 12 0.425 0.0040&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 40 12 0.425 0.0036&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 41 12 0.426 0.0033&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 42 12 0.426 0.0030&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 43 12 0.426 0.0028&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 44 12 0.426 0.0025&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 45 12 0.426 0.0023&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 46 12 0.426 0.0021&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 47 12 0.427 0.0019&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 48 12 0.427 0.0017&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 49 12 0.427 0.0016&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 50 12 0.427 0.0014&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 51 12 0.427 0.0013&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 52 12 0.427 0.0012&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 53 12 0.427 0.0011&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 54 12 0.427 0.0010&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 55 12 0.427 0.0009&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 56 12 0.427 0.0008&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 57 12 0.427 0.0008&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 58 12 0.427 0.0007&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 59 12 0.427 0.0006&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 60 12 0.427 0.0006&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 61 12 0.427 0.0005&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 62 12 0.427 0.0005&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 63 12 0.427 0.0004&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 64 12 0.427 0.0004&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 65 12 0.427 0.0004&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;If &lt;code&gt;penalty&lt;/code&gt; were not specified, all of the &lt;code&gt;lambda&lt;/code&gt; values would be computed.&lt;/p&gt;

&lt;p&gt;To get the predictions for this specific value of &lt;code&gt;lambda&lt;/code&gt; (aka &lt;code&gt;penalty&lt;/code&gt;):&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# First, get the processed version of the test set predictors:&lt;/span&gt;

test_normalized &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bake&lt;/span&gt;(norm_recipe, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; ames_test, &lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;())

test_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

test_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;rename&lt;/span&gt;(`random forest` &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(glmn_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; test_normalized) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;rename&lt;/span&gt;(glmnet &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred)

)

test_results

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 731 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Sale_Price `random forest` glmnet&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 5.33 5.22 5.27&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 5.02 5.21 5.17&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 5.27 5.25 5.23&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 5.60 5.51 5.25&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 5.28 5.24 5.25&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 5.17 5.19 5.19&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 7 5.02 4.97 5.19&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 8 5.46 5.50 5.49&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 9 5.44 5.46 5.48&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 10 5.33 5.50 5.47&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # … with 721 more rows&lt;/span&gt;

test_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;metrics&lt;/span&gt;(truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Sale_Price, estimate &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; glmnet)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 3 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .metric .estimator .estimate&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 rmse standard 0.132 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 rsq standard 0.410 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 mae standard 0.0956&lt;/span&gt;

test_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;gather&lt;/span&gt;(model, prediction, &lt;span style=&#34;color:#666&#34;&gt;-&lt;/span&gt;Sale_Price) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; prediction, y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Sale_Price)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_abline&lt;/span&gt;(col &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;green&amp;#34;&lt;/span&gt;, lty &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;2&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(alpha &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.4&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;facet_wrap&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt;model) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;coord_fixed&lt;/span&gt;()

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/glmn-pred-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;p&gt;This final plot compares the performance of the random forest and regularized regression models.&lt;/p&gt;

&lt;h2 id=&#34;session-information&#34;&gt;Session information&lt;/h2&gt;

&lt;pre&gt;&lt;code&gt;#&amp;gt; ─ Session info ───────────────────────────────────────────────────────────────

#&amp;gt; setting value

#&amp;gt; version R version 3.6.2 (2019-12-12)

#&amp;gt; os macOS Mojave 10.14.6

#&amp;gt; system x86_64, darwin15.6.0

#&amp;gt; ui X11

#&amp;gt; language (EN)

#&amp;gt; collate en_US.UTF-8

#&amp;gt; ctype en_US.UTF-8

#&amp;gt; tz America/Denver

#&amp;gt; date 2020-04-17

#&amp;gt;

#&amp;gt; ─ Packages ───────────────────────────────────────────────────────────────────

#&amp;gt; package * version date lib source

#&amp;gt; AmesHousing * 0.0.3 2017-12-17 [1] CRAN (R 3.6.0)

#&amp;gt; broom * 0.5.5 2020-02-29 [1] CRAN (R 3.6.0)

#&amp;gt; dials * 0.0.6 2020-04-03 [1] CRAN (R 3.6.2)

#&amp;gt; dplyr * 0.8.5 2020-03-07 [1] CRAN (R 3.6.0)

#&amp;gt; ggplot2 * 3.3.0 2020-03-05 [1] CRAN (R 3.6.0)

#&amp;gt; glmnet * 3.0-2 2019-12-11 [1] CRAN (R 3.6.0)

#&amp;gt; infer * 0.5.1 2019-11-19 [1] CRAN (R 3.6.0)

#&amp;gt; parsnip * 0.1.0 2020-04-09 [1] CRAN (R 3.6.2)

#&amp;gt; purrr * 0.3.3 2019-10-18 [1] CRAN (R 3.6.0)

#&amp;gt; randomForest * 4.6-14 2018-03-25 [1] CRAN (R 3.6.0)

#&amp;gt; ranger * 0.12.1 2020-01-10 [1] CRAN (R 3.6.0)

#&amp;gt; recipes * 0.1.10 2020-03-18 [1] CRAN (R 3.6.0)

#&amp;gt; rlang 0.4.5 2020-03-01 [1] CRAN (R 3.6.0)

#&amp;gt; rsample * 0.0.6 2020-03-31 [1] CRAN (R 3.6.2)

#&amp;gt; tibble * 2.1.3 2019-06-06 [1] CRAN (R 3.6.2)

#&amp;gt; tidymodels * 0.1.0 2020-02-16 [1] CRAN (R 3.6.0)

#&amp;gt; tune * 0.1.0 2020-04-02 [1] CRAN (R 3.6.2)

#&amp;gt; workflows * 0.1.1 2020-03-17 [1] CRAN (R 3.6.0)

#&amp;gt; yardstick * 0.0.6 2020-03-17 [1] CRAN (R 3.6.0)

#&amp;gt;

#&amp;gt; [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

&lt;/code&gt;&lt;/pre&gt;</description>

</item>

<item>

<title>Classification models using a neural network</title>

<link>https://rapidsurveys.io/learn/models/parsnip-nnet/</link>

<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>

<guid>https://rapidsurveys.io/learn/models/parsnip-nnet/</guid>

<description>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;

&lt;p&gt;To use the code in this article, you will need to install the following packages: keras and tidymodels. You will also need the python keras library installed (see &lt;code&gt;?keras::install_keras()&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;We can create classification models with the tidymodels package

&lt;a href=&#34;https://tidymodels.github.io/parsnip/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;parsnip&lt;/a&gt; to predict categorical quantities or class labels. Here, let&amp;rsquo;s fit a single classification model using a neural network and evaluate using a validation set. While the

&lt;a href=&#34;https://tidymodels.github.io/tune/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;tune&lt;/a&gt; package has functionality to also do this, the parsnip package is the center of attention in this article so that we can better understand its usage.&lt;/p&gt;

&lt;h2 id=&#34;fitting-a-neural-network&#34;&gt;Fitting a neural network&lt;/h2&gt;

&lt;p&gt;Let&amp;rsquo;s fit a model to a small, two predictor classification data set. The data are in the modeldata package (part of tidymodels) and have been split into training, validation, and test data sets. In this analysis, the test set is left untouched; this article tries to emulate a good data usage methodology where the test set would only be evaluated once at the end after a variety of models have been considered.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;data&lt;/span&gt;(bivariate)

&lt;span style=&#34;color:#00f&#34;&gt;nrow&lt;/span&gt;(bivariate_train)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; [1] 1009&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;nrow&lt;/span&gt;(bivariate_val)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; [1] 300&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A plot of the data shows two right-skewed predictors:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(bivariate_train, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; A, y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; B, col &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Class)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(alpha &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.2&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/biv-plot-1.svg&#34; width=&#34;576&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s use a single hidden layer neural network to predict the outcome. To do this, we transform the predictor columns to be more symmetric (via the &lt;code&gt;step_BoxCox()&lt;/code&gt; function) and on a common scale (using &lt;code&gt;step_normalize()&lt;/code&gt;). We can use

&lt;a href=&#34;https://tidymodels.github.io/recipes/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;recipes&lt;/a&gt; to do so:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;biv_rec &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;recipe&lt;/span&gt;(Class &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; ., data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; bivariate_train) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_BoxCox&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;())&lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;step_normalize&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;()) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;prep&lt;/span&gt;(training &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; bivariate_train, retain &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;TRUE&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# We will juice() to get the processed training set back&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# For validation:&lt;/span&gt;

val_normalized &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bake&lt;/span&gt;(biv_rec, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; bivariate_val, &lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;())

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# For testing when we arrive at a final model: &lt;/span&gt;

test_normalized &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bake&lt;/span&gt;(biv_rec, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; bivariate_test, &lt;span style=&#34;color:#00f&#34;&gt;all_predictors&lt;/span&gt;())

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;We can use the keras package to fit a model with 5 hidden units and a 10% dropout rate, to regularize the model:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;set.seed&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;57974&lt;/span&gt;)

nnet_fit &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mlp&lt;/span&gt;(epochs &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;100&lt;/span&gt;, hidden_units &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;5&lt;/span&gt;, dropout &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0.1&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_mode&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;classification&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Also set engine-specific `verbose` argument to prevent logging the results: &lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;keras&amp;#34;&lt;/span&gt;, verbose &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(Class &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; ., data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;juice&lt;/span&gt;(biv_rec))

nnet_fit

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; parsnip model object&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Fit time: 8.7s &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Model&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Model: &amp;#34;sequential&amp;#34;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ________________________________________________________________________________&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Layer (type) Output Shape Param # &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ================================================================================&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; dense (Dense) (None, 5) 15 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ________________________________________________________________________________&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; dense_1 (Dense) (None, 5) 30 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ________________________________________________________________________________&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; dropout (Dropout) (None, 5) 0 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ________________________________________________________________________________&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; dense_2 (Dense) (None, 2) 12 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ================================================================================&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Total params: 57&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Trainable params: 57&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Non-trainable params: 0&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; ________________________________________________________________________________&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;model-performance&#34;&gt;Model performance&lt;/h2&gt;

&lt;p&gt;In parsnip, the &lt;code&gt;predict()&lt;/code&gt; function can be used to characterize performance on the validation set. Since parsnip always produces tibble outputs, these can just be column bound to the original data:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;val_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

bivariate_val &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(

&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(nnet_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; val_normalized),

&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(nnet_fit, new_data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; val_normalized, type &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;prob&amp;#34;&lt;/span&gt;)

)

val_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;slice&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;5&lt;/span&gt;)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 5 x 6&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; A B Class .pred_class .pred_One .pred_Two&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;fct&amp;gt; &amp;lt;fct&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 1061. 74.5 One Two 0.473 0.527 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 1241. 83.4 One Two 0.484 0.516 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 939. 71.9 One One 0.636 0.364 &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 813. 77.1 One One 0.925 0.0746&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 1706. 92.8 Two Two 0.355 0.645&lt;/span&gt;

val_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;roc_auc&lt;/span&gt;(truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Class, .pred_One)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 1 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .metric .estimator .estimate&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 roc_auc binary 0.815&lt;/span&gt;

val_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;accuracy&lt;/span&gt;(truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Class, .pred_class)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 1 x 3&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; .metric .estimator .estimate&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 accuracy binary 0.737&lt;/span&gt;

val_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;conf_mat&lt;/span&gt;(truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Class, .pred_class)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Truth&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Prediction One Two&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; One 150 27&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Two 52 71&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Let&amp;rsquo;s also create a grid to get a visual sense of the class boundary for the validation set.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;a_rng &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;range&lt;/span&gt;(bivariate_train&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;A)

b_rng &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;range&lt;/span&gt;(bivariate_train&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;B)

x_grid &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;expand.grid&lt;/span&gt;(A &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;seq&lt;/span&gt;(a_rng[1], a_rng[2], length.out &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;100&lt;/span&gt;),

B &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;seq&lt;/span&gt;(b_rng[1], b_rng[2], length.out &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;100&lt;/span&gt;))

x_grid_trans &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bake&lt;/span&gt;(biv_rec, x_grid)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Make predictions using the transformed predictors but &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# attach them to the predictors in the original units: &lt;/span&gt;

x_grid &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

x_grid &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(nnet_fit, x_grid_trans, type &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;prob&amp;#34;&lt;/span&gt;))

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(x_grid, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; A, y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; B)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_contour&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(z &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred_One), breaks &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.5&lt;/span&gt;, col &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;black&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; bivariate_val, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(col &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; Class), alpha &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;0.3&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/biv-boundary-1.svg&#34; width=&#34;576&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;session-information&#34;&gt;Session information&lt;/h2&gt;

&lt;pre&gt;&lt;code&gt;#&amp;gt; ─ Session info ───────────────────────────────────────────────────────────────

#&amp;gt; setting value

#&amp;gt; version R version 3.6.2 (2019-12-12)

#&amp;gt; os macOS Mojave 10.14.6

#&amp;gt; system x86_64, darwin15.6.0

#&amp;gt; ui X11

#&amp;gt; language (EN)

#&amp;gt; collate en_US.UTF-8

#&amp;gt; ctype en_US.UTF-8

#&amp;gt; tz America/Denver

#&amp;gt; date 2020-04-17

#&amp;gt;

#&amp;gt; ─ Packages ───────────────────────────────────────────────────────────────────

#&amp;gt; package * version date lib source

#&amp;gt; broom * 0.5.5 2020-02-29 [1] CRAN (R 3.6.0)

#&amp;gt; dials * 0.0.6 2020-04-03 [1] CRAN (R 3.6.2)

#&amp;gt; dplyr * 0.8.5 2020-03-07 [1] CRAN (R 3.6.0)

#&amp;gt; ggplot2 * 3.3.0 2020-03-05 [1] CRAN (R 3.6.0)

#&amp;gt; infer * 0.5.1 2019-11-19 [1] CRAN (R 3.6.0)

#&amp;gt; keras 2.2.5.0 2019-10-08 [1] CRAN (R 3.6.0)

#&amp;gt; parsnip * 0.1.0 2020-04-09 [1] CRAN (R 3.6.2)

#&amp;gt; purrr * 0.3.3 2019-10-18 [1] CRAN (R 3.6.0)

#&amp;gt; recipes * 0.1.10 2020-03-18 [1] CRAN (R 3.6.0)

#&amp;gt; rlang 0.4.5 2020-03-01 [1] CRAN (R 3.6.0)

#&amp;gt; rsample * 0.0.6 2020-03-31 [1] CRAN (R 3.6.2)

#&amp;gt; tibble * 2.1.3 2019-06-06 [1] CRAN (R 3.6.2)

#&amp;gt; tidymodels * 0.1.0 2020-02-16 [1] CRAN (R 3.6.0)

#&amp;gt; tune * 0.1.0 2020-04-02 [1] CRAN (R 3.6.2)

#&amp;gt; workflows * 0.1.1 2020-03-17 [1] CRAN (R 3.6.0)

#&amp;gt; yardstick * 0.0.6 2020-03-17 [1] CRAN (R 3.6.0)

#&amp;gt;

#&amp;gt; [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

&lt;/code&gt;&lt;/pre&gt;</description>

</item>

<item>

<title>Nested resampling</title>

<link>https://rapidsurveys.io/learn/work/nested-resampling/</link>

<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>

<guid>https://rapidsurveys.io/learn/work/nested-resampling/</guid>

<description>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;

&lt;p&gt;To use the code in this article, you will need to install the following packages: furrr, kernlab, mlbench, scales, and tidymodels.&lt;/p&gt;

&lt;p&gt;In this article, we discuss an alternative method for evaluating and tuning models, called

&lt;a href=&#34;https://scholar.google.com/scholar?hl=en&amp;amp;as_sdt=0%2C7&amp;amp;q=%22nested&amp;#43;resampling%22&amp;#43;inner&amp;#43;outer&amp;amp;btnG=&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;nested resampling&lt;/a&gt;. While it is more computationally taxing and challenging to implement than other resampling methods, it has the potential to produce better estimates of model performance.&lt;/p&gt;

&lt;h2 id=&#34;resampling-models&#34;&gt;Resampling models&lt;/h2&gt;

&lt;p&gt;A typical scheme for splitting the data when developing a predictive model is to create an initial split of the data into a training and test set. If resampling is used, it is executed on the training set. A series of binary splits is created. In rsample, we use the term &lt;em&gt;analysis set&lt;/em&gt; for the data that are used to fit the model and the term &lt;em&gt;assessment set&lt;/em&gt; for the set used to compute performance:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;figs/resampling.svg&#34; width=&#34;70%&#34; style=&#34;display: block; margin: auto;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;A common method for tuning models is

&lt;a href=&#34;https://rapidsurveys.io/learn/work/tune-svm/&#34;&gt;grid search&lt;/a&gt; where a candidate set of tuning parameters is created. The full set of models for every combination of the tuning parameter grid and the resamples is fitted. Each time, the assessment data are used to measure performance and the average value is determined for each tuning parameter.&lt;/p&gt;

&lt;p&gt;The potential problem is that once we pick the tuning parameter associated with the best performance, this performance value is usually quoted as the performance of the model. There is serious potential for &lt;em&gt;optimization bias&lt;/em&gt; since we use the same data to tune the model and to assess performance. This would result in an optimistic estimate of performance.&lt;/p&gt;

&lt;p&gt;Nested resampling uses an additional layer of resampling that separates the tuning activities from the process used to estimate the efficacy of the model. An &lt;em&gt;outer&lt;/em&gt; resampling scheme is used and, for every split in the outer resample, another full set of resampling splits are created on the original analysis set. For example, if 10-fold cross-validation is used on the outside and 5-fold cross-validation on the inside, a total of 500 models will be fit. The parameter tuning will be conducted 10 times and the best parameters are determined from the average of the 5 assessment sets. This process occurs 10 times.&lt;/p&gt;

&lt;p&gt;Once the tuning results are complete, a model is fit to each of the outer resampling splits using the best parameter associated with that resample. The average of the outer method&amp;rsquo;s assessment sets are a unbiased estimate of the model.&lt;/p&gt;

&lt;p&gt;We will simulate some regression data to illustrate the methods. The mlbench package has a function &lt;code&gt;mlbench::mlbench.friedman1()&lt;/code&gt; that can simulate a complex regression data structure from the

&lt;a href=&#34;https://scholar.google.com/scholar?hl=en&amp;amp;q=%22Multivariate&amp;#43;adaptive&amp;#43;regression&amp;#43;splines%22&amp;amp;btnG=&amp;amp;as_sdt=1%2C7&amp;amp;as_sdtp=&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;original MARS publication&lt;/a&gt;. A training set size of 100 data points are generated as well as a large set that will be used to characterize how well the resampling procedure performed.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(mlbench)

sim_data &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(n) {

tmp &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;mlbench.friedman1&lt;/span&gt;(n, sd &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;)

tmp &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cbind&lt;/span&gt;(tmp&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;x, tmp&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;y)

tmp &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;as.data.frame&lt;/span&gt;(tmp)

&lt;span style=&#34;color:#00f&#34;&gt;names&lt;/span&gt;(tmp)&lt;span style=&#34;color:#00f&#34;&gt;[ncol&lt;/span&gt;(tmp)] &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;y&amp;#34;&lt;/span&gt;

tmp

}

&lt;span style=&#34;color:#00f&#34;&gt;set.seed&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;9815&lt;/span&gt;)

train_dat &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;sim_data&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;100&lt;/span&gt;)

large_dat &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;sim_data&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;10&lt;/span&gt;^5)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;nested-resampling&#34;&gt;Nested resampling&lt;/h2&gt;

&lt;p&gt;To get started, the types of resampling methods need to be specified. This isn&amp;rsquo;t a large data set, so 5 repeats of 10-fold cross validation will be used as the &lt;em&gt;outer&lt;/em&gt; resampling method for generating the estimate of overall performance. To tune the model, it would be good to have precise estimates for each of the values of the tuning parameter so let&amp;rsquo;s use 25 iterations of the bootstrap. This means that there will eventually be &lt;code&gt;5 * 10 * 25 = 1250&lt;/code&gt; models that are fit to the data &lt;em&gt;per tuning parameter&lt;/em&gt;. These models will be discarded once the performance of the model has been quantified.&lt;/p&gt;

&lt;p&gt;To create the tibble with the resampling specifications:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(tidymodels)

results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;nested_cv&lt;/span&gt;(train_dat,

outside &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;vfold_cv&lt;/span&gt;(repeats &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;5&lt;/span&gt;),

inside &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bootstraps&lt;/span&gt;(times &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;25&lt;/span&gt;))

results

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; [1] &amp;#34;nested_cv&amp;#34; &amp;#34;vfold_cv&amp;#34; &amp;#34;rset&amp;#34; &amp;#34;tbl_df&amp;#34; &amp;#34;tbl&amp;#34; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; [6] &amp;#34;data.frame&amp;#34;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # Nested resampling:&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # outer: 10-fold cross-validation repeated 5 times&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # inner: Bootstrap sampling&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 50 x 4&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; splits id id2 inner_resamples &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;named list&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;named list&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold01 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold02 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold03 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold04 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold05 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold06 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 7 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold07 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 8 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold08 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 9 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold09 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 10 &amp;lt;split [90/10]&amp;gt; Repeat1 Fold10 &amp;lt;tibble [25 × 2]&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # … with 40 more rows&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The splitting information for each resample is contained in the &lt;code&gt;split&lt;/code&gt; objects. Focusing on the second fold of the first repeat:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;splits[[2]]

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;Training/Validation/Total&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;90/10/100&amp;gt;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;&amp;lt;90/10/100&amp;gt;&lt;/code&gt; indicates the number of observations in the analysis set, assessment set, and the original data.&lt;/p&gt;

&lt;p&gt;Each element of &lt;code&gt;inner_resamples&lt;/code&gt; has its own tibble with the bootstrapping splits.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;inner_resamples[[5]]

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # Bootstrap sampling &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # A tibble: 25 x 2&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; splits id &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;list&amp;gt; &amp;lt;chr&amp;gt; &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1 &amp;lt;split [90/31]&amp;gt; Bootstrap01&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 2 &amp;lt;split [90/33]&amp;gt; Bootstrap02&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 3 &amp;lt;split [90/37]&amp;gt; Bootstrap03&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 4 &amp;lt;split [90/31]&amp;gt; Bootstrap04&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 5 &amp;lt;split [90/32]&amp;gt; Bootstrap05&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 6 &amp;lt;split [90/32]&amp;gt; Bootstrap06&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 7 &amp;lt;split [90/36]&amp;gt; Bootstrap07&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 8 &amp;lt;split [90/34]&amp;gt; Bootstrap08&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 9 &amp;lt;split [90/29]&amp;gt; Bootstrap09&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 10 &amp;lt;split [90/31]&amp;gt; Bootstrap10&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; # … with 15 more rows&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;These are self-contained, meaning that the bootstrap sample is aware that it is a sample of a specific 90% of the data:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;inner_resamples[[5]]&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;splits[[1]]

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;Training/Validation/Total&amp;gt;&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; &amp;lt;90/31/90&amp;gt;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;To start, we need to define how the model will be created and measured. Let&amp;rsquo;s use a radial basis support vector machine model via the function &lt;code&gt;kernlab::ksvm&lt;/code&gt;. This model is generally considered to have &lt;em&gt;two&lt;/em&gt; tuning parameters: the SVM cost value and the kernel parameter &lt;code&gt;sigma&lt;/code&gt;. For illustration purposes here, only the cost value will be tuned and the function &lt;code&gt;kernlab::sigest&lt;/code&gt; will be used to estimate &lt;code&gt;sigma&lt;/code&gt; during each model fit. This is automatically done by &lt;code&gt;ksvm&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;After the model is fit to the analysis set, the root-mean squared error (RMSE) is computed on the assessment set. &lt;strong&gt;One important note:&lt;/strong&gt; for this model, it is critical to center and scale the predictors before computing dot products. We don&amp;rsquo;t do this operation here because &lt;code&gt;mlbench.friedman1&lt;/code&gt; simulates all of the predictors to be standardized uniform random variables.&lt;/p&gt;

&lt;p&gt;Our function to fit the model and compute the RMSE is:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(kernlab)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# `object` will be an `rsplit` object from our `results` tibble&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# `cost` is the tuning parameter&lt;/span&gt;

svm_rmse &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(object, cost &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;) {

y_col &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;ncol&lt;/span&gt;(object&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;data)

mod &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;svm_rbf&lt;/span&gt;(mode &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;regression&amp;#34;&lt;/span&gt;, cost &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; cost) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;set_engine&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;kernlab&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;fit&lt;/span&gt;(y &lt;span style=&#34;color:#666&#34;&gt;~&lt;/span&gt; ., data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;analysis&lt;/span&gt;(object))

holdout_pred &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;predict&lt;/span&gt;(mod, &lt;span style=&#34;color:#00f&#34;&gt;assessment&lt;/span&gt;(object) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; dplyr&lt;span style=&#34;color:#666&#34;&gt;::&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;select&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;-&lt;/span&gt;y)) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(&lt;span style=&#34;color:#00f&#34;&gt;assessment&lt;/span&gt;(object) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; dplyr&lt;span style=&#34;color:#666&#34;&gt;::&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;select&lt;/span&gt;(y))

&lt;span style=&#34;color:#00f&#34;&gt;rmse&lt;/span&gt;(holdout_pred, truth &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; y, estimate &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; .pred)&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;.estimate

}

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# In some case, we want to parameterize the function over the tuning parameter:&lt;/span&gt;

rmse_wrapper &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(cost, object) &lt;span style=&#34;color:#00f&#34;&gt;svm_rmse&lt;/span&gt;(object, cost)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;For the nested resampling, a model needs to be fit for each tuning parameter and each bootstrap split. To do this, create a wrapper:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# `object` will be an `rsplit` object for the bootstrap samples&lt;/span&gt;

tune_over_cost &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(object) {

&lt;span style=&#34;color:#00f&#34;&gt;tibble&lt;/span&gt;(cost &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;2&lt;/span&gt; ^ &lt;span style=&#34;color:#00f&#34;&gt;seq&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;-2&lt;/span&gt;, &lt;span style=&#34;color:#666&#34;&gt;8&lt;/span&gt;, by &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;)) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mutate&lt;/span&gt;(RMSE &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;map_dbl&lt;/span&gt;(cost, rmse_wrapper, object &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; object))

}

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Since this will be called across the set of outer cross-validation splits, another wrapper is required:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# `object` is an `rsplit` object in `results$inner_resamples` &lt;/span&gt;

summarize_tune_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(object) {

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# Return row-bound tibble that has the 25 bootstrap results&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;map_df&lt;/span&gt;(object&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;splits, tune_over_cost) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# For each value of the tuning parameter, compute the &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;# average RMSE which is the inner bootstrap estimate. &lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;group_by&lt;/span&gt;(cost) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;summarize&lt;/span&gt;(mean_RMSE &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;mean&lt;/span&gt;(RMSE, na.rm &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;TRUE&lt;/span&gt;),

n &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;length&lt;/span&gt;(RMSE))

}

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Now that those functions are defined, we can execute all the inner resampling loops:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;tuning_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;map&lt;/span&gt;(results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;inner_resamples, summarize_tune_results)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Alternatively, since these computations can be run in parallel, we can use the furrr package. Instead of using &lt;code&gt;map()&lt;/code&gt;, the function &lt;code&gt;future_map()&lt;/code&gt; parallelizes the iterations using the

&lt;a href=&#34;https://cran.r-project.org/web/packages/future/vignettes/future-1-overview.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;future package&lt;/a&gt;. The &lt;code&gt;multisession&lt;/code&gt; plan uses the local cores to process the inner resampling loop. The end results are the same as the sequential computations.&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(furrr)

&lt;span style=&#34;color:#00f&#34;&gt;plan&lt;/span&gt;(multisession)

tuning_results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;future_map&lt;/span&gt;(results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;inner_resamples, summarize_tune_results)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The object &lt;code&gt;tuning_results&lt;/code&gt; is a list of data frames for each of the 50 outer resamples.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s make a plot of the averaged results to see what the relationship is between the RMSE and the tuning parameters for each of the inner bootstrapping operations:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;&lt;span style=&#34;color:#00f&#34;&gt;library&lt;/span&gt;(scales)

pooled_inner &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; tuning_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt; bind_rows

best_cost &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(dat) dat&lt;span style=&#34;color:#00f&#34;&gt;[which.min&lt;/span&gt;(dat&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;mean_RMSE),]

p &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(pooled_inner, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; cost, y &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; mean_RMSE)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;scale_x_continuous&lt;/span&gt;(trans &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#39;log2&amp;#39;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;xlab&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;SVM Cost&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;ylab&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;Inner RMSE&amp;#34;&lt;/span&gt;)

&lt;span style=&#34;color:#00f&#34;&gt;for &lt;/span&gt;(i in &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;length&lt;/span&gt;(tuning_results))

p &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; p &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_line&lt;/span&gt;(data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; tuning_results[[i]], alpha &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;.2&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_point&lt;/span&gt;(data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;best_cost&lt;/span&gt;(tuning_results[[i]]), pch &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;16&lt;/span&gt;, alpha &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;3&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;/&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;4&lt;/span&gt;)

p &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt; p &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;geom_smooth&lt;/span&gt;(data &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; pooled_inner, se &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;FALSE&lt;/span&gt;)

p

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/rmse-plot-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Each gray line is a separate bootstrap resampling curve created from a different 90% of the data. The blue line is a LOESS smooth of all the results pooled together.&lt;/p&gt;

&lt;p&gt;To determine the best parameter estimate for each of the outer resampling iterations:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;cost_vals &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

tuning_results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;map_df&lt;/span&gt;(best_cost) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;select&lt;/span&gt;(cost)

results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;bind_cols&lt;/span&gt;(results, cost_vals) &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mutate&lt;/span&gt;(cost &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;factor&lt;/span&gt;(cost, levels &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;paste&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;2&lt;/span&gt; ^ &lt;span style=&#34;color:#00f&#34;&gt;seq&lt;/span&gt;(&lt;span style=&#34;color:#666&#34;&gt;-2&lt;/span&gt;, &lt;span style=&#34;color:#666&#34;&gt;8&lt;/span&gt;, by &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#666&#34;&gt;1&lt;/span&gt;))))

&lt;span style=&#34;color:#00f&#34;&gt;ggplot&lt;/span&gt;(results, &lt;span style=&#34;color:#00f&#34;&gt;aes&lt;/span&gt;(x &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; cost)) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;geom_bar&lt;/span&gt;() &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;xlab&lt;/span&gt;(&lt;span style=&#34;color:#ba2121&#34;&gt;&amp;#34;SVM Cost&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#666&#34;&gt;+&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;scale_x_discrete&lt;/span&gt;(drop &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#008000;font-weight:bold&#34;&gt;FALSE&lt;/span&gt;)

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;figs/choose-1.svg&#34; width=&#34;672&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Most of the resamples produced an optimal cost value of 2.0, but the distribution is right-skewed due to the flat trend in the resampling profile once the cost value becomes 10 or larger.&lt;/p&gt;

&lt;p&gt;Now that we have these estimates, we can compute the outer resampling results for each of the 50 splits using the corresponding tuning parameter value:&lt;/p&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre style=&#34;;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-r&#34; data-lang=&#34;r&#34;&gt;results &lt;span style=&#34;color:#666&#34;&gt;&amp;lt;-&lt;/span&gt;

results &lt;span style=&#34;color:#666&#34;&gt;%&amp;gt;%&lt;/span&gt;

&lt;span style=&#34;color:#00f&#34;&gt;mutate&lt;/span&gt;(RMSE &lt;span style=&#34;color:#666&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;map2_dbl&lt;/span&gt;(splits, cost, svm_rmse))

&lt;span style=&#34;color:#00f&#34;&gt;summary&lt;/span&gt;(results&lt;span style=&#34;color:#666&#34;&gt;$&lt;/span&gt;RMSE)

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; Min. 1st Qu. Median Mean 3rd Qu. Max. &lt;/span&gt;

&lt;span style=&#34;color:#408080;font-style:italic&#34;&gt;#&amp;gt; 1.57 2.09 2.68 2.69 3.25 4.25&lt;/span&gt;