Processing math: 100%

class: center, middle, inverse, title-slide

# Steering students past the ‘true model myth’
## OZCOTS 2021
### Damjan Vukcevic
### …with Margarita Moreno-Betancur, John Carlin, Sue Finch, Ian Gordon & Lyle Gurrin
### 9 July 2021

---

<style type="text/css">
.remark-slide-content {
    font-size: 24px;
}
</style>

# Student's predicament

`\(X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)\)`

`\(Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)\)`

--

Want to compare `\(\mu_1\)` and `\(\mu_2\)`

--

Can we assume `\(\sigma_1 = \sigma_2\)`?

--

In R, which of these should we run?
```
t.test(x, y)
t.test(x, y, var.equal = TRUE)
```

---

# The 'true model myth'

--

Analysis process:

1. Determine the best model
2. Derive (all of the) answers from this model

--

Implicit assumptions:

* Our goal is to find the 'true' model
* We can use our 'best' model as if it were the 'true' model

--

(Similar to misuse of statistical significance?  
 An overly 'black and white' view of the data?  
 Ignores model uncertainty...)

---

# Antidotes

The idea of a 'statistical investigation'

--

See:

* [Wild & Pfannkuch (1999)](https://doi.org/10.1111/j.1751-5823.1999.tb00442.x)
* **Robert Gould**'s keynote talk yesterday, mentioned 'The Data Cycle'

--

A statistical investigation will typically investigate **multiple** models

--

(...and we might never need to choose between them!)

--

Let's show such examples to students.

---

# Reasons for using multiple models

1. Comparing & optimising performance
2. Exploring different assumptions
3. Exploring different questions
4. Varying the desired estimation properties

---

# 1. Comparing & optimising performance

--

Routinely done for **predictive modelling**

--

...including creating ensembles of **multiple** models

---

# Example: species distribution modelling

![](images/mee313496-fig-0002-m-modified.png)

.footnote[
.font70[
Adapted from [Ingram, Vukcevic & Golding (2020)](https://doi.org/10.1111/2041-210X.13496)
]
]

---

# 2. Exploring different assumptions

--

Common scenario: **sensitivity analyses**

---

# Example: t-test

.pull-left[

**Different variances** (Welch approximation)

```r
t.test(x, y)
```

`\(t = 7.85\)`

`\(\mathrm{df} = 36.1\)`

`\(\textrm{p-value} = 2.5 \times 10^{-9}\)`

`\(\textrm{95% CI} = (3.00, 5.08)\)`

]

--

.pull-right[

**Pooled variance**

```r
t.test(x, y, var.equal = TRUE)
```

`\(t = 7.32\)`

`\(\mathrm{df} = 43\)`

`\(\textrm{p-value} = 4.5 \times 10^{-9}\)`

`\(\textrm{95% CI} = (2.93, 5.15)\)`

]

---

# Example: prior sensitivity analysis

![](index_files/figure-html/bayes-sensitivity-1.png)

---

# Example: causal inference

![](images/example-contentious-confounder.png)

---

# 3. Exploring different questions

Some clear examples:

* Changing the response variable
* Changing the 'primary' explanatory variables

--

But sometimes it's less clear...

---

# Example: ANOVA vs polynomial regression

.pull-left[
.center[

![](index_files/figure-html/anova-polynomial-data-1.png)

]
]

--

.pull-right[
.center[

![](index_files/figure-html/anova-polynomial-fits-1.png)

]
]

---

# Example: ANOVA vs polynomial regression

.pull-left[
.center[

![](index_files/figure-html/anova-polynomial-data-1.png)

]
]

.pull-right[
.center[

![](index_files/figure-html/anova-polynomial-fits2-1.png)

]
]

---

# 4. Varying the desired estimation properties

--

Typical trade-off: **bias** vs **variance**

---

# Example: ANOVA vs polynomial regression

.pull-left[
.center[

![](index_files/figure-html/anova-polynomial-data-1.png)

]
]

.pull-right[
.center[

![](index_files/figure-html/anova-polynomial-fits2-1.png)

]
]
---

# When teaching students...

Create examples that feature **multiple** models/techniques.

--

### Handy reference

Possible reasons for using multiple models:

1. Comparing & optimising performance
2. Exploring different assumptions
3. Exploring different questions
4. Varying the desired estimation properties

Notes for current slide

Notes for next slide

Steering students past the ‘true model myth’OZCOTS 2021Damjan Vukcevic…with Margarita Moreno-Betancur, John Carlin, Sue Finch, Ian Gordon & Lyle Gurrin9 July 20211 / 17

Student's predicament

$X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)$

$Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)$

2 / 17

Student's predicament

$X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)$

$Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)$

Want to compare $\mu_1$ and $\mu_2$

2 / 17

Student's predicament

$X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)$

$Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)$

Want to compare $\mu_1$ and $\mu_2$

Can we assume $\sigma_1 = \sigma_2$ ?

2 / 17

Student's predicament

$X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)$

$Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)$

Want to compare $\mu_1$ and $\mu_2$

Can we assume $\sigma_1 = \sigma_2$ ?

In R, which of these should we run?

t.test(x, y)
t.test(x, y, var.equal = TRUE)

2 / 17

The 'true model myth'3 / 17

The 'true model myth'

Analysis process:

Determine the best model
Derive (all of the) answers from this model

3 / 17

The 'true model myth'

Analysis process:

Determine the best model
Derive (all of the) answers from this model

Implicit assumptions:

Our goal is to find the 'true' model
We can use our 'best' model as if it were the 'true' model

3 / 17

The 'true model myth'

Analysis process:

Determine the best model
Derive (all of the) answers from this model

Implicit assumptions:

Our goal is to find the 'true' model
We can use our 'best' model as if it were the 'true' model

(Similar to misuse of statistical significance?
An overly 'black and white' view of the data?
Ignores model uncertainty...)

3 / 17

Antidotes

The idea of a 'statistical investigation'

4 / 17

Antidotes

The idea of a 'statistical investigation'

Wild & Pfannkuch (1999)
Robert Gould's keynote talk yesterday, mentioned 'The Data Cycle'

4 / 17

Antidotes

The idea of a 'statistical investigation'

Wild & Pfannkuch (1999)
Robert Gould's keynote talk yesterday, mentioned 'The Data Cycle'

A statistical investigation will typically investigate multiple models

4 / 17

Antidotes

The idea of a 'statistical investigation'

Wild & Pfannkuch (1999)
Robert Gould's keynote talk yesterday, mentioned 'The Data Cycle'

A statistical investigation will typically investigate multiple models

(...and we might never need to choose between them!)

4 / 17

Antidotes

The idea of a 'statistical investigation'

Wild & Pfannkuch (1999)
Robert Gould's keynote talk yesterday, mentioned 'The Data Cycle'

A statistical investigation will typically investigate multiple models

(...and we might never need to choose between them!)

Let's show such examples to students.

4 / 17

Reasons for using multiple modelsComparing & optimising performance
Exploring different assumptions
Exploring different questions
Varying the desired estimation properties
5 / 17

1. Comparing & optimising performance6 / 17

1. Comparing & optimising performance

Routinely done for predictive modelling

6 / 17

1. Comparing & optimising performance

Routinely done for predictive modelling

...including creating ensembles of multiple models

6 / 17

Example: species distribution modelling

Adapted from Ingram, Vukcevic & Golding (2020)

7 / 17

2. Exploring different assumptions8 / 17

2. Exploring different assumptions

Common scenario: sensitivity analyses

8 / 17

Example: t-test

Different variances (Welch approximation)

t.test(x, y)

$t = 7.85$

$\mathrm{df} = 36.1$

$\textrm{p-value} = 2.5 \times 10^{-9}$

$\textrm{95% CI} = (3.00, 5.08)$

9 / 17

Example: t-test

Different variances (Welch approximation)

t.test(x, y)

$t = 7.85$

$\mathrm{df} = 36.1$

$\textrm{p-value} = 2.5 \times 10^{-9}$

$\textrm{95% CI} = (3.00, 5.08)$

Pooled variance

t.test(x, y, var.equal = TRUE)

$t = 7.32$

$\mathrm{df} = 43$

$\textrm{p-value} = 4.5 \times 10^{-9}$

$\textrm{95% CI} = (2.93, 5.15)$

9 / 17

Example: prior sensitivity analysis

10 / 17

Example: causal inference

11 / 17

3. Exploring different questions

Some clear examples:

Changing the response variable
Changing the 'primary' explanatory variables

12 / 17

3. Exploring different questions

Some clear examples:

Changing the response variable
Changing the 'primary' explanatory variables

But sometimes it's less clear...

12 / 17

Example: ANOVA vs polynomial regression

13 / 17

Example: ANOVA vs polynomial regression

13 / 17

Example: ANOVA vs polynomial regression

14 / 17

4. Varying the desired estimation properties15 / 17

4. Varying the desired estimation properties

Typical trade-off: bias vs variance

15 / 17

Example: ANOVA vs polynomial regression

16 / 17

When teaching students...

Create examples that feature multiple models/techniques.

17 / 17

When teaching students...

Create examples that feature multiple models/techniques.

Handy reference

Possible reasons for using multiple models:

Comparing & optimising performance
Exploring different assumptions
Exploring different questions
Varying the desired estimation properties

17 / 17

Student's predicament

$X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)$

$Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)$

2 / 17

Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

Esc	Back to slideshow