class: center, middle, inverse, title-slide # Steering students past the ‘true model myth’ ## OZCOTS 2021 ### Damjan Vukcevic ### …with Margarita Moreno-Betancur, John Carlin, Sue Finch, Ian Gordon & Lyle Gurrin ### 9 July 2021 --- <style type="text/css"> .remark-slide-content { font-size: 24px; } </style> # Student's predicament `\(X_1, X_2, \dots, X_n \sim \mathrm{N}(\mu_1, \sigma_1^2)\)` `\(Y_1, Y_2, \dots, Y_m \sim \mathrm{N}(\mu_2, \sigma_2^2)\)` -- Want to compare `\(\mu_1\)` and `\(\mu_2\)` -- Can we assume `\(\sigma_1 = \sigma_2\)`? -- In R, which of these should we run? ``` t.test(x, y) t.test(x, y, var.equal = TRUE) ``` --- # The 'true model myth' -- Analysis process: 1. Determine the best model 2. Derive (all of the) answers from this model -- Implicit assumptions: * Our goal is to find the 'true' model * We can use our 'best' model as if it were the 'true' model -- (Similar to misuse of statistical significance? An overly 'black and white' view of the data? Ignores model uncertainty...) --- # Antidotes The idea of a 'statistical investigation' -- See: * [Wild & Pfannkuch (1999)](https://doi.org/10.1111/j.1751-5823.1999.tb00442.x) * **Robert Gould**'s keynote talk yesterday, mentioned 'The Data Cycle' -- A statistical investigation will typically investigate **multiple** models -- (...and we might never need to choose between them!) -- Let's show such examples to students. --- # Reasons for using multiple models 1. Comparing & optimising performance 2. Exploring different assumptions 3. Exploring different questions 4. Varying the desired estimation properties --- # 1. Comparing & optimising performance -- Routinely done for **predictive modelling** -- ...including creating ensembles of **multiple** models --- # Example: species distribution modelling  .footnote[ .font70[ Adapted from [Ingram, Vukcevic & Golding (2020)](https://doi.org/10.1111/2041-210X.13496) ] ] --- # 2. Exploring different assumptions -- Common scenario: **sensitivity analyses** --- # Example: t-test .pull-left[ **Different variances** (Welch approximation) ```r t.test(x, y) ``` `\(t = 7.85\)` `\(\mathrm{df} = 36.1\)` `\(\textrm{p-value} = 2.5 \times 10^{-9}\)` `\(\textrm{95% CI} = (3.00, 5.08)\)` ] -- .pull-right[ **Pooled variance** ```r t.test(x, y, var.equal = TRUE) ``` `\(t = 7.32\)` `\(\mathrm{df} = 43\)` `\(\textrm{p-value} = 4.5 \times 10^{-9}\)` `\(\textrm{95% CI} = (2.93, 5.15)\)` ] --- # Example: prior sensitivity analysis <!-- --> --- # Example: causal inference  --- # 3. Exploring different questions Some clear examples: * Changing the response variable * Changing the 'primary' explanatory variables -- But sometimes it's less clear... --- # Example: ANOVA vs polynomial regression .pull-left[ .center[ <!-- --> ] ] -- .pull-right[ .center[ <!-- --> ] ] --- # Example: ANOVA vs polynomial regression .pull-left[ .center[  ] ] .pull-right[ .center[ <!-- --> ] ] --- # 4. Varying the desired estimation properties -- Typical trade-off: **bias** vs **variance** --- # Example: ANOVA vs polynomial regression .pull-left[ .center[  ] ] .pull-right[ .center[  ] ] --- # When teaching students... Create examples that feature **multiple** models/techniques. -- ### Handy reference Possible reasons for using multiple models: 1. Comparing & optimising performance 2. Exploring different assumptions 3. Exploring different questions 4. Varying the desired estimation properties