What are these new e-things? And why are they “safe”?

Damjan Vukcevic

EBS mini-seminar, Monash University

29 May 2026

Overview

How we want to analyse our data…

Overview

  1. Sequential analysis
  2. E-statistics
  3. (Briefly…) Sequential intersection tests

Ask me if interested

  • Auditing preferential elections

Sequential analysis

Sequential probability ratio test (SPRT)

Developed by Wald (1945)

Birth of sequential analysis

Binary data example

Data (X_i): 0, 0, 1, 1, 0, 1, \dots

Hypotheses: \begin{cases} H_0\colon p = p_0 \\ H_1\colon p = p_1 \end{cases}

Binary data example

Test statistic: S_t = \frac{L_t(p_1)}{L_t(p_0)} = \frac{\Pr(X_1, \dots, X_t \mid H_1)} {\Pr(X_1, \dots, X_t \mid H_0)} = \frac{p_1^{Y_t} (1 - p_1)^{t - Y_t}} {p_0^{Y_t} (1 - p_0)^{t - Y_t}}

Where: Y_t = \sum_{i=1}^t X_i

Binary data example

Binary data example

Stopping rule: \begin{cases} S_t \leqslant \frac{\beta}{1 - \alpha} & \Rightarrow \text{accept } H_0 \\ S_t \geqslant \frac{1 - \beta}{\alpha} & \Rightarrow \text{accept } H_1 \\ \text{otherwise} & \Rightarrow \text{keep sampling} \end{cases}

Error control:

  • Type 1 error \leqslant \alpha
  • Type 2 error \leqslant \beta

Binary data example

A “safe” procedure

  • Can “peek” at the data and stop if significant (optional stopping)
  • Can collect more data if not yet significant (optional continuation)

Remove the lower boundary

  • Set \beta = 0
  • Stopping rule:
    Reject H_0 once S_t \geqslant 1 / \alpha
  • Never “accept” H_0
  • No type II error (power = 1), but might have n = \infty
  • This is a “test of power one”.

What makes the SPRT “safe”?

Martingale

“Stays the same” on average

Supermartingale 🦸

“Stays the same or reduces in value” on average

(Super)martingales

Martingale condition: \mathbb{E}\left(S_{t+1} \mid X_1, \dots, X_t\right) = S_t

Supermartingale condition: \mathbb{E}\left(S_{t+1} \mid X_1, \dots, X_t\right) \leqslant S_t

Ville’s inequality

Let S_0, S_1, S_2, \dots be a test supermartingale (TSM):
has non-negative terms (S_i \geqslant 0) and starting value S_0 = 1.

For any \alpha > 0, \Pr\left(\max_t (S_t) \geqslant \frac{1}{\alpha}\right) \leqslant \alpha

Tests based on martingales

  • S_t should be a test supermartingale (TSM) under H_0
  • S_t should grow quickly under H_1

Tests based on martingales

Decision rule:

  • Reject H_0 if S_t \geqslant 1 / \alpha
  • Otherwise keep sampling (or give up)

Ville’s inequality \Rightarrow anytime-valid type I error control

Growth rate under H_1 \rightarrow statistical efficiency (“power”)

Further reading

Ramdas et al. (2023)

  • Safe anytime-valid inference (SAVI)

E-statistics

E-variable

  • A non-negative statistic E with: \mathbb{E}_{H_0}(E) \leqslant 1
  • Large E → strong evidence against H_0
  • A realisation of E is an e-value.

Markov’s inequality

Let S be a non-negative random variable (S \geqslant 0).

For any \alpha > 0, \Pr\left(S \geqslant \frac{1}{\alpha}\right) \leqslant \alpha \cdot \mathbb{E}(S)

Let S = E, an e-variable. Under H_0 we have: \Pr\left(E \geqslant \frac{1}{\alpha}\right) \leqslant \alpha

Testing using e-values

Reject H_0 if E \geqslant 1 / \alpha

Markov’s inequality \Rightarrow type I error control

Note

  • Alternative to using a p-value
  • Generalises a likelihood ratio

Combining e-variables

A, B, C are e-variables

A, B, C independent \Rightarrow the product is an e-variable: E = A \cdot B \cdot C

The average is always an e-variable: E = \frac{A + B + C}{3}

(Easier than combining p-values!)

E-process

  • Like a TSM, but more general.
  • Sequence (E_t) with \mathbb{E}_{H_0}(E_\tau) \leqslant 1, \quad \text{for all stopping times } \tau.
  • Connection with TSMs: E_t \leqslant \inf_{P \in H_0} S_t^{(P)}, \quad \text{where } S_t^{(P)} \text{ is a TSM for } P.
  • Ville’s inequality applies (anytime-valid type I error control).

Further reading

Chugg et al. (2026)

  • E-values as statistical evidence
  • Comparison with Bayes factors, likelihood ratios and p-values.

P-values from e-values

\text{Let } P_t = \frac{1}{E_t}

E_t > \frac{1}{\alpha} \quad \Longrightarrow \quad P_t < \alpha

  • P_t is an anytime-valid p-variable.
  • A realisation of P_t is an anytime-valid p-value.

Sequential intersection tests

Intersection null hypothesis

H_0 = H_0^1 \cap H_0^2 \cap \dots \cap H_0^k

Need to reject at least one of the H_0^j

…but which ones are false?

Adaptive weighting

Base TSM: for H_0^j

S_{j,t} = \prod_{i = 1}^t s_{j, i}

Intersection TSM: for H_0

S_t = \prod_{i = 1}^t \frac{\sum_j w_{j, i} \, s_{j, i}}{\sum_jw_{j, i}}

The weights w_{j, i} can depend on the data up until time i - 1.

Adaptive weighting in action

Ek et al. (2023)

Summary

  • Sequential analysis allows for “safe” inference

  • Modern tools use TSMs and e-processes

  • Complex inference problems can be tackled sequentially with finite-sample guarantees.

Questions?

References 📚

Image sources 🖼️

The coin flipping graphic was designed by Wannapik.

The photograph of Abraham Wald is freely available from Wikipedia.

The photograph of the 18th century print is freely available from the Digital Commonwealth; the original sits at the Boston Public Library.

Appendix

Binary data example

Sample size distribution

  • Parameters:

    • p_0 = 0.5
    • p_1 = 0.7 = p
    • \alpha = \beta = 0.1

  • n = 38 for fixed-n test with same power

  • SPRT often stops earlier

SPRT in general

H_0\colon X \sim f_0(\cdot) \quad \text{vs} \quad H_1\colon X \sim f_1(\cdot)

Test statistic: S_t = \frac{f_1(X_1, X_2, \dots, X_t)} {f_0(X_1, X_2, \dots, X_t)}

Test statistic, with iid observations: S_t = \prod_{i=1}^t \frac{f_1(X_i)}{f_0(X_i)} = S_{t - 1} \times \frac{f_1(X_t)}{f_0(X_t)}

The SPRT is a martingale

At time t: \mathbb{E}_{H_0} \left(\frac{f_1(X_t)}{f_0(X_t)}\right) = \int \frac{f_1(x)}{f_0(x)} f_0(x) \, dx = \int f_1(x) \, dx = 1

This implies the martingale condition.

Beyond simple hypotheses

H_0\colon \theta = \theta_0 \quad \text{vs} \quad H_1\colon \theta \in \mathcal{A}

Plug-in strategy: S_t = S_{t-1} \times \frac{f(X_t \mid \hat\theta_t)}{f(X_t \mid \theta_0)} \hat\theta_t = a(X_1, X_2, \dots, X_{t-1}) \in \mathcal{A}

Beyond simple hypotheses

H_0\colon \theta = \theta_0 \quad \text{vs} \quad H_1\colon \theta \in \mathcal{A}

Mixture strategy: S_t = S_{t-1} \times \frac{\int_{\mathcal{A}} f(X_t \mid \theta) \, g_t(\theta) \, d\theta}{f(X_t \mid \theta_0)} g_t(\theta) \text{ is a distribution over } \mathcal{A} g_t(\theta) \text{ can depend on } X_1, X_2, \dots, X_{t-1}

Martingales: origin

Martingales: origin

“Martingales”: 🎲 gambling strategies from 18th century France ♣️

These strategies cannot “beat the house”.

Martingales (in mathematics): represent the 💰 wealth over time of a gambler playing a fair game.

Betting analogy

Test statistic: S_t, a TSM under H_0

Intrepretation:

  • S_t is the wealth of a gambler betting against H_0
  • The gambler starts with $1
  • They can vary their bets based on past data.

Gambler gets rich \rightarrow evidence against H_0

Ville’s inequality: “one in a million chance of becoming a millionaire”

Confidence sequences

Fixed-sample hypothesis test \longleftrightarrow confidence interval (CI)
Sequential hypothesis test \longleftrightarrow confidence sequence (CS)

“E-hacking”

Reanalysing the data with different test statistics (“betting strategies”) and only reporting the best one.

This is not a safe procedure.

Fundamental principle:

Declare your bet before you play.

Pre-registration is still an important tool.

However, anytime-valid methods retain flexibility even after pre-specification.