Director-AI Julia Threshold Tuner

DirectorThresholdTuner

Offline analytics companion for Director-AI. Given historical (score, label) pairs — produced by any scorer backend against a labelled eval set (AggreFact, HaluEval, custom KB) — compute:

1. Grid-search optimum — threshold that maximises balanced accuracy on the sample.
2. Bootstrap 95% CI — non-parametric resampling over the per-example pairs to bound threshold uncertainty.
3. Bayesian posterior — Turing.jl model with Beta prior on the threshold and Bernoulli likelihood, yielding a posterior mean and 95% credible interval.

The module is deliberately standalone — no FFI into Python, no runtime dependency from Director-AI on Julia. Operators run it as a CLI against exported score logs, inspect the CI band, and commit a threshold back into DirectorConfig.

ScoredExample

One labelled observation from a scorer run.

Fields:

• score::Float64 — scorer output in $[0, 1]$ (higher = more confident the response is grounded / supported).
• label::Bool — ground-truth label (true = grounded, false = hallucinated). Mapped from 0/1 integers on load.

TuneResult

Combined output of tune. All thresholds are on the $[0, 1]$ scale.

balanced_accuracy(examples, threshold)

Standard balanced-accuracy metric: the arithmetic mean of the true positive rate and the true negative rate at the given threshold.

Returns NaN when either class is empty, matching scikit-learn's behaviour for degenerate label distributions.

bayesian_threshold_posterior(examples; n_samples, n_warmup, rng)

Fit a simple generative model: a threshold $t$ drawn from $Beta(2, 2)$ where each example's label is Bernoulli(0.99 or 0.01) depending on whether its score exceeds $t$. MCMC with NUTS yields the posterior over $t$.

Returns (mean, std, lo, hi, chain) with lo/hi the 95% credible-interval bounds and chain the underlying Chains object for downstream diagnostics.

bootstrap_threshold_ci(examples; n_resamples, ci, rng)

Non-parametric bootstrap over (score, label) pairs. For each resample, optimise_threshold is called; the returned thresholds form the empirical distribution whose ci-level quantiles give the confidence interval.

• n_resamples — default 2000. The law of large numbers suggests

  1000 for stable 95% quantiles.

• ci — default 0.95.

Returns a named tuple (lo, hi, mean, std, samples) with samples being the raw bootstrap vector (useful for plotting).

load_scores_jsonl(path) -> Vector{ScoredExample}

Read a JSON-lines file. Each line must be an object with at least a numeric score field and a label field (accepted as boolean, 0/1, or the strings "true"/"false"). Other keys are ignored, so the Python feeder can emit a superset of fields.

optimise_threshold(examples; grid)

Grid-search the threshold that maximises balanced accuracy. grid defaults to a 201-point linear grid over $[0, 1]$ (0.5% steps).

Returns (threshold, balanced_accuracy).

save_result_json(path, result)

Write the TuneResult as a JSON object to path, replacing any existing file. The schema is a flat, language-neutral document — deliberately matched to what the Python side expects from tools/prepare_threshold_data.py.

tune(examples; rng, n_bootstrap, n_bayes_samples, n_bayes_warmup)

Orchestrates all three analyses and returns a TuneResult. This is the single call sites should use from Python or the CLI.