Docs / Statistics
Statistics
Kraftverk treats statistics as decision aids with transparent defaults — not as a lab coat for marketing claims.
Per-sample summary
For a numeric vector of Kraft Index (or category) samples, Kraftverk records:
n, mean, median, min, max- Variance, standard deviation, coefficient of variation (CoV)
- Percentiles: p10 / p25 / p75 / p90 / p95
- Approximate mean confidence interval
- IQR outlier count
Comparison classes
Comparing candidate vs baseline (higher-is-better Kraft Index samples):
| Class | Meaning |
|---|---|
CONFIRMED_IMPROVEMENT | CIs separated; ≥ ~3% and candidate CI above baseline |
LIKELY_IMPROVEMENT | Positive shift without full confirmation |
NO_SIGNIFICANT_CHANGE | Overlap / below indifference threshold |
LIKELY_REGRESSION | Negative shift without full confirmation |
CONFIRMED_REGRESSION | CIs separated; ≤ ~-3% |
UNSTABLE_RESULT | High CoV or too few samples |
Defaults: 95% CI (z≈1.96), indifference 0.5%, outlier IQR k=1.5.
How the optimizer uses this
Search may track provisional improvements. Acceptance after validation requires stability PASS plus a LIKELY_* or CONFIRMED_* improvement class. High variance becomes UNSTABLE_RESULT / stability FAIL → reject.
kraftverk compare <a> <b>
kraftverk explain <experiment-id>Honesty
Small N and non-normal benches mean classifications are not formal hypothesis tests. Milestone work prioritizes transparency over pretending lab-grade inferential statistics. If the data is noisy, Kraftverk says so.
Do not treat a single short run as gospel across machines, suite versions, or background load. Fingerprints and suite versions matter.