Measuring What We Claim

The Brier score is the primary metric NEXUS uses to evaluate prediction quality. It measures the mean squared difference between predicted probabilities and actual outcomes. A score of 0.0 represents perfect prediction, while 1.0 represents the worst possible score.

For each resolved prediction, the Brier score is calculated as (forecast probability - outcome)^2, where outcome is 1 if the event occurred and 0 if it did not. A model that assigns 0.9 probability to an event that occurs receives a Brier contribution of 0.01, while assigning 0.9 to an event that does not occur yields 0.81.

NEXUS maintains a rolling 90-day Brier score alongside a lifetime aggregate. The rolling window allows detection of calibration drift, where the model begins systematically over- or under-estimating probabilities in a particular domain.

Every resolved prediction feeds back into the NEXUS calibration engine. When a prediction resolves (the event either occurs or the prediction window expires), the system records the outcome against the original forecast and updates category-level calibration weights.

The feedback loop operates on three timescales. Immediate adjustment applies a Bayesian update to the confidence modifier for the specific signal type. Weekly recalibration recomputes bin-level calibration curves across all categories. Monthly review triggers a full model audit that can adjust base rates, feature weights, and confidence thresholds.

This process has reduced the 90-day rolling Brier score from 0.241 at system launch to the current 0.187, a 22.4% improvement over 14 months of operation.