Calibration Is the Objective
Calibra Labs develops forecasting systems that produce well-calibrated probability estimates on real-world questions. We build ensemble methods, evaluate them on public benchmarks, and publish what we learn.
What We Do
Approach
We study how to produce probability estimates that are both sharp and well-calibrated. A forecast of 70% should resolve positively about 70% of the time. This sounds simple. In practice, it requires careful methodology: combining diverse information sources, managing model disagreement, and resisting the temptation to be overconfident.
Our systems are evaluated against proper scoring rules on questions with unambiguous resolution criteria. We compete on public benchmarks where the results speak for themselves.
Calibration First
We optimize for calibration under proper scoring rules. A forecast is only as good as its reliability across many predictions.
Diverse Evidence
Good forecasts integrate multiple information sources. No single model or data stream is sufficient on its own.
Public Evaluation
We compete on benchmarks with published scores. Claims about forecasting ability should be verifiable.
Current Work
Benchmarks
We are currently competing on ForecastBench, a dynamic benchmark of forecasting accuracy maintained by the Forecasting Research Institute. ForecastBench evaluates systems on 500 questions spanning prediction markets, economic indicators, geopolitical events, and other real-world outcomes, scored on calibration and accuracy.
Results will appear here once our first round of scores is published. ForecastBench scores are updated biweekly.
We are also exploring evaluation on other forecasting benchmarks and tournaments. More details as results become available.
Get in Touch
Contact
For inquiries about our research, collaboration opportunities, or anything else, reach out at contact@calibralabs.org.