Finding highly skilled forecasters in real-time by comparing individuals to the wisdom of the crowd

What is it about?

Past work has shown that knowing how accurate forecasters have been in the past is a reliable way to predict who is likely to be accurate in the future. The problem is this requires a time lag. Forecasters must first make predictions, and then those predictions must resolve before they can be scored for accuracy. We show it is possible to evaluate forecasters in real-time based on their actual predictions by comparing them to the aggregate Wisdom of the Crowd.

Featured Image

Photo by jose aljovin on Unsplash

Photo by jose aljovin on Unsplash

Why is it important?

Finding analysts who can reliably make accurate predictions can help anticipate real worlds problems. However, until now, evaluating those analysts required asking them to make predictions, and then waiting for their predictions to resolve before their skill could be understood. Our method shows this can be done in real-time, without requiring any knowledge of the outcomes about the events they made predictions about.