what's up brothers and sisters?
we offering up to 50K usd? happy we got 19 quality signals (and many more that look good! but, come on! some good talent on the board). lets go. we got the cash, you got the talent. CASH is waiting.
6 Replies
kinda funny when I was pitching to our stakeholders: "the worst case is we DON'T pay out a lot of money". well, that was a take me to the mountain event, but, in a way that's reality. we'd be HAPPY to pay 50K to winners. if it doesn't work out that way, we'll roll the cash pot over into new contests, and, by the way, not on same target prediction. It's up to all of you.
Please help me understand evaluation more precisely. Let's consider current LB#1 as an example. Why is it #1? All metrics from LB are far from being first, especially City & Global novelties. I thought the most important ones are Sharpe & City/Global Novelty, but seems to be false. So, what should happen to somebody to overcome current LB#1 and become new leader? And last question: Do you recalculate LB scores from time to time, or it remains consistent for the whole time of the competition? Thanks!
Thanks for the thoughtful question — it gets at something subtle, so let me lay out the actual mechanism.
The board is not ranked by any single metric. It ranks quality signals, and "quality" is determined by a greedy, diversification-aware selection — not by Sharpe alone, and not by novelty alone:
1. We take all competition signals whose Sharpe is statistically significant (positive beyond ~2 standard errors — a noisy Sharpe doesn't qualify).
2. We sort those candidates by Sharpe, descending.
3. We maintain an "occupied" set, seeded with the existing legacy signals (see the legacy signal panel).
4. Walking down the Sharpe-sorted list, a signal is admitted only if its correlation with every signal already in the occupied set is below the threshold (|corr| ≤ 0.5, i.e. at least ~60° apart). When admitted, it joins the occupied set, so each subsequent signal must also be uncorrelated with it.
By the way, this is all laid out in the Overview of the competition as well as in the COMPETITION markdown file we provided in the zip file!
So the leaderboard ordering is the order in which signals get admitted by this process; after that, the rest of the leaderboard is ranked purely by Sharpe.
Why is the current #1 ranked first despite unremarkable novelty? Because it was the highest-Sharpe signal that was sufficiently uncorrelated with the already-occupied set to be admitted first. Two things to note:
1. The novelty numbers you see (City / Global) measure distance to the nearest of all signals on the board, including the legacy board. The selection, by contrast, only checks correlation against the occupied/selected set. A signal can sit close to some signal that was never selected — giving it a modest novelty score — while still being independent from everything that actually occupies the portfolio. That's exactly why a high-Sharpe signal with middling novelty can lead.
2. Novelty is informational; it is not the sort key.
A concrete example — your exact case. The signal at #1 actually has a lower Sharpe than two other signals on the board. Those two higher-Sharpe signals are each strongly correlated with one existing legacy signal, so they're treated as redundant and don't qualify as quality signals. The #1 signal's correlation with that same legacy signal falls below the 0.5 cutoff, so it counts — and leads:
Correlation is measured against the legacy signal signal_305ff072:
| Signal | Sharpe | Correlation | Result |
|---|---|---|---|
| signal_fb519c1f | 0.0502 | +0.61 | rejected (redundant) |
| signal_a730af01 | 0.0479 | +0.53 | rejected (redundant) |
| signal_deec7b91 | 0.0447 | +0.44 | admitted (now #1) |
The two stronger-Sharpe signals cross the 0.5 line against an already-occupied legacy signal; the #1 signal stays just under it, so it's the one that represents genuinely new alpha.
An analogy. Think of every signal as a city on the globe, and a city's quality as how far north it is — the closer to the North Pole, the better (that's the Sharpe). But we're not just collecting northerly cities; we're building a map where cities are spread out, because two cities in the same spot tell you nothing new.
So the rule for adding a city to the map is:
1. Consider candidate cities from furthest north to furthest south.
2. The map already has some cities on it to begin with — the legacy signals.
3. A new city is added only if it's far enough away from every city already on the map. If it sits too close to one that's already there, it's turned away — even if the newcomer is further north than the city it's crowding. Once a city is added, it's on the map, and every later city has to keep its distance from it too.
That's exactly what happened here. Someone founded a city very far north — but right next to an existing legacy city (signal_305ff072). It didn't matter that the newcomer sat further north than that legacy city: it was too close, so it was turned away as redundant. By founding a city a bit further away — and giving up a little latitude in the process — the same person got a city admitted to the map, and because it was the northernmost of the admitted cities, it landed at #1.
What does it take to overtake #1? You need a signal that is both:
-
\high Sharpe (and significantly so), and
-
\uncorrelated (|corr| ≤ 0.5) with the signals already occupying the space — the legacy/background set and any higher-Sharpe signals ahead of you.
Pure Sharpe won't do it if you're too correlated with what's already there; pure novelty won't do it if your Sharpe isn't strong and significant. The win condition is genuine, independent alpha.
Do scores get recalculated? Yes — continuously. We score against real market data as it arrives, re-run the evaluation daily, and re-run this selection. So as the live month accumulates more out-of-sample data, every signal's Sharpe updates and the quality selection can shift day to day. The rules don't change during the competition — but the numbers they operate on are live.
Oh thanks a lot! Huge answer, yeah all in one place for me to understand the evaluation steps!
Sure . By the way . It’s pretty easy to show that if you take some signalsgenerated by look back windows of varying length. You can see by computing the correlations between those signals that the correlation decays as a function of delta of look back length . Roughly speaking . Same goes for smoothing your final signal with an EMA . So that’s one way to test whether your look backs and smoothing are in the right ballpark to not collide with existing signals . Well at least with the legacy signals .
But that’s just one dimension of moving away from correlated signals . Other ideas are how you feature engineer the features and , well I never tested this but maybe make use of any seasonality effects you might detect
By the way . We WANT to pay 50k. That is our optimal scenario .
Sign in to reply.