Two “Failed to run” results, local runner is clean

Same here, a few Failed to run submissions and error not reproducible locally.

Hi,
Thanks for reporting. We are looking at it.

Hi radiant-allomancer, we traced both runs. The failure is an AssertionError("prediction overlaps training prefix") – which is a guard inside your own submission, not a platform error. It fires because on our side predict() is called on feature rows that overlap the training slice: the serialization step trains on the current month's holdout-trimmed prefix and then probes predict() over the full frame (a timing check; its output is discarded), and daily scoring later predicts the growing current month against a model trained on its prefix. That call pattern is inherent to how current-month scoring works, and it is not exercised by the local runner – which is why your local run is clean. We accept the criticism that this contract wasn't documented; the kit README now states it explicitly (treat predict() as pure inference over whatever frame it receives, without asserting disjointness from training rows). Resubmitting with the guard removed will onboard normally. Neither run consumed a submission slot ("Failed to run" doesn't count against your 20).

Hi vocationalomission, we traced all four runs, and the fault was on our side. Your model is fully within the published speed limits (your measured train 39.7 s / predict 42.9 s against limits of 240 s / 60 s), but our overfitting gate ran its permutations under a fixed 90-minute budget – which arithmetically cannot accommodate models that are legal-but-not-fast. Your runs hit that ceiling, and the result was then mislabeled as a failure of your code. We have fixed the budget policy (the gate's time budget now scales with each model's own measured cost) and re-ran your four submissions under it – no resubmission was needed. All four completed: three onboarded successfully and are on your board; the fourth was rejected by the overfitting screen – a genuine model-quality verdict this time, shown on your submissions page. The earlier failed runs never counted against your submission cap; a run that completes successfully is a normal submission and counts as usual.

Hi alexander, thanks again for tracing the earlier two. Following your guidance I resubmitted with the prediction overlaps training prefix guard removed (my run uploaded 2026-08-24 12:40 UTC), and it also came back Failed to run / CANT_RUN. Locally the full-frame and growing-month predict() calls now run cleanly, so I've clearly missed a second place my code assumes something about the frame it is handed. Could you share the exact traceback for that run, the way you did before? My suspicion is a second guard assuming predict() only ever sees rows present when generate_features() was called - if the scoring path calls generate_features() on a narrower frame than a later predict(), I would like to fix that properly rather than guess. Thanks!

Hi alexander - reading your reply to vocationalomission, I think my case may be the same root cause rather than a second guard in my code. My model is heavy per fit (a wide random-feature solve): well inside the published per-call limits locally, but expensive when the gate retrains it many times over - exactly the legal-but-not-fast profile that hit the fixed 90-minute permutation budget. My three affected runs were uploaded 2026-08-20 10:44 UTC, 2026-08-20 11:10 UTC and 2026-08-24 12:40 UTC (the last one is the guard-removed version built on your earlier diagnosis). Could you check whether these hit the same budget ceiling, and if so re-run them under the new scaling policy as you did for vocationalomission? Happy to resubmit instead if that is cleaner on your side - I just want to avoid rebuilding against a cause that turns out not to be mine. Thanks for the quick turnarounds on all of this.

Hi radiant-allomancer, thanks for the quick turnaround on v122 – we traced the new failure, and here is the traceback you asked for: AssertionError: all-six-feature canonical sort tie. It fired on exactly 1 of the 101 anti-overfitting permutations – those retrain your model on label-shuffled variants of the data, and under one shuffle your canonical feature sort produced a tie your code treats as impossible. We don't adjudicate a run in which any permutation crashed (a data-dependent crash could bias the test), so the run was refused rather than scored.

On your budget question: we checked that specifically, and it is not the ceiling this time. The other 100 permutations completed within the scaled budget – the new policy already accommodates your model's retrain cost. The single crashed permutation is the only thing between v122 and a verdict.

On re-running the other submissions: v119/v120 still contain the overlap guard, so re-running them would fail identically at the serialization step – not worth your wait. Fix the tie assert in v122 (a stable tiebreaker instead of an assert) and it should onboard. The general rule, now in the kit README: treat train()/predict() as pure functions over whatever frame they receive – any assert about data properties can fire on the gate's shuffled inputs even when your real-data runs never trip it. As before, failed runs don't consume submission slots.

A clarification on my previous message, since I described the mechanism loosely. The gate's permutations are not label shuffles: it permutes the time rows of the whole panel – features and the residual panel together – and recomputes the target from the permuted data. So anything derived from row order changes, not just the target. That is why a feature-only quantity like your canonical sort can tie there while it never does on real data, and why the stable tiebreaker is the right fix.

Thanks alexander - that clarification is genuinely useful, and it explains our case exactly: a feature-only quantity that can never tie on real data can tie once rows are permuted. The stable tiebreaker fixed it and our resubmission reached a real verdict.

Two questions on the new "check novelty first" option, so I spend the 10 checks well:

1. What does the check actually run? Novelty only (correlation distance to the legacy pot, other admitted signals and your background book), or the overfitting gate too? Put differently: if a signal comes back novel, has it already cleared the overfitting screen, or can it still be rejected by that screen when it goes on to score?

2. What does the report contain - a binary novel/not-novel, or the actual correlation distance (and to the nearest occupant)? A distance would let people aim the next attempt rather than guess, which seems in everyone's interest given how the season pot accumulates.

Asking because the checks are metered, and the answer decides whether they are best spent immediately before a slot or as a mapping tool between cycles.

Hi how are you.

I need a halp

That's a really clear answer, thanks, and the folded vs signed distinction is worth knowing before spending one.

Lexy asked for opinions, so here's mine, from the position of someone whose submissions keep dying on the other gate.

The probe is useful, but it prices the check I keep passing and skips the one that keeps stopping me. My last two entries were both well clear on correlation and both got refused by the overfitting screen. A novelty probe would have said yes to each of them and saved me nothing. Anyone submitting variations of an existing signal has the opposite problem and will get real value from it, so it's not the wrong feature, it just doesn't bite for the people trying to bring something structurally different.

What would help far more is any signal about the overfitting screen before committing. Even something coarse, like where a model sat relative to the permutation distribution on a run that was refused, would tell me whether I'm marginally short or nowhere near. Right now a rejection is one bit, so the only way to learn the shape of that gate is to keep spending attempts on it, which I don't think is what you want either.

One small thing on the probe itself: since the signed number reads more novel than the board will later show, it'd be worth saying that in the UI next to the number. It's easy to read a comfortable-looking distance and assume it carries over to the published figure.

Not really, that would mean running the permutations every time someone probes, which is expensive for you.

What I want is the number you already have. When you refuse a run you know where it landed against the permutations. I get back one bit. So I can't tell whether I missed by a hair or was never close.

Putting that number on the refusal page costs you nothing and saves people burning attempts to work out something you already know.

Something else worth knowing, probably hits other people too.

Nonius said early on to slow the features down, longer lookbacks. I did that. Every version where the slowing is on the input side gets refused by the overfitting screen. Same model otherwise, one knob. Rolling or EWM on the features, refused. Leave them contemporaneous, goes through. Output side is fine, I have an accepted signal on a 1440h output span.

My guess is that rolling features on permuted rows are still smooth in the permuted order, and so is the overlapping target, so the retrain lines them up in sample and the null goes up. Could be wrong about the why.

Either way the advice and the gate disagree if you slow inputs, and the refusal doesn't tell you that.

Hi radiant-allomancer,

Hmmm, not sure mathematically that slowing down features should automatically lead to higher overfitting rates. having said that, if you optimize by looping over tons of lookbacks, that operation could yeiled overfitting. will take a look later on on your submsisssions to see if I can glean some thoughts on why it's happening. the main thing is this: with a high cost of turnover, you need to ffind the right cadence of turnover. slowing down features is one way, slowing down your prodictions (ie, ema) is another way. the only other options for mitigating high turnover and cost burn is having some threshold underwhich your predictions materailly change, but that gets more into strategy development.

Could well be wrong on the why, so worth saying what the versions actually are.

The lookback is a hardcoded constant in each file, not something the model fits or searches over. Each refused one is a single changed line against a parent that had already been accepted. I did build a few different lookbacks, but only one number is ever baked into a submitted file and you retrain it from scratch, so whatever I picked between shouldn't be visible to the gate.

Tightest pair if you do look: accepted 18 Aug 21:25, refused 19 Aug 06:00. Nine hours apart, same model, the refused one has a rolling window on the inputs. Same parent again refused on 20 Aug 12:11 with a different window length. There's an earlier one too, accepted 16 Aug 03:47 against refused 18 Aug 19:40, that one an EWM on the inputs.

On turnover, agreed, and that's already where I am. EMA on the output plus a threshold that holds the book unless the new prediction moves enough to pay for itself. Going at the input side was me having run out of output-side options, not preferring it.

One thing I want to get right before I use any of the novelty checks.

The blurb says a novel signal goes on to scoring automatically. Does that mean the check has then committed it, using a submission slot and putting it in the pot if it passes? Or is there a point after the number comes back where I can still say no?

Reason I care: if it auto-commits, then checking something genuinely new near the end of a cycle is the worst time to do it, because it lands in the pot for the rest of the season in a cycle it can't place in. If I can stop it, the check is useful any time.

Hi, radiant-allomancer, I'm fairly sure you can say "no". and not have it commit. @Lexy did the implementation. I'll try it myself right now

Hi, radiant-allomancer, got the answer, when it runs, you have the choice to either let it register and take a slot, or reject the submission.

Novelty reported
Not novel — global 7.82°, quality-gate 7.82°
Discarded