Overfitting issues
Hi guys, quick question about the overfitting check. I've had several submissions flagged as OVERFITTING_FAILED despite passing my own local validation (walk-forward Sharpe, a permutation-based check comparing real vs. shuffled-label retraining, and reasonable regularization), and in some cases very similar submissions (same underlying signal family, just different windows/feature subsets) get inconsistent results, one passes and a close variant fails. Is there any general guidance on what the check actually looks for beyond raw Sharpe significance. Does it look at stability across time blocks, sensitivity to feature perturbation, or something else? Not asking about my specific submissions, just trying to have a general understanding of the mechanism so I can validate more effectively before using submission slots. Thanks!
2 Replies
Nonius24d agoHi Frozenyogurt,
the overfitting test is not really a sharpe signficance test, it's more like is the sharpe significantly better when training on real data than on "noise". sounds like you did such a test, but I don't know how many trials you did.
Best
Marc
Hi Marc, thanks so much, that's really helpful context! If it's not too much trouble, would it be possible to get a bit more visibility on a each of my failed submission. Is there a way to see the actual margin/p-value from the real test (rather than just the flagged/not-flagged result), or would you be able to take a quick look at one of my flagged submissions and let me know roughly where it fell short? No worries if that's not something you can share, just trying to calibrate my own local testing better. Appreciate the help!
Sign in to reply.