How does the overfitting check decide?

Hi Nonius,

Could you clarify how the OVERFITTING_FAILED check reaches its verdict? Specifically: after retraining on shuffled labels, is the comparison against the real-label score a percentile-style test (real score vs the distribution of shuffled-label scores), or a threshold scaled by the dispersion of the shuffled runs, or something else (e.g. a fixed margin)? And is the retrain repeated multiple times or once?

Reason for asking: models whose flexible parts legitimately fit the label structure also fit shuffled labels somewhat better, which widens the shuffled-score spread. Whether the check normalizes by that spread or by rank changes which model classes can pass it at all, so knowing the rule would save wasted submissions on both sides.

Thanks!