AlphaNova
3

Request for review of potentially anomalous submission in Biweekly Competition — Season 1, Cycle 1

yuuuiv's avataryuuuiv
19d agocompetition-6-s1-b1

Hello AlphaNova Team,

I would like to request a review of a submission currently appearing near the top of the Biweekly Competition — Season 1, Cycle 1 leaderboard.

The submission is:

  • User: rudiyantoamdkom
  • Signal: signal_cf8edf53d881e57b

I noticed that the performance profile of this submission appears unusually smooth and stable compared with what I would normally expect from a genuine predictive signal on this dataset.

I also noticed that the competition rules explicitly state:

“Submissions with implausibly high validation Sharpe may be flagged for data reverse-engineering and disqualified.”

Given the unusually smooth performance profile and the very strong validation characteristics of this submission, I am concerned that it may be worth checking whether the result was produced entirely through legitimate predictive modeling, or whether it may have unintentionally benefited from dataset structure, target construction, evaluation-period information, leakage, or other forms of data reverse-engineering.

I am not asserting that the participant has violated the rules. I would simply appreciate it if the competition team could review the submission and confirm that it complies with the competition's anti-leakage and anti-reverse-engineering requirements.

In particular, could you please verify whether:

  • no future or target-derived information was used;
  • no hidden evaluation structure or unintended dataset artifact was exploited;
  • the validation Sharpe and unusually smooth performance are considered plausible under the competition framework.

Since this competition contributes to the Season 1 standings, I believe it would be helpful to ensure that the leaderboard remains fully consistent with the stated competition rules.

Thank you for your review.

Best regards

3 Replies

2
Nonius's avatarNonius19d ago

Hi yuuuiv,

Thanks for flagging this, will take a look and revert.

Best

NONIUS

0
Nonius's avatarNonius19d ago

Hi — thank you for the careful flag, and for the measured way you raised it.

We've completed the review. You were right to be suspicious: the submission's performance profile is not consistent with legitimate prediction on this target. Our examination of signal_cf8edf53d881e57b (and a closely related submission from the same account) found that the strategy inadvertently made use of future information — the no-look-ahead rule was violated, though from what we can see this looks like an honest mistake in how a couple of transforms were expressed, rather than deliberate reverse-engineering.

To your specific questions: yes, future-derived information entered the signal; no hidden evaluation artifact or target-construction exploit was involved; and no, the validation Sharpe is not plausible under the competition framework — that's precisely what triggered scrutiny.

On why it wasn't caught at submission time: our automated leakage screening rejected the common expressions of look-ahead but missed a corner case in how this particular form was written. As the rules note, passing the scan has never been a certification — "the no-look-ahead rule applies however a transform is expressed" — and this incident is exactly why that sentence exists. We won't be detailing the specific detection methods or the specific gap here (for obvious reasons), but the corner case is being closed, and we're adding further safeguards beyond pattern-matching as part of the same change.

Actions being taken:

  1. The affected signals will be re-flagged PEEK_THE_FUTURE and removed from the quality set and standings. The leaderboard will reflect this at an upcoming refresh.
  2. The participant will be notified with enough detail to fix their code — the underlying modeling work isn't the problem, the temporal handling is, and they're welcome to resubmit a causal version.
  3. Our screening is being hardened so this class of corner case is caught automatically going forward.

Season 1 standings will stay consistent with the stated rules — that's a commitment, and reports like this one help us keep it. Thanks again.

Best,
NONIUS

0
yuuuiv's avataryuuuiv19d ago

Hi NONIUS,

Thank you very much for the thorough review and detailed explanation.

I really appreciate the transparency around both the finding and the actions being taken. It’s reassuring to know that the issue was identified as a look-ahead problem rather than a deliberate attempt at reverse-engineering, and that the participant will have an opportunity to correct the temporal handling and resubmit a compliant version.

I also appreciate the clarification regarding the automated screening process. It makes sense that passing the initial scan is not the same as being fully certified against all possible forms of leakage, especially when edge cases can arise from how transforms are expressed.

Thank you as well for taking the report seriously and for strengthening the safeguards going forward. I’m glad the issue could be resolved in a way that preserves both fairness and the integrity of the Season 1 standings.

Best regards

Request for review of potentially anomalous submission in Biweekly Competition — Season 1, Cycle 1 | AlphaNova Discussions