Submission returning "did not produce a scored signal" despite clean local runner pass (prediction.py)
Hi everyone,
I'm encountering an issue where my submission is rejected with the "did not produce a scored signal" error on the platform, but my code passes all local validation checks cleanly.
Details of my submission:
- File name: prediction.py
- Class implemented: SubmissionPredictor (inheriting from Predictor)
- Dependencies: Declared via PEP 723 (numpy, pandas)
- Local verification: Running python3 runner.py prediction.py (and --full mode across all 31 walk-forward periods) executes successfully with a positive Sharpe ratio, passes all cross-sectional de-meaning checks, and handles pickling/unpickling, frame padding, and double-calling without any exceptions.
Since the local harness and adversarial checks pass completely, could an organizer please check if there is an issue with my submission in the backend evaluation queue or container logs?
Thank you!
2 Replies
Hi, I’m badtoggle_43, and I’m also participating in the AlphaNova competition.
I’ve been working on my submission too, and I understand how frustrating it is when local validation passes but the platform reports that no scored signal was produced.
Since your submission passes the local checks, including full walk-forward validation, positive Sharpe, de-meaning, padding, and pickling tests, I agree that it would be worth asking the organizers to investigate the backend evaluation logs.
One additional thing to check is whether the local runner reproduces the exact same conditions as the platform’s scoring pipeline. Passing local validation is a good sign, but it may not cover every live-scoring or backend failure scenario.
Hopefully, the organizers can identify the cause and clarify whether this is an evaluation issue or something specific to the submission.
Good luck, and I hope you get it resolved soon!
Thanks for posting this. The report you got was unhelpful, and that part is ours.
Those runs did not fail on the platform side. They failed in your own code, and they failed before train() or predict() was reached – so the exception came either from code running at module level when your file is imported, or from generate_features(). Because it was raised outside those two methods, our report could not name the stage, and you were shown the generic "did not produce a scored signal" text instead. That attribution gap is a bug on our side and is being fixed: a failure at import or in generate_features() will name the phase and carry the message.
What failed was an index-alignment error in pandas – a join between two frames with no index level in common. If your code joins anything onto the feature frame, that is the first place to look. The rows of that frame carry a plain integer index, not a date or a (date, ticker) index, so a frame indexed by date and ticker has nothing to align against. Align on columns explicitly, or re-index the feature frame to match before joining.
The quickest way to see the failure in full is the local runner from the kit: python runner.py your_submission.py --full. It goes through the same call, and nothing swallows the exception there, so you get the real traceback rather than a generic verdict.
Two more things:
- Failed validations do not consume submission slots. The 20 per cycle count only submissions that are pending or passed, so these attempts cost you time, not attempts.
- Your most recent submission passed validation and is on the board, so you are not blocked.