No logs, but the submission failed...

[tgbaffetti]

I am hosting a private ML hackathon on Codabench (https://www.codabench.org/competitions/17664/) . Sometimes I have this problem:

• I submit the model in the right .zip format
• the training runs smoothly through the ingestion program (no logs)
• then it stops and its status is "failed"
  I received no logs through the submission panel. The only message that I see is the "i" close to the status "failed" and it says "submission failed: bad or empty .zip file. See logs for more details"... but as I said I have no logs and the zip file is clearly not empty because it works for the ingestion phase... At the beginning I thought it was a memory issue because I was not facing this problem with some light model. However I have this problem even with models that require a peak of 6GB of RAM while running...
  Can you please help me understand what is the issue?

[ObadaS]

Can you send us a copy of a submission that fails like this ?
You can send it to info@codabench.org

[Didayolo]

Hi @tgbaffetti,

Testing the submission you sent (sample_code_submission_PCAMLP_3.zip) on the latest compute worker code (queue Codabench-Test), I get different error logs that looks solvable:

Traceback (most recent call last):
  File "/app/program/ingestion.py", line 176, in <module>
    D = M.preprocess(train_dir)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ingested_program/model.py", line 82, in preprocess
    state, phi = load_simulation(os.path.join(data_folder, sim_name))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ingested_program/utils.py", line 55, in load_simulation
    state = np.load(os.path.join(sim_folder, 'state.npz'))['data']
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/numpy/lib/_npyio_impl.py", line 454, in load
    fid = stack.enter_context(open(os.fspath(file), "rb"))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/input_data/train/sineSweep_f1_f80_A02/state.npz'

Have you seen this error before?

[Didayolo]

Hi,

It looks like the problem is solved? We were able to run submissions on a clone of your competition, and I can see the last uploaded competition (https://www.codabench.org/competitions/18039/) has a finished submission.

We hope that all is going well for your event.