Different handling of programs that fail to run? #192

Activity

1. codelion commented on Aug 10, 2025

   

   codelion

   on Aug 10, 2025

   Last edited by codelion

   Member

   More actions

   Thanks for raising this interesting point! Actually, OpenEvolve already implements a similar mechanism through its artifacts side-channel system. When programs fail with errors, the system captures detailed error information and passes it to the LLM in the next iteration, effectively achieving what you're suggesting.

   How It Works

   1. Error Capture in Evaluators

   When evaluation fails, evaluators can return detailed error information as artifacts. Here's an example from examples/circle_packing_with_artifacts/evaluator.py:331-349:

   except Exception as e:
       error_msg = f"Evaluation failed completely: {str(e)}"
       traceback.print_exc()
       return EvaluationResult(
           metrics={
               "sum_radii": 0.0,
               # ... other metrics set to 0
           },
           artifacts={
               "stderr": error_msg,
               "traceback": traceback.format_exc(),
               "failure_stage": "program_execution", 
               "suggestion": "Check for syntax errors, import issues, or runtime exceptions",
           },
       )

   2. Artifacts in Next Iteration's Prompt

   These artifacts are automatically included in the next generation's prompt through openevolve/prompt/sampler.py:476-479:

   # Format artifacts section if enabled and available
   artifacts_section = ""
   if self.config.include_artifacts and program_artifacts:
       artifacts_section = self._render_artifacts(program_artifacts)

   The artifacts appear in the prompt as:

   ## Last Execution Output
   
   ### stderr
   NameError: name 'np' is not defined
   
   ### traceback
   Traceback (most recent call last):
     File "...", line 15, in <module>
       centers = np.zeros((26, 2))
   NameError: name 'np' is not defined
   
   ### failure_stage
   program_execution
   
   ### suggestion
   Check for syntax errors, import issues, or runtime exceptions
   

   3. Real-World Example

   The examples/circle_packing_with_artifacts/README.md:89-112 shows a concrete example:

   Generation N: Program fails with overlapping circles

   centers[5] = [0.3, 0.3]  
   centers[6] = [0.3, 0.3]  # Same position - error!

   Artifacts captured:

   Circles 5 and 6 overlap: dist=0.000000, r1+r2=0.200000
   

   Generation N+1: LLM sees the overlap details and fixes it

   centers[5] = [0.3, 0.3]
   centers[6] = [0.5, 0.3]  # Different position - fixed!

   Configuration

   To enable artifacts (enabled by default):

   # In config.yaml
   include_artifacts: true
   max_artifact_bytes: 10000

   Or via environment variable:

   export ENABLE_ARTIFACTS=true

   Benefits Over Additional Error-Fixing Iteration

   The current approach has several advantages:

   1. No extra LLM calls - Error context is naturally included in the next evolution iteration
   2. Maintains evolution flow - Programs still compete based on fitness, even failed ones
   3. Rich context - The LLM sees not just the error but also performance metrics and evolution history
   4. Flexible - Evaluators can provide domain-specific error guidance

   For Your Use Case

   If you're seeing many syntax errors with Gemini 2.5 Pro and Claude Opus, you might want to:

   1. Ensure artifacts are enabled in your config
   2. Enhance your evaluator to capture more detailed error information
   3. Consider adding domain-specific error suggestions in the artifacts

   The system already provides the error feedback mechanism you're looking for - it just needs to be leveraged through proper evaluator implementation!

   Let me know if you'd like examples of how to enhance your evaluator to provide better error feedback through artifacts.
2. benjaminy commented on Aug 10, 2025

   

   benjaminy

   on Aug 10, 2025

   Author

   More actions

   Thank you for your detailed response. I am already using artifacts. (Of course maybe the details matter and I could be doing something better.) One sanity check: include_artifacts defaults to True, right? So this mechanism is enabled if no value is given in the config file?

   Is there any special logic/handling for the keys in the artifacts dict in your example (stderr, traceback, failure_stage, suggestion)?

   I wouldn't say I'm seeing many errors. Maybe 5% of iterations (rough estimate with high variance). But when I see an iteration fail because the generated code refers to a nonexistent variable or function it feels like I would rather not include that version in the archive at all. Either give the LLM a chance to fix the uninteresting mistake(s) or "skip" that version entirely. I can see how that would be bad if the system was struggling to generate any good versions; I can see the importance of error message feedback in that case. But that's not what I'm seeing.

   Your "Maintain evolution flow" point feels off to me, but I'm not sure. Say I'm trying to optimize cars and in one test a car gets a nail in its tire and completely fails. That doesn't seem like an interesting test to include. Or maybe a better analogy would be the test driver shows up drunk and crashes to no particular fault of that car design.
3. algorithmicsuperintelligence

   locked and limited conversation to collaborators on Aug 10, 2025
4. codelion

   converted this issue into a discussion #194 on Aug 10, 2025