Repository navigation
The issue of the temperature parameter #162
Description
I’ve encountered something confusing while using the OpenEvolve framework, and I’d appreciate some clarification.
In the OpenAILLM class in your openai.py file, there is this code block when setting up generation parameters:
if self.api_base == "https://api.openai.com/v1" and str(self.model).lower().startswith("o"):
# For o-series models
params = {
"model": self.model,
"messages": formatted_messages,
"max_completion_tokens": kwargs.get("max_tokens", self.max_tokens),
}
else:
params = {
"model": self.model,
"messages": formatted_messages,
"temperature": kwargs.get("temperature", self.temperature),
"top_p": kwargs.get("top_p", self.top_p),
"max_tokens": kwargs.get("max_tokens", self.max_tokens),
}
It seems that when the model name starts with "o" (e.g., "o4-mini"), the temperature parameter is not passed to the API.
I’ve also tested an example based on your function_minimization. I set temperature values like 0.3 vs 0.7 with o4-mini(just set one model) and observed significant behavioral differences in the output, which suggests that these parameters are indeed supported and have an effect.
This is my config.yaml
max_iterations: 10
checkpoint_interval: 5
log_level: "INFO"
primary_model: "o4-mini"
primary_model_weight: 1.0
temperature: 0.3
api_base: "XXXX"
api_key: "XXXX"
max_tokens: 4096
prompt:
include_artifacts: true
system_message: "You are an expert programmer specializing in optimization algorithms. Your task is to improve a function minimization algorithm to find the global minimum of a complex function with many local minima. The function is f(x, y) = sin(x) * cos(y) + sin(x*y) + (x^2 + y^2)/20. Focus on improving the search_algorithm function to reliably find the global minimum, escaping local minima that might trap simple algorithms."
evaluator_system_message: "You need to ensure each round score is improved by the last best one"
num_top_programs: 1
max_artifact_bytes: 4096
use_template_stochasticity: true
artifact_security_filter: true
database:
population_size: 1
archive_size: 1
num_islands: 1
elite_selection_ratio: 0.3
exploitation_ratio: 0.7
evaluator:
timeout: 60
cascade_evaluation: true
enable_artifacts: true
diff_based_evolution: false
allow_full_rewrites: true
Testing result
When Temperature = 0.7
0.767818. 0.954938 0.945888 0.952813 0.951586 0.779630 0.952998 0.797274 0.787243 0.986230 0.974237
Temperature = 0.3
0.767818 0.706419 0.784706 0.777148 0.988748 0.994253 0.702351 0.797274 0.999715 0.981818 0.999708
three questions:
Q1 Why are the two results different? If the temperature cannot be passed, the result should be the same.
Q2 Can OpenEvolve guarantee that each successive round’s score improves over the previous one and demonstrates a steadily converging trend?
Q3 Whether or not they can set elite_selection_ratio=1.0 and exploitation_ratio=1.0 at the same time?
Could you please provide some explanations? Thank you so much.
Activity
codelion commentedon Jul 25, 2025
Yeah it was added for older models
• o1
• o1-mini
• o3-mini
They do not allow sampling via temperature. We can tighten the conditional to only skip it for these models and warn the user. Would appreciate a PR if you can do it I will take a look at it and merge.
Ethan-han921 commentedon Jul 25, 2025
When I tried 'o3mini', give this config
There is no warning or error, it shows:
2025-07-24 22:03:10,917 - INFO - Sampled model: o3-mini
[DEBUG] LLM params: dict_keys(['model', 'messages', 'max_completion_tokens'])
2025-07-24 22:03:10,925 - INFO - New MAP-Elites cell occupied: {'complexity': 6, 'diversity': 0}
2025-07-24 22:03:10,925 - INFO - Population size (3) exceeds limit (1), removing 2 programs
2025-07-24 22:03:10,925 - INFO - Population size after cleanup: 2
2025-07-24 22:03:10,925 - INFO - Iteration 1: Program 6f158a7a-30b7-4080-bf7c-f2ce5f09fa6f (parent: c6a4d687-8eb8-4c92-8479-c92d54f9badf) completed in 16.44s
2025-07-24 22:03:10,925 - INFO - Metrics: runs_successfully=1.0000, value_score=0.7352, distance_score=0.2402, overall_score=0.1813, standard_deviation_score=0.3251, speed_score=1.0000, reliability_score=1.0000, combined_score=0.5064, success_rate=1.0000
2025-07-24 22:03:27,838 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-07-24 22:03:28,608 - INFO - Evaluated program 4f83cf3e-cd76-4031-b26b-7add371dcf6e in 0.77s: runs_successfully=1.0000, value_score=0.9991, distance_score=0.9738, overall_score=0.9970, standard_deviation_score=0.9738, speed_score=1.0000, reliability_score=1.0000, combined_score=0.9852, success_rate=1.0000
2025-07-24 22:03:28,609 - INFO - Sampled model: o3-mini
[DEBUG] LLM params: dict_keys(['model', 'messages', 'max_completion_tokens'])
2025-07-24 22:03:28,616 - INFO - New MAP-Elites cell occupied: {'complexity': 9, 'diversity': 9}
2025-07-24 22:03:28,616 - INFO - Population size (3) exceeds limit (1), removing 2 programs
2025-07-24 22:03:28,616 - INFO - Population size after cleanup: 2
2025-07-24 22:03:28,616 - INFO - New best program 4f83cf3e-cd76-4031-b26b-7add371dcf6e replaces c6a4d687-8eb8-4c92-8479-c92d54f9badf (combined_score: 0.7678 → 0.9852, +0.2174)
2025-07-24 22:03:28,616 - INFO - Iteration 2: Program 4f83cf3e-cd76-4031-b26b-7add371dcf6e (parent: cb41e7cb-5427-4976-bb2b-fc18f6d95ee4) completed in 17.69s
2025-07-24 22:03:28,616 - INFO - Metrics: runs_successfully=1.0000, value_score=0.9991, distance_score=0.9738, overall_score=0.9970, standard_deviation_score=0.9738, speed_score=1.0000, reliability_score=1.0000, combined_score=0.9852, success_rate=1.0000
2025-07-24 22:03:28,617 - INFO - 🌟 New best solution found at iteration 2: 4f83cf3e-cd76-4031-b26b-7add371dcf6e
2025-07-24 22:03:46,329 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-07-24 22:03:46,522 - INFO - Evaluated program 2969095a-00c3-4e6b-97e7-1c510907d500 in 0.19s: runs_successfully=1.0000, value_score=0.9166, distance_score=0.3720, overall_score=0.9258, standard_deviation_score=0.3893, speed_score=1.0000, reliability_score=1.0000, combined_score=0.6289, success_rate=1.0000
Ethan-han921 commentedon Jul 25, 2025
I will try.