This class implements FunSearch. This code is an implementation of Funsearch (https://www.nature.com/articles/s41586-023-06924-6) and is heavily inspired by the original code (https://github.com/google-deepmind/funsearch) . It's a Flow in charge of starting, stopping and managing (passing around messages) the FunSearch process. It passes messages around to the following subflows:
ProgramDBFlow: which is in charge of storing and retrieving programs.
SamplerFlow: which is in charge of sampling programs.
EvaluatorFlow: which is in charge of evaluating programs.
Configuration Parameters:
name (str): The name of the flow. Default: "FunSearchFlow".
description (str): The description of the flow. Default: "A flow implementing FunSearch"
subflows_config (Dict[str,Any]): A dictionary of subflows configurations. Default:
ProgramDBFlow: By default, it uses the ProgramDBFlow class and uses its default parameters.
SamplerFlow: By default, it uses the SamplerFlow class and uses its default parameters.
EvaluatorFlow: By default, it uses the EvaluatorFlow class and uses its default parameters.
Input Interface:
from (str): The flow from which the message is coming from. It can be one of the following: "FunSearch", "SamplerFlow", "EvaluatorFlow", "ProgramDBFlow".
operation (str): The operation to perform. It can be one of the following: "start", "stop", "get_prompt", "get_best_programs_per_island", "register_program".
content (Dict[str,Any]): The content associated to an operation. Here is the expected content for each operation:
"start":
num_samplers (int): The number of samplers to start up. Note that it's still restricted by the number of workers available. Default: 1.
"stop":
No content. Pass either an empty dictionary or None. Works also with no content.
"get_prompt":
No content. Pass either an empty dictionary or None. Works also with no content.
"get_best_programs_per_island":
No content. Pass either an empty dictionary or None. Works also with no content.
Output Interface:
retrieved (Dict[str,Any]): The retrieved data.
Citation:
@Article{FunSearch2023,
author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein},
journal = {Nature},
title = {Mathematical discoveries from program search with large language models},
year = {2023},
doi = {10.1038/s41586-023-06924-6}
}
make_request_for_prompt
def make_request_for_prompt()
This method makes a request for a prompt. It sends a message to itself with the operation "get_prompt" which will trigger the flow to call the ProgramDBFlow to get a prompt.
request_samplers
def request_samplers(input_message: FlowMessage)
This method requests samplers. It sends a message to itself with the operation "get_prompt" which will trigger the flow to call the ProgramDBFlow to get a prompt.
Arguments:
input_message (FlowMessage): The input message that triggered the request for samplers.
get_next_state
def get_next_state(input_message: FlowMessage)
This method determines the next state of the flow based on the input message. It will return the next state based on the current state and the message received.
Arguments:
input_message (FlowMessage): The input message that triggered the request for the next state.
Returns:
str: The next state of the flow.
set_up_flow_state
def set_up_flow_state()
This method sets up the state of the flow. It's called at the beginning of the flow.
This method saves a message to the state of the flow. It's used to keep track of state on a per message basis (i.e., state of the flow depending on the message received and id).
This method renames a key in the state of the flow in the "msg_requests" dictonary. It's used to rename a key in the state of the flow (i.e., rename a message id).
Arguments:
old_key (str): The old key to rename.
new_key (str): The new key to rename to.
message_in_state
def message_in_state(msg_id: str) -> bool
This method checks if a message is in the state of the flow (in "msg_requests" dictionary). It returns True if the message is in the state, otherwise it returns False.
Arguments:
msg_id (str): The id of the message to check if it's in the state.
Returns:
bool: True if the message is in the state, otherwise False.
This method pops a message from the state of the flow (in "msg_requests" dictionary). It the state associate to a message and removes it from the state.
Arguments:
msg_id (str): The id of the message to pop from the state.
Returns:
Dict[str,Any]: The state associated with the message id.
This method registers the input message data to the flow state. It's called everytime a new input message is received.
Arguments:
input_message (FlowMessage): The input message
call_program_db
def call_program_db(input_message)
This method calls the ProgramDBFlow. It sends a message to the ProgramDBFlow with the data of the input message.
Arguments:
input_message (FlowMessage): The input message to send to the ProgramDBFlow.
call_evaluator
def call_evaluator(input_message)
This method calls the EvaluatorFlow. It sends a message to the EvaluatorFlow with the data of the input message.
Arguments:
input_message (FlowMessage): The input message to send to the EvaluatorFlow.
call_sampler
def call_sampler(input_message)
This method calls the SamplerFlow. It sends a message to the SamplerFlow with the data of the input message.
Arguments:
input_message (FlowMessage): The input message to send to the SamplerFlow.
generate_reply
def generate_reply(input_message: FlowMessage)
This method generates a reply to a message sent to user. It packages the output message and sends it.
Arguments:
input_message (FlowMessage): The input message to generate a reply to.
run
def run(input_message: FlowMessage)
This method runs the flow. It's the main method of the flow. It's called when the flow is executed.
ProgramDBFlowModule
ProgramDBFlow Objects
class ProgramDBFlow(AtomicFlow)
This class implements a ProgramDBFlow. It's a flow that stores programs and their scores in a database. It can also query the database for the best programs or generate a prompt containing stored programs in order to evolve them with a SamplerFlow. This code is an implementation of Funsearch (https://www.nature.com/articles/s41586-023-06924-6) and is heavily inspired by the original code (https://github.com/google-deepmind/funsearch)
Configuration Parameters:
name (str): The name of the flow. Default: "ProgramDBFlow"
description (str): A description of the flow. This description is used to generate the help message of the flow. Default: " A flow that saves programs in a database of islands"
artifact_to_evolve_name (str): The name of the artifact/program to evolve. Default: "solve_function"
evaluate_function (str): The function used to evaluate the program. No Default value. This MUST be passed as a parameter.
evaluate_file_full_content (str): The full content of the file containing the evaluation function. No Default value. This MUST be passed as a parameter.
num_islands: The number of islands to use. Default: 3
reset_period: The period in seconds to reset the islands. Default: 3600
artifacts_per_prompt: The number of previous artifacts/programs to include in a prompt. Default: 2
temperature: The temperature of the island. Default: 0.1
temperature_period: The period in seconds to change the temperature. Default: 30000
sample_with_replacement: Whether to sample with replacement. Default: False
portion_of_islands_to_reset: The portion of islands to reset. Default: 0.5
template (dict): The template to use for a program. Default: {"preface": ""}
Input Interface:
operation (str): The operation to perform. It can be one of the following: ["register_program","get_prompt","get_best_programs_per_island"]
Output Interface:
retrieved (Any): The retrieved data. It can be one of the following:
If the operation is "get_prompt", it can be a dictionary with the following keys
code (str): The code of the prompt
version_generated (int): The version of the prompt generated
island_id (int): The id of the island that generated the prompt
header (str): The header of the prompt
If the operation is "register_program", it can be a string with the message "Program registered" or "Program failed to register"
If the operation is "get_best_programs_per_island", it can be a dictionary with the following keys:
best_island_programs (List[Dict[str,Any]]): A list of dictionaries with the following keys:
rank (int): The rank of the program (1 is the best)
score (float): The score of the program
program (str): The program
island_id (int): The id of the island that generated the program
Citation:
@Article{FunSearch2023,
author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein},
journal = {Nature},
title = {Mathematical discoveries from program search with large language models},
year = {2023},
doi = {10.1038/s41586-023-06924-6}
}
set_up_flow_state
def set_up_flow_state()
This method sets up the state of the flow and clears the previous messages.
get_prompt
def get_prompt()
This method gets a prompt from an island. It returns the code, the version generated and the island id.
reset_islands
def reset_islands()
This method resets the islands. It resets the worst islands and copies the best programs to the worst islands.
This method registers a program in an island. It also updates the best program if needed.
Arguments:
program (AbstractArtifact): The program to register
island_id (int): The id of the island to register the program
scores_per_test (ScoresPerTest): The scores per test of the program
get_best_programs
def get_best_programs() -> List[Dict[str, Any]]
This method returns the best programs per island.
run
def run(input_message: FlowMessage)
This method runs the flow. It performs the operation requested in the input message.
SamplerFlowModule
SamplerFlowModule.SamplerFlow
SamplerFlow Objects
class SamplerFlow(ChatAtomicFlow)
This class implements a SamplerFlow. It is a flow that queries a LLM to generate a response to a given input. This class is a child of ChatAtomicFlow.
and expects the same parameters as ChatAtomicFlow (see https://huggingface.co/aiflows/ChatFlowModule).
Configuration Parameters:
name (str): The name of the flow. Default: "SamplerFlowModule"
description (str): A description of the flow. Default: "A flow that queries an LLM model to generate prompts for the Sampler flow"
backend Dict[str,Any]: The backend of the flow. Used to call models via an API.
See litellm's supported models and APIs here: https://docs.litellm.ai/docs/providers.
The default parameters of the backend are all defined at aiflows.backends.llm_lite.LiteLLMBackend
(also see the defaul parameters of litellm's completion parameters: https://docs.litellm.ai/docs/completion/input#input-params-1).
Except for the following parameters who are overwritten by the ChatAtomicFlow in ChatAtomicFlow.yaml:
model_name (Union[Dict[str,str],str]): The name of the model to use. Default: "gpt-4"
When using multiple API providers, the model_name can be a dictionary of the form
{"provider_name": "model_name"}. E.g. {"openai": "gpt-3.5-turbo", "azure": "azure/gpt-3.5-turbo"}
Default: "gpt-3.5-turbo" (the name needs to follow the name of the model in litellm https://docs.litellm.ai/docs/providers).
n (int) : The number of answers to generate. Default: 1
max_tokens (int): The maximum number of tokens to generate. Default: 2000
temperature float: The temperature of the generation. Default: 1.0
top_p float: An alternative to sampling with temperature. It instructs the model to consider the results of
the tokens with top_p probability. Default: 0.4
frequency_penalty (number): It is used to penalize new tokens based on their frequency in the text so far. Default: 0.0
presence_penalty (number): It is used to penalize new tokens based on their existence in the text so far. Default: 0.0
stream (bool): Whether to stream the response or not. Default: false
system_message_prompt_template (Dict[str,Any]): The template of the system message. It is used to generate the system message. Default: See SamplerFlow.yaml for default.
init_human_message_prompt_template (Dict[str,Any]): The prompt template of the human/user message used to initialize the conversation
(first time in). It is used to generate the human message. It's passed as the user message to the LLM. Default: See SamplerFlow.yaml for default.
human_message_prompt_template (Dict[str,Any]): The prompt template of the human/user message (message used everytime the except the first time in).
It's passed as the user message to the LLM. Default: See SamplerFlow.yaml for default.
previous_messages (Dict[str,Any]): Defines which previous messages to include in the input of the LLM. Note that if first_kand last_k are both none,
all the messages of the flows's history are added to the input of the LLM. Default:
first_k (int): If defined, adds the first_k earliest messages of the flow's chat history to the input of the LLM. Default: 1
last_k (int): If defined, adds the last_k latest messages of the flow's chat history to the input of the LLM. Default: 1
Input Interface Initialized (Expected input the first time in flow):
header (str): A header message to include in prompt
code (str): The "example" samples to generate our new sample from.
Input Interface (Expected input the after the first time in flow):
header (str): A header message to include in prompt
code (str): The "example" samples to generate our new sample from.
Output Interface:
api_output (str): The output of the API call. It is the response of the LLM to the input.
from (str): The name of the flow that generated the output. It's always "SamplerFlow"
Citation:
@Article{FunSearch2023,
author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein},
journal = {Nature},
title = {Mathematical discoveries from program search with large language models},
year = {2023},
doi = {10.1038/s41586-023-06924-6}
}
run
def run(input_message)
This method calls the backend of the flow (so queries the LLM). It calls the backend with the previous messages of the flow.
name (str): The name of the flow. Default: "EvaluatorFlow"
description (str): A description of the flow. This description is used to generate the help message of the flow. Default: "A flow that evaluates code on tests"
py_file (str): The python code containing the evaluation function. No default value. This MUST be passed as a parameter.
function_to_run_name (str): The name of the function to run (the evaluation function) in the evaluator file. No default value. This MUST be passed as a parameter.
test_inputs (Dict[str,Any]): A dictionary of test inputs to evaluate the program. Default: {"test1": None, "test2": None}
timeout_seconds (int): The maximum number of seconds to run the evaluation function before returning an error. Default: 10
run_error_score (int): The score to return if the evaluation function fails to run. Default: -100
use_test_input_as_key (bool): Whether to use the test input parameters as the key in the output dictionary. Default: False
Input Interface:
artifact (str): The program/artifact to evaluate.
Output Interface:
scores_per_test (Dict[str, Dict[str, Any]]): A dictionary of scores per test input.
Citation:
@Article{FunSearch2023,
author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein},
journal = {Nature},
title = {Mathematical discoveries from program search with large language models},
year = {2023},
doi = {10.1038/s41586-023-06924-6}
}
load_functions
def load_functions()
Load the functions from the evaluator py file with ast parsing
