A PBE task proposer model. Model that generates a random sequence of functions that will transform given inputs to outputs.
We start with the pre-trained OPT 125M model, then finetune on a random list of functions.
This can be later be optimized using RL.
The SFT training ensures that the model starts with a good baseline for future RL training.
The model is trained to output a valid list of functions from the common_functions.py file.
We measure accuracy, ratio of valid tasks generated.
When evaluated using this gives the result of
1python wandering_light/evals/evaluate_proposer.py --model abhishekraok/proposer-basicfns-opt125m-sft2k --solver-model abhishekraok/induction-basicfns-opt125m-longsft
2EvalResult(parse_rate=0.96, avg_function_count=2.02, avg_function_count_ratio=1.38, solver_success_rate=0.15, num_samples=100, frac_non_zero_std=0.31)
