This repository contains one of the models fine-tuned using the SOAR (Self-improving Operators for Automated program Refinements) framework, as presented in the paper:
Large Language Models (LLMs) have become incredibly powerful, but they often hit a wall when faced with truly complex reasoning tasks that require discovering a solution from scratch. Simply throwing more computing power or using a bigger model often yields diminishing returns. But what if a model could learn from its own experience, getting smarter with every attempt?
We introduce a framework called SOAR (Self-improving Operators for Automated program Refinements) that does just that. By creating a "virtuous cycle" of evolutionary search and learning, SOAR enables AI models to bootstrap their own capabilities and solve problems previously beyond their reach. we tested SOAR on the Abstraction and Reasoning Corpus (ARC-AGI-1), a notoriously difficult benchmark designed to challenge an AI's core reasoning abilities. We show that using SOAR with only open weight LLM, we can significantly outperforming much larger closed source LLMs.
We have released a dataset containing 5 million ARC solutions. For solutions that successfully solve an original ARC task, we deduplicate entries by their code to ensure uniqueness. For solutions that correspond to new synthetic tasks generated via hindsight relabeling, we deduplicate based on their output results. This approach ensures a diverse and high-quality dataset for further research and development.
Evolutionary Search (Sample & Refine): SOAR uses an LLM to generate an initial pool of thousands of candidate programs (the "sampling" step). It then tests these programs and uses the LLM again to intelligently modify or "refine" the most promising ones based on their performance.
Learning from Hindsight: SOAR takes all the programs generated during the search phase—including both successes and failures—and uses them as training data. The key insight is that any failed program is simply a correct program for a different task. By "relabeling" these failed attempts as correct solutions for the synthetic tasks they inadvertently solve, SOAR creates a diverse dataset to learn from.
This process creates a powerful feedback loop: the fine-tuned model becomes better at sampling and refining, which leads to a more effective search in the next iteration, which in turn generates even better training data. And unlike previous approaches that rely on human-engineered domain-specific languages or human-generated solutions, SOAR learns to synthesize programs in Python solely from its own synthesis attempts, encompassing both successes and failures.
How to Use the Model
The primary use of this model is to generate a Python function that solves an ARC task. The input to the model should be a formatted prompt containing the training and test examples of the ARC task.
Here's a quick example to get started:
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
34model_id ="julien31/Soar-qwen-7b"# or any other Soar-qwen model5tokenizer = AutoTokenizer.from_pretrained(model_id)6model = AutoModelForCausalLM.from_pretrained(7 model_id,8 torch_dtype=torch.bfloat16,# Use torch.float16 for GPUs that don't support bfloat169 device_map="auto",10)1112prompt = "def solve_arc_task(input_grid, output_grid):\
13 \\\"\\\"\\\"Given an ARC-AGI task, transform the input grid to the output grid by applying a series of operations.\
14 \\\"\\\"\\\""1516messages =[17{"role":"user","content": prompt}18]1920text = tokenizer.apply_chat_template(21 messages,22 tokenize=False,23 add_generation_prompt=True24)2526model_inputs = tokenizer(text, return_tensors="pt").to(model.device)2728generated_ids = model.generate(29 model_inputs.input_ids,30 max_new_tokens=256,31 do_sample=True,32 temperature=0.7,33 top_p=0.8,34 repetition_penalty=1.1,35 eos_token_id=tokenizer.eos_token_id,36 pad_token_id=tokenizer.pad_token_id,# This is often the same as eos_token_id for Qwen models37)3839# Decode only the newly generated text40decoded_output = tokenizer.decode(generated_ids[0, model_inputs.input_ids.shape[1]:], skip_special_tokens=True)41print(decoded_output)
For a complete, end-to-end example of how to format the prompt, run inference, execute the generated code, and visualize the results, please refer to the official repository and notebook:
To run SOAR, please refer to execution instructions located in the experience folder.
For simple instructions on running sampling and refinement with SOAR, as well as exploring the dataset, please see the Jupyter notebooks provided in the notebook folder. These notebooks walk through the basic SOAR step, including how to generate candidate solutions, perform refinement, and analyze results. This hands-on guide will help you get started quickly and understand each step of the SOAR process.
