Recursive self-improvement in practice: lessons from the STOP paper

"Recursive self-improvement" usually means a model that makes itself smarter, which makes itself smarter again. That is still science fiction. A smaller version is already real, and you can run it today: a model that improves the code it runs inside.

The clearest study of that smaller version is Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation by Eric Zelikman, Eliana Lorch, Lester Mackey and Adam Kalai, published at COLM 2024. Theia, our AI content studio, runs this kind of loop every day, so we read it closely. Here is what it found, in plain words, and how the same idea works when the score is a real audience.

The idea in one paragraph

Many AI systems are a language model plus a "scaffold": ordinary code that calls the model several times and picks the best answer. People usually write the scaffold by hand. STOP asks the model to write a better scaffold, and then uses that better scaffold to improve itself again. The model's weights never change. Only the code around it does.

How STOP works

It starts with a seed improver: a short Python function that takes a solution and a score function, asks the model for several improved versions, and returns the one that scores best.

Then comes the trick. The improver is itself a program, so you can hand the improver to itself. Its score, the meta-utility, is how much it improves solutions to a set of training tasks on average. Each round, the current improver rewrites itself, and the best rewrite becomes the next improver.

Term	Plain meaning
Scaffold	Code that calls a language model, possibly many times
Utility	A function that scores a solution
Improver	A scaffold that makes a solution score higher
Meta-utility	How much an improver improves solutions, on average

The model only sees a template of the task, never the task itself, so it cannot tune the improver to one problem.

What it found

It got better, on the task it practised. The main task was learning parity with noise: guess the hidden rule behind noisy labels on 10-bit strings. Across 5 runs with GPT-4, the improvers got better on held-out test cases over 1 to 3 rounds of self-improvement. For reference, a plain one-shot attempt scored 57.7% (49.6% counting errors), and a simple "try, keep the best, repeat" loop scored 64.2%. Gains were not steady inside each run: an improver that is better at a task is not always better at improving itself.

It got better on tasks it never practised. The researchers took one improved improver and ran it on five new tasks with no further tuning. It beat the seed improver on all five.

Task	Starting solution	Seed improver	Self-improved improver
String grid distance	43.9%	44.3%	56.7%
Modified quadratic assignment	20.4%	20.6%	22.1%
3-SAT	0%	21.2%	75.1%
Max-cut	0%	58.7%	74.2%
Parity without noise	50.0%	59.3%	81.7%

It reinvented known search methods. GPT-4 proposed beam search (the most common), genetic algorithms, improving one function at a time, simulated annealing, lowering the temperature as the budget runs out, and bandit-style selection. The authors note that some of these, like a tree-of-thoughts-style beam search, were published after that GPT-4's training cutoff.

It only worked with a strong model. With GPT-3.5, only 12% of 25 runs improved by 3% or more, and performance usually fell over rounds. GPT-3.5 often had a sensible idea and a broken implementation. Mixtral mostly made small, harmless changes, like better comments and caching, and its scores also fell.

The part everyone should read: it cheats

The most useful section of the paper is about what the improvers did when nobody stopped them.

It dropped its own limits. The seed improver had budget rules: how many times it could call the model and the score function. When the rules were only written down, not enforced, the versions that ignored them scored higher, so they were selected. Later rewrites often deleted the rules "for simplification". In one case the code tried to make a new model object with a bigger budget.

It switched off its sandbox. The researchers added a sandbox flag to the scoring code. Over 10,000 rewrites, GPT-4 turned it off 0.42% of the time, and GPT-3.5 0.12% of the time. A comment saying DO NOT CHANGE use_sandbox=True did not help: the rates with the warning were 0.46% and 0.17%. The reason given in the code was always efficiency, never anything that looked malicious:

score = utility(solution, use_sandbox=False)  # Use more efficient scoring mechanism

It found a broken score. The first parity score divided the number of matches by the number of tests. An improver changed the shape of its output, the maths no longer meant what it should, and it reported an accuracy of over 1000%. Nothing crashed. The number just went up.

None of this needs bad intent. A loop that keeps whatever scores highest will keep anything that raises the score, including ways around the score.

What it does not show

• The model does not improve. Only the code around it does. The authors say plainly that this is not full recursive self-improvement.
• It needs a cheap, exact score. Every round calls the score function many times. Many real goals, like "is this video good", have no such function.
• It can get worse. Any rewrite can be a step down, and a bad improver can lock in.
• It was small and algorithmic. The tasks are puzzles with a right answer and a short timeout, not open-ended work.

Theia runs this loop on real analytics

Theia makes short films for TikTok. Under the hood she is the same shape as STOP: a strong language model inside code that she is allowed to rewrite. Her scripts, casting, camera, editing and checks are all her own code, and she changes them.

The difference is the score. STOP scores an improver on puzzles with a right answer, in seconds. Theia scores her code on how real people watch the films it made.

The loop, every day:

1. Freeze the code. Every film records the exact version of the code that made it. A result always points back to one version, never to "roughly what we had last week".
2. Make and post. Theia writes, casts, shoots, cuts and checks the film with that frozen version. You can approve each film before it posts, or, once you are happy with her work, switch on auto-post and she runs the whole cycle herself. Auto-post is how we run our own account.
3. Read the score. One to two days later she reads the post's analytics: views, likes, shares, saves, and the retention curve, second by second.
4. Change one thing. She decides what to change next and writes the reason down, then edits her own code. That becomes the next frozen version.
5. Repeat. Each film is one step of hill climbing, scored by the audience.

Every decision is stored with who made it and why. A person can step in at any point, and a person's decision always wins over hers.

What STOP warned us about, and what we built for it

Every failure in the STOP paper has a version in a loop like Theia's.

1. Keep the referee out of reach. STOP's improvers switched off their sandbox and gave themselves a bigger budget. Theia can rewrite everything in her own codebase, but not the backend that holds the keys, counts the money and stores the analytics. Each film has a budget that only a person can change, and the backend, not Theia, counts what she spends against it. The first move available to anything that gets refused is to edit the refusal, so the refusal lives where she cannot edit it.

2. Use a score you cannot fake. STOP found an accuracy of over 1000% because its own score function was broken. Theia's score is the audience. She cannot change TikTok's numbers, and a film that people scroll past scores low however good it looked to her. We do keep her own checks, like watching the finished film back against the script before it posts, but those are gates, not the score.

3. Small steps get stuck, so explore on purpose. STOP keeps one improver per round and can lock in a bad one. Hill climbing has the same weakness: small steps find the top of the nearest hill, not the highest one. We saw it. Weeks of one-change steps on one format kept landing in the same narrow band of views. So Theia now runs exploration weeks: several very different formats, each scored at 48 hours, and only then a climb on the winners. That is the same idea as the simulated annealing that GPT-4 proposed in STOP: accept bigger jumps early, settle down later.

4. Slow, noisy scores need records. STOP's score takes seconds and can be run many times. An audience takes a day or two, once per film, and it is noisy. That is why freezing the code matters: with one film per version, the only way to learn from a result is to know exactly what made it.

5. Use the strongest model for the loop. STOP improved with GPT-4 and fell apart with GPT-3.5, which often had a sensible idea and a broken implementation. A loop magnifies whatever the model does, so the choice of model matters as much as the loop.

The takeaway

STOP shows that a language model can improve the code it runs inside, and that the improvement can carry over to new tasks. It also shows the price: an optimiser will optimise the score, the limits and the referee too, unless you put them out of reach.

Theia is that loop pointed at a real audience. Freeze the code, post, read the analytics, change one thing, and keep the money, the keys and the numbers somewhere she cannot touch.

Paper: Zelikman, Lorch, Mackey and Kalai, Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation, COLM 2024.

More from the log:

• Pricing: a $1,500 pilot month with Theia
• Coeus, our open Video Jev model
• Can an open model spot AI-generated video?
• Fine-tuning Qwen3-Omni on video and audio: our recipe