Teaching small LLM’s unreasonable problems?

by Cascha van Wanrooij and Evertjan Peer

In recent weeks, Deepseek R1 emerged as a milestone in AI development — an open-source reasoning model from Chinese startup Deepseek that promises to match the performance of OpenAI's o1 model. While OpenAI kept their methods secret, Deepseek took a different approach by open sourcing their models and methods. These revealed: reinforcement learning (RL), in its relative simplicity, was the key to enable reasoning.

Deepseek implemented a straightforward reward model that incentivized correct answers and encouraged the model to document its thinking in <think> tags. Through this process, the model developed reasoning capabilities for STEM problems, particularly in mathematics. It learned to self-correct and validate its assumptions, all while being trained with significantly less computational power than its American counterparts.

Recreating reasoning in smaller models

This kicked off a flood of attempts to recreate this reasoning in smaller models on the cheap (simpleRL, TinyZero, s1). The underlying promise was appealing: if a problem solution can be quickly and automatically verified, a model should be able to learn that problem through RL. And the early results seemed promising — some claimed o1-preview-level math skills for $6 in compute, while others trained a 3 billion parameter model on math problems for $30. These budget-friendly successes made us wonder: could we validate this approach ourselves with a different type of problem?

An unreasonable problem: Dutch train station anagrams

Recently, NS trains added a frustrating mini-game to their onboard screens: anagrams of station names. The game is deceptively difficult, leaving you with that nagging feeling of inadequacy when you can't crack it. Turning to your favorite chatbot (without reasoning enabled) won't help much — even the strong models only get lucky hits here and there with longer anagrams.

But reasoning models? We fed these puzzles to Deepseek R1 and found it solved almost all of them. The process wasn't exactly efficient — the model's chain of thought mixed sensible strategies like letter frequency counting with brute force guesses at common stations. Still, it managed to verify its guesses and usually found the right answer. Keep in mind that R1 never trained specifically on anagrams — it learned mathematical reasoning. This sparked our idea: could we teach much smaller models to crack anagrams using a targeted reinforcement learning approach?

While for a lot of humans solving an anagram is a matter of "you either see it or you don’t," there is a certain reasoning strategy to be derived from the R1 thinking streams. In the most general form, we argue the reasoning consists of the following steps: (1) make an educated guess, (2) verify that educated guess (3), end if correct, or else repeat from (1). So in order for us to apply the reasoning tuning approach we’d need a model that can make educated guesses (possibly by counting characters), and teach it to verify those guesses until it has found the right solution.

Building the RL training pipeline

First, we needed a base model with decent knowledge of Dutch station names — you can't solve an anagram without knowing the answer exists. Llama 3.1 8B fit our needs, showing solid recall of most Dutch train stations during initial prompting tests. We weren't too worried about occasional hallucinated stations since the anagram constraint naturally limits answers to words using exactly the given letters.

For the reinforcement learning piece, we got lucky. Right after the Deepseek paper, a library called TinyZero was released, implementing PPO (Proximal Policy Optimization) for language models that we could use directly. While PPO differs from Deepseek's GPRO approach, it had already proven to be effective at teaching 3B-parameter models to solve basic math problems.

With the framework in place, we built our training setup. The data came from Wikipedia — a list of all 399 Dutch train station names, with 20% held back for validation. We created our training examples by randomly scrambling these names and wrapping them in prompts that asked the model to unscramble them, show its reasoning in <think> tags, and give its answer in <answer> tags. 

The reward structure couldn't be simpler: 

1 point for correct answers,
0 for everything else. 

We chose this simple structure because we wanted to see if reinforcement learning indeed just worked for this new class of problems.

Then, we fired up a rented instance with four of NVIDIA’s newest H200 chips, each equipped with a hefty 141GB of VRAM. Our excitement grew as the average reward climbed steadily — from under 5% solve rate to 20%, then 30%, eventually passing 50% on the training set. But something wasn't right...

The model wasn’t really learning anagrams

While validation performance jumped to 18% early on, it plateaued there despite continued gains on the training set. Our model had become a clever cheater — when faced with longer anagrams (anything over five letters), it relied on simple heuristics to make educated guesses. The validation metrics told the story: instead of finding valid letter permutations, it was spitting out valid station names that often had nothing to do with the scrambled letters.

We also noticed the responses getting shorter over time. Some initial shortening made sense as the model dropped ineffective strategies. But there was a clear pattern: mid-training, the model would guess and verify its answers, while by the end, it just jumped straight to guessing. This suggested the model found its own reasoning more harmful than helpful. Since our reward system didn't consider response length, this makes sense — if a reasoning step gives even slightly worse results, the model learns to skip it and race to the <answer> tag. The reverse would be true if reasoning helped — we'd see longer responses as the model took time to think things through.

Assessing the true difficulty of the task

So, we took a step back to do what we should have done at the start: assess the true difficulty of our task. Just because a 671 billion parameter model could crack these anagrams didn't automatically mean our 8 billion parameter model could follow suit — even if it could recall station names when being prompted to do so. We needed to understand what was holding our smaller models back.

We ran tests across several models as presented in the table below. 

We observe a steep decline in validation performance with smaller model sizes. Even within the seemingly decent 41% success rate for 70B hid a problem: performance declined linearly as word length increased.

Testing our theory with common words

To better understand this limitation, we shifted our experiments to words that should be deeply embedded in our models' knowledge. Using Llama 8B, we assembled a diverse dataset of 4,000 words spanning everyday categories like animals and fruits, cultural entries like top albums and artists, and structured lists like Fortune 500 companies and Nobel laureates.

Our hypothesis: if the issue was simply the model's unfamiliarity with Dutch train stations, then it should perform better on these widely-known terms.

The results were revealing — and confirmed our concerns. Even with this more familiar dataset, we saw quickly stagnating training performance and similarly unimpressive validation results. Without the ability to start memorizing solutions, we observed that no strong reasoning capability developed to solve the unscrambling task, even for common words the model definitely knew.

While performance improved on shorter words during early training epochs, longer words (>12 characters) remained practically unsolvable. Even more telling was our experiment with chemical elements — despite being a small set of just 118 highly common terms, the model failed to develop effective reasoning strategies for unscrambling them.

The "Capability Unlocking Principle"

These experiments pointed to a fundamental insight: anagram reasoning and verification capabilities are simply not present within smaller base models, making them difficult to unlock through minimal RL fine-tuning alone.

This observation aligns with other research in the field. The paper “Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs” suggests that successful RL applications on large language models work by activating latent reasoning skills already present in the model. It's similar to adding "unreal engine" to image generator prompts — it improves quality by accessing existing patterns rather than creating new ones.

For practitioners, this points to what we call the "Capability Unlocking Principle": RL works best when building upon what models already know. This might explain why the "$6 DeepSeek R1" experiments succeeded with math problems, as base models are heavily pre-trained to reason on mathematical content. In addition, data contamination may be more likely in mathematical problems.

Conclusion: setting realistic expectations

If you're planning to use RL with smaller models, set realistic expectations: an 8B model won't suddenly master domain-specific tasks just because you add a reward signal. Before investing in RL training, test your base model extensively on the skills needed for your target task — if it shows no signs of the capability you're trying to enhance, RL likely won't help.

Consider the NS train anagrams: even with perfect verification of answers, our 8B model couldn't develop reliable unscrambling abilities because the fundamental pattern recognition wasn't there to begin with.

The big learning? RL excels at probability adjustment. It can help a model better utilize its skills, but it can't teach those skills from scratch. The sparse reward signal in small-scale experiments isn't enough to bootstrap new capabilities.