Back to Reinforcement Learning & Simulation

Reinforcement Learning & Simulation

How Reinforcement Learning Improves Large Language Models

Kimi reinforcement learning researcher Whisper discusses post-training, verifiable rewards, inference-time scaling and the methods behind Kimi k1.5.

Zha Dami2025-01-2212 min readWeChat original
Contents

Starting on Monday evening in the week of this January 2025 article, DeepSeek R1 and Kimi k1.5, two Chinese models targeting OpenAI o1-level performance, launched in succession and dominated social media. With internet text data close to exhaustion and language-model pretraining approaching a bottleneck, reinforcement learning for post-training has become a widely accepted way to improve performance.

We recently published an introduction to reinforcement learning in robotics, A Deep Dive into Robot Reinforcement Learning. For this article, I am delighted to welcome Kimi RL expert Whisper to explain Kimi k1.5: Scaling Reinforcement Learning with LLMs. Whisper completed an undergraduate degree at Shanghai Jiao Tong University, followed by master's and doctoral studies at EPFL's LASA robotics laboratory in Switzerland, focusing on robot control and reinforcement learning.

Original article outline

Contents

👇

Part one: reinforcement learning in post-training

1. Has pretraining reached its limit—is RL post-training now king?

2. Which types of data are running out?

3. Earlier post-training methods

4. How does current RL differ from RLHF?

5. Which kinds of RL are used?

6. Can RL improve open-ended text responses?

7. Training efficiency and GPU requirements

Part two: Kimi k1.5 and scaling RL with LLMs

8. What is Kimi's k-series?

9. Inference-time scaling

10. Understanding long-to-short reasoning

11. How does Long2Short work?

12. Does RL improve reasoning effectively?

13. Long-context scaling

14. Some interesting observations

Part I: reinforcement learning in post-training

1. Has scaling pretraining data reached its limit? Is RL post-training the next direction?

That may be an industry trend, with data limitations at its core. Pretraining has been scaled for several years, so returns gradually diminish, but there is still much to do: improve quality and find more diverse data, for example. I see that work as never-ending. RL scaling, meanwhile, is only beginning, and we hope to see a steeper performance curve there. Through interaction and policy optimization, RL can further unlock an existing model's potential and compensate for slower data growth.

2. Which data is reaching its limit: text or images?

For a language-only model, text volume and quality are central. Multimodal models also require images, audio or video for learning and reasoning across modalities. Image and video data still have room to grow, although they require more complex processing. Ideally, a multimodal model would both understand data deeply and generate high-quality content, training generation and understanding within one framework.

Current understanding capabilities are reasonably good. Our Kimi k1 and k1.5, for example, understand images relatively well. We included many mathematical and logical reasoning problems during training, and the models generalize fairly well on those tasks.

We tried asking a model to infer where a photograph was taken. Its estimates were quite reliable. But images do not contribute much to the model's core intelligence: reasoning still depends primarily on training with text. Images may contribute more through grounding, helping it understand spatial relationships and specific situations.

3. What post-training methods were common before reinforcement learning?

A year earlier, people might begin post-training with supervised fine-tuning, or SFT. After seeing some improvement, they would optimize further with direct preference optimization, or DPO, and that would largely complete the stage.

In the original large-model training recipe, we first clean a large dataset for pretraining. The initial model cannot hold a conversation. SFT teaches it conversation, instruction following and related capabilities.

The model is then broadly usable. Earlier pipelines might also add reinforcement learning from human feedback, or RLHF, but that differs from the RL used today. DPO can be viewed as a simplified version of RLHF. SFT helps, but too much can harm model performance.

DPO stands for Direct Preference Optimization. It replaces the complex reinforcement learning step in conventional RLHF with direct optimization on preference data. Its central idea is to use human preferences without a complex RL algorithm. It is a lightweight preference-alignment approach suited to improving outputs when substantial human preference data is available, without running the full reinforcement learning pipeline.

4. How does current post-training RL differ from earlier RLHF?

The main distinction is the source of the reward. RLHF obtains rewards from a reward model trained on prepared data. Training might use paired responses: one people consider good and another they consider bad.

The reward model learns to score responses from these pairs. Asked about today's weather, for example, an unrelated greeting is a poor response, while accurate weather information is a good one. The model learns to score different answers accordingly.

That reward model is then used to train the policy network. Training cannot continue for too many steps, though, because the policy may overfit the reward model. Once that happens, the reward model no longer provides effective feedback and needs to be retrained.

Current post-training RL relies more on ground truth, such as whether the model completed a task correctly, wrote correct code or answered a question accurately. This is closer to a gold-standard reward and is more reliable. Because the policy has a harder time exploiting or deceiving that standard, reinforcement learning can continue for longer.

RLHF fundamentally depends on human judgments about which answers are good or bad. Much current post-training RL focuses on questions with definite answers determined by objective criteria rather than subjective opinions.

The methods therefore address different kinds of problems. RLHF is more appropriate for subjective tasks such as dialogue, where evaluation varies between people. RL with objective rewards primarily targets clear-answer tasks such as mathematics and coding. To sum up, for mathematics and programming problems, RL post-training is currently considered the optimal approach. This assessment reflects the interviewee’s view in January 2025.

5. Which kinds of RL are used in post-training?

Reinforcement learning has many categories, including value-based and policy-based approaches. Current mainstream methods are mostly policy-based.

Over time, researchers have moved away from value-based methods in this setting. Policy-based RL is easier to scale and has advantages for large-scale training and deployment, helping tackle harder problems.

Policy-based methods are widely used, including the previously common Proximal Policy Optimization, or PPO. PPO was considered very effective early on, but it may no longer be necessary for LLMs, or its benefits may be limited. Current approaches often use simpler policy-based training algorithms rather than a complex PPO framework.

Another distinction is model-free versus model-based, where model means a transition model. RL used for large language models is currently model-free; model-based methods are more common in robotics.

In on-policy methods, training data is sampled from the current policy, and fresh data is collected as the policy changes. Off-policy methods can continue using previously sampled data after the model changes. On-policy emphasizes freshness and consistency, while off-policy can reuse historical data more efficiently. Our algorithm has a degree of off-policy behavior. Technical clarification: an on-policy algorithm can make multiple optimization updates within a rollout batch; the distinction does not mean that every individual gradient step must be followed by new sampling.

Before we released this paper, there was no industry consensus on RL for post-training, and OpenAI o1's RL training method had not been disclosed. As more papers appear, these methods will attract increasing attention.

6. Is there also a trend toward RL post-training for text responses?

That mainly depends on whether the outcome can be evaluated accurately. How do you reliably decide which of two essays is better? It is difficult because there is no clear, universal standard that completely objectively measures quality. Evaluation often involves subjectivity.

Text responses therefore usually still need RLHF. Another possibility is to develop a sophisticated evaluation mechanism or pipeline that provides more accurate judgments. If that is available, direct optimization becomes possible as well.

7. How does RL affect training efficiency and GPU requirements?

RL has not significantly reduced training cost; it remains expensive. It relies on extensive search, requiring substantial compute, with generation and training both taking place.

For a task such as solving mathematics, the model generates an answer and receives a reward according to correctness. We may compare it with ground truth, or execute generated code against test cases. This answer-generation and evaluation process forms a rollout.

After generating answers, further training is required. Repeated generation, verification and training gradually improve performance.

Part II: Kimi k1.5 and scaling RL with LLMs

8. What is Kimi's k-series?

The k-series emphasizes long responses for complex problems, including mathematics competitions and Codeforces programming contests. The first version, k0, focused on mathematics. k1 extended to multimodality, and k1.5 received more training on top of k1 and improved performance.

The series gradually adds visual capabilities, particularly for 2D images. Pretraining naturally contains many internet images, while images used in RL post-training are more targeted. Geometry problems require combining text and diagrams; aptitude tests and intelligence tests also contain visual reasoning. Such data specifically trains mathematical, geometric and other image-based reasoning.

Overall, the research strengthens complex problem-solving and visual reasoning while extending applications through multimodal data. Coding has relatively little need for multimodality, but mathematics and school-level chemistry and physics problems often need it more.

Interestingly, training on problem-solving images also improves spatial understanding. Given a photograph taken from an office window, for example, the model can reason about its contents and infer the location. Combining visual recognition and spatial reasoning improves its usefulness in real situations.

9. What is inference-time scaling?

Inference-time scaling means that spending more time reasoning can substantially improve performance. Generating output is itself computation. Given longer to think, a model can try alternatives and correct itself. It might calculate an initial answer, notice an error, reflect and revise before reaching a more accurate result. This can work much better than answering immediately.

Longer reasoning makes responses feel slower to users but can improve answer quality. The tradeoff depends on the situation. For a simple greeting, a slow response worsens the experience.

Ideally, a model would support both long and short responses and decide when to reason at length or respond quickly. Current technology has not fully reached that point. Performance depends on training data: well-covered situations are easier to optimize, while others remain less mature.

One solution is to provide separate options so users can choose a fast response or higher-quality reasoning. This gives flexibility and balances model capabilities against practical needs.

10. Understanding long CoT, short CoT and Long2Short

Long chain of thought, or CoT, uses a lengthy reasoning output—perhaps ten thousand tokens for one problem. The model may not actually need that much. It might still answer correctly with a shorter trace, such as two thousand tokens.

Long2Short asks whether we can compress output as far as possible while retaining correct problem-solving. The goal is to remove unnecessary reasoning redundancy without sacrificing accuracy, making the model more efficient and avoiding pointless length.

We seek a balance between reasoning accuracy and efficiency: maintain strong performance while reducing wasted computation.

During RL training, reasoning traces often grow longer and longer. The reward depends on whether the answer is correct rather than on reasoning length, so the model gradually develops longer reasoning patterns that reinforce themselves.

Although lengthy reasoning may look redundant, deeper reflection and repeated attempts often significantly improve final accuracy.

The process resembles a student solving mathematics: rather than giving an immediate answer, the model works step by step, sometimes returning to earlier steps to reflect, check and find a better approach. This repeated reasoning can improve correctness.

Excessive length, however, can introduce unnecessary complexity. We want to limit reasoning rather than let it grow indefinitely. Performance often improves as reasoning becomes longer, but once the problem's difficulty is adequately addressed, further length no longer helps. The goal is the best performance at an appropriate length, rather than length for its own sake.

11. How does Long2Short work?

There are several ways to move from long to short reasoning.

First is model merging: combine a long-reasoning model with a short-reasoning model to obtain an intermediate output length.

Second is a length penalty in the RL reward, encouraging a balance between accuracy and length. This is an elegant way to optimize the tradeoff.

Third is shortest-response selection. Among correct answers generated by a long-reasoning model, select the shortest reasoning paths as training data for another model. Like knowledge distillation, this turns the long model's capabilities into more efficient reasoning, encouraging the shortest path that still solves the problem correctly.

Fourth is direct preference optimization, or DPO, which the transcript describes as using reinforcement learning to directly favor shorter reasoning. It is a simple, fast way to pursue the optimization goal. Technical clarification: DPO actually trains directly on preference pairs without an explicit RL rollout-and-update loop. The Kimi report distinguishes DPO from its separate Long2Short RL method.

12. Is RL for reasoning more effective than simply increasing model size?

An interesting finding is that a small model can approach a larger model's performance by generating longer outputs.

Allowing more reasoning can partly compensate for the capability gap caused by a smaller model. In effect, longer inference time and a longer reasoning process help it achieve results closer to those of the larger model.

13. Understanding long-context scaling

We extended the RL context window to 128k and found that performance improved with greater context length. To improve efficiency, we use partial rollouts, reusing large portions of previous trajectories to generate new ones rather than paying to generate everything from scratch.

Extending to 128k reflects a broader trend. As reasoning length increases, performance continues to improve: the model can handle harder problems and give more accurate answers. This shows the potential of longer reasoning for large-scale inference.

During training, capabilities improved as the window expanded beyond 32k. But autoregressive generation creates a clear problem: longer outputs are slower to produce, significantly reducing training efficiency.

An optimization strategy is to divide long generation into sections. The transcript gives a 128k response as an example: one model generates the first 32k, another a middle 32k section, and a third the last 32k. Segmenting generation can address the slowdown associated with long reasoning while preserving its completeness and efficiency, and suggests a direction for further expansion. Technical clarification: those three 32k portions do not total 128k. The report describes saving unfinished trajectories and continuing them across training iterations, while other workers handle shorter trajectories. It does not describe independent models generating arbitrary sections in parallel; later autoregressive sections still depend on the earlier context.

The method can also detect repetition early during generation. Repetition is a common characteristic of large language models. We can stop repeated content and penalize it, so even lengthy answers contain little repetition.

Policy optimization improvements primarily concern the algorithm, alongside a simple framework that is practical to implement. In fact, components such as MCTS, value networks and process rewards can all be considered unnecessary: we can do without them entirely. Overall, the simplest approach is often the most effective. That is the opposite of the conclusions emphasized by many earlier papers exploring MCTS and process reward models. This is the interviewee’s assessment of the Kimi k1.5 approach, rather than a claim that these techniques have no value in other settings.

14. Some interesting observations

As generated text becomes longer, interesting behaviors emerge. The model may say things like wait, let me reconsider, or I now feel fairly confident, before giving its answer. Once I noticed it produce a phrase sounding like ou-rui, followed by the correct answer. I initially wondered whether it referred to something Euclidean, European or Swiss. After thinking about it, I suspected it was a phonetic rendering of all right—an additional transliteration phenomenon. Models sometimes produce unexpectedly amusing expressions.

References

https://github.com/MoonshotAI/Kimi-k1.5

Smart Note · Reinforcement Learning & Simulation