Deepseek has not too long ago made fairly a buzz within the AI group, because of its spectacular efficiency at comparatively low prices. I believe this can be a excellent alternative to dive deeper into how Massive Language Fashions (LLMs) are educated. On this article, we are going to give attention to the Reinforcement Studying (RL) aspect of issues: we are going to cowl TRPO, PPO, and, extra not too long ago, GRPO (don’t fear, I’ll clarify all these phrases quickly!)
I’ve aimed to maintain this text comparatively straightforward to learn and accessible, by minimizing the maths, so that you received’t want a deep Reinforcement Studying background to comply with alongside. Nonetheless, I’ll assume that you’ve some familiarity with Machine Studying, Deep Studying, and a fundamental understanding of how LLMs work.
I hope you benefit from the article!
The three steps of LLM coaching
Earlier than diving into RL specifics, let’s briefly recap the three foremost phases of coaching a Massive Language Mannequin:
- Pre-training: the mannequin is educated on a large dataset to foretell the subsequent token in a sequence based mostly on previous tokens.
- Supervised Superb-Tuning (SFT): the mannequin is then fine-tuned on extra focused knowledge and aligned with particular directions.
- Reinforcement Studying (usually referred to as RLHF for Reinforcement Studying with Human Suggestions): that is the main target of this text. The principle aim is to additional refine responses’ alignments with human preferences, by permitting the mannequin to study immediately from suggestions.
Reinforcement Studying Fundamentals
Earlier than diving deeper, let’s briefly revisit the core concepts behind Reinforcement Studying.
RL is sort of simple to know at a excessive degree: an agent interacts with an setting. The agent resides in a selected state throughout the setting and might take actions to transition to different states. Every motion yields a reward from the setting: that is how the setting offers suggestions that guides the agent’s future actions.
Take into account the next instance: a robotic (the agent) navigates (and tries to exit) a maze (the setting).
- The state is the present scenario of the setting (the robotic’s place within the maze).
- The robotic can take totally different actions: for instance, it could possibly transfer ahead, flip left, or flip proper.
- Efficiently navigating in the direction of the exit yields a optimistic reward, whereas hitting a wall or getting caught within the maze ends in unfavourable rewards.
Simple! Now, let’s now make an analogy to how RL is used within the context of LLMs.
RL within the context of LLMs
When used throughout LLM coaching, RL is outlined by the next elements:
- The LLM itself is the agent
- Setting: every little thing exterior to the LLM, together with person prompts, suggestions programs, and different contextual info. That is mainly the framework the LLM is interacting with throughout coaching.
- Actions: these are responses to a question from the mannequin. Extra particularly: these are the tokens that the LLM decides to generate in response to a question.
- State: the present question being answered together with tokens the LLM has generated to this point (i.e., the partial responses).
- Rewards: this is a little more difficult right here: not like the maze instance above, there may be often no binary reward. Within the context of LLMs, rewards often come from a separate reward mannequin, which outputs a rating for every (question, response) pair. This mannequin is educated from human-annotated knowledge (therefore “RLHF”) the place annotators rank totally different responses. The aim is for higher-quality responses to obtain increased rewards.
Notice: in some circumstances, rewards can truly get less complicated. For instance, in DeepSeekMath, rule-based approaches can be utilized as a result of math responses are typically extra deterministic (right or mistaken reply)
Coverage is the ultimate idea we’d like for now. In RL phrases, a coverage is just the technique for deciding which motion to take. Within the case of an LLM, the coverage outputs a likelihood distribution over potential tokens at every step: briefly, that is what the mannequin makes use of to pattern the subsequent token to generate. Concretely, the coverage is set by the mannequin’s parameters (weights). Throughout RL coaching, we alter these parameters so the LLM turns into extra more likely to produce “higher” tokens— that’s, tokens that produce increased reward scores.
We regularly write the coverage as:
the place a is the motion (a token to generate), s the state (the question and tokens generated to this point), and θ (mannequin’s parameters).
This concept of discovering the most effective coverage is the entire level of RL! Since we don’t have labeled knowledge (like we do in supervised studying) we use rewards to regulate our coverage to take higher actions. (In LLM phrases: we alter the parameters of our LLM to generate higher tokens.)
TRPO (Belief Area Coverage Optimization)
An analogy with supervised studying
Let’s take a fast step again to how supervised studying usually works. you’ve labeled knowledge and use a loss operate (like cross-entropy) to measure how shut your mannequin’s predictions are to the true labels.
We are able to then use algorithms like backpropagation and gradient descent to attenuate our loss operate and replace the weights θ of our mannequin.
Recall that our coverage additionally outputs possibilities! In that sense, it’s analogous to the mannequin’s predictions in supervised studying… We’re tempted to write down one thing like:
the place s is the present state and a is a potential motion.
A(s, a) known as the benefit operate and measures how good is the chosen motion within the present state, in comparison with a baseline. That is very very like the notion of labels in supervised studying however derived from rewards as an alternative of express labeling. To simplify, we will write the benefit as:
In observe, the baseline is calculated utilizing a worth operate. This can be a widespread time period in RL that I’ll clarify later. What you have to know for now’s that it measures the anticipated reward we might obtain if we proceed following the present coverage from the state s.
What’s TRPO?
TRPO (Belief Area Coverage Optimization) builds on this concept of utilizing the benefit operate however provides a crucial ingredient for stability: it constrains how far the brand new coverage can deviate from the outdated coverage at every replace step (just like what we do with batch gradient descent for instance).
- It introduces a KL divergence time period (see it as a measure of similarity) between the present and the outdated coverage:
- It additionally divides the coverage by the outdated coverage. This ratio, multiplied by the benefit operate, offers us a way of how helpful every replace is relative to the outdated coverage.
Placing all of it collectively, TRPO tries to maximize a surrogate goal (which includes the benefit and the coverage ratio) topic to a KL divergence constraint.
PPO (Proximal Coverage Optimization)
Whereas TRPO was a big development, it’s not used broadly in observe, particularly for coaching LLMs, attributable to its computationally intensive gradient calculations.
As a substitute, PPO is now the popular method in most LLMs structure, together with ChatGPT, Gemini, and extra.
It’s truly fairly just like TRPO, however as an alternative of imposing a tough constraint on the KL divergence, PPO introduces a “clipped surrogate goal” that implicitly restricts coverage updates, and vastly simplifies the optimization course of.
Here’s a breakdown of the PPO goal operate we maximize to tweak our mannequin’s parameters.
GRPO (Group Relative Coverage Optimization)
How is the worth operate often obtained?
Let’s first speak extra in regards to the benefit and the worth capabilities I launched earlier.
In typical setups (like PPO), a worth mannequin is educated alongside the coverage. Its aim is to foretell the worth of every motion we take (every token generated by the mannequin), utilizing the rewards we acquire (do not forget that the worth ought to characterize the anticipated cumulative reward).
Right here is the way it works in observe. Take the question “What’s 2+2?” for example. Our mannequin outputs “2+2 is 4” and receives a reward of 0.8 for that response. We then go backward and attribute discounted rewards to every prefix:
- “2+2 is 4” will get a price of 0.8
- “2+2 is” (1 token backward) will get a price of 0.8γ
- “2+2” (2 tokens backward) will get a price of 0.8γ²
- and so forth.
the place γ is the low cost issue (0.9 for instance). We then use these prefixes and related values to coach the worth mannequin.
Necessary be aware: the worth mannequin and the reward mannequin are two various things. The reward mannequin is educated earlier than the RL course of and makes use of pairs of (question, response) and human rating. The worth mannequin is educated concurrently to the coverage, and goals at predicting the long run anticipated reward at every step of the technology course of.
What’s new in GRPO
Even when in observe, the reward mannequin is usually derived from the coverage (coaching solely the “head”), we nonetheless find yourself sustaining many fashions and dealing with a number of coaching procedures (coverage, reward, worth mannequin). GRPO streamlines this by introducing a extra environment friendly technique.
Keep in mind what I stated earlier?
In PPO, we determined to make use of our price operate because the baseline. GRPO chooses one thing else: Here’s what GRPO does: concretely, for every question, GRPO generates a bunch of responses (group of dimension G) and makes use of their rewards to calculate every response’s benefit as a z-score:
the place rᵢ is the reward of the i-th response and μ and σ are the imply and commonplace deviation of rewards in that group.
This naturally eliminates the necessity for a separate worth mannequin. This concept makes a number of sense when you concentrate on it! It aligns with the worth operate we launched earlier than and likewise measures, in a way, an “anticipated” reward we will acquire. Additionally, this new technique is nicely tailored to our downside as a result of LLMs can simply generate a number of non-deterministic outputs through the use of a low temperature (controls the randomness of tokens technology).
That is the primary concept behind GRPO: eliminating the worth mannequin.
Lastly, GRPO provides a KL divergence time period (to be actual, GRPO makes use of a easy approximation of the KL divergence to enhance the algorithm additional) immediately into its goal, evaluating the present coverage to a reference coverage (usually the post-SFT mannequin).
See the ultimate formulation beneath:
And… that’s largely it for GRPO! I hope this provides you a transparent overview of the method: it nonetheless depends on the identical foundational concepts as TRPO and PPO however introduces further enhancements to make coaching extra environment friendly, quicker, and cheaper — key elements behind DeepSeek’s success.
Conclusion
Reinforcement Studying has grow to be a cornerstone for coaching right now’s Massive Language Fashions, notably by way of PPO, and extra not too long ago GRPO. Every technique rests on the identical RL fundamentals — states, actions, rewards, and insurance policies — however provides its personal twist to steadiness stability, effectivity, and human alignment:
• TRPO launched strict coverage constraints through KL divergence
• PPO eased these constraints with a clipped goal
• GRPO took an additional step by eradicating the worth mannequin requirement and utilizing group-based reward normalization. After all, DeepSeek additionally advantages from different improvements, like high-quality knowledge and different coaching methods, however that’s for one more time!
I hope this text gave you a clearer image of how these strategies join and evolve. I consider that Reinforcement Studying will grow to be the primary focus in coaching LLMs to enhance their efficiency, surpassing pre-training and SFT in driving future improvements.
In the event you’re fascinated by diving deeper, be happy to take a look at the references beneath or discover my earlier posts.
Thanks for studying, and be happy to depart a clap and a remark!
Need to study extra about Transformers or dive into the maths behind the Curse of Dimensionality? Take a look at my earlier articles:
Transformers: How Do They Transform Your Data?
Diving into the Transformers architecture and what makes them unbeatable at language taskstowardsdatascience.com
The Math Behind “The Curse of Dimensionality”
Dive into the “Curse of Dimensionality” concept and understand the math behind all the surprising phenomena that arise…towardsdatascience.com
References:
Source link
