ChatGPT evolved from InstructGPT.

Overview
Its training can be divided into four stages:
- Mask-based self-supervised training
- Supervised QA training on human-labeled data
- Supervised training on human rankings of answer preference
- Reinforcement learning from the learned human preferences to train a discriminative model
Self-supervised training

Self-supervised training is used for text generation. During training, the model is given the preceding words and asked to predict the next one. The prediction is a probability distribution over possible words, and beam search can sample the next word from it. That newly generated word is then fed back into the model to produce the following output. Because the output is obtained by random sampling, the result is different each time.
Self-supervised training alone can already carry out the question-answering task in ChatGPT. For example, given “Where is the highest mountain in the world?”, the question is fed into the model for text generation, and the generated text is treated as the answer. This kind of model still has major shortcomings, though:
- Because each output is obtained by random sampling, the result is not stable from run to run.
- Because of the complexity of the training text, reliable answers are often hard to obtain.
To address these problems, ChatGPT uses human-labeled QA data for supervised training.
Supervised QA training
To make the network more accurate at question answering, a set of {“question”, “answer”} pairs is collected first. These pairs are fed into the model for supervised training so that its answers become more reliable.
If the labeled data were large enough, the model’s performance would be good enough. Human-labeled QA data, however, is very hard to collect and expensive. ChatGPT’s training therefore uses only a small amount of labeled data, and then improves performance further through the later steps.
Supervised learning of human preferences
Compared with labeling QA text pairs by hand, it is faster and easier for people to rank the answers ChatGPT generates. In practice, ChatGPT can already produce correct answers; it simply does not know which one is better. OpenAI hired a large number of people to score and rank the answers produced by ChatGPT, then used the collected data to train an evaluation network.
During training of the evaluation network, a question and a ChatGPT-generated answer are fed in together, and the network outputs a score.
A higher score means the answer better matches human preferences.
The loss used to train the evaluation model can be written as:
Here $r_{\theta}$ is the evaluation network being trained, $x$ is the question fed into ChatGPT, and $y_w$ and $y_l$ are two different answers produced by ChatGPT, with $y_w$ the better one. The loss therefore trains the network to better tell answers of different quality apart. The binomial coefficient in front of the loss corresponds to the number of answers ChatGPT produces for the same question. Ranking those answers yields many pairs, which greatly improves how efficiently the data is used.
Reinforcement learning
The objective of reinforcement learning is
$$ \begin{aligned} \operatorname{objective}(\phi)= & E_{(x, y) \sim D_{\pi_{\phi}^{\mathrm{RL}}}}\left[r_{\theta}(x, y)-\beta \log \left(\pi_{\phi}^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]+ \\ & \gamma E_{x \sim D_{\text {prerrain }}}\left[\log \left(\pi_{\phi}^{\mathrm{RL}}(x)\right)\right] \end{aligned} $$The reinforcement-learning objective has three terms. The first is the evaluation model $r_{\theta}(x, y)$. The goal of reinforcement learning is to adjust ChatGPT’s parameters so that the evaluation model assigns the highest possible score to the answers ChatGPT produces.
The second term is effectively a KL divergence. $\pi^\mathrm{SFT}$ is the ChatGPT model trained on human-labeled QA data, and $\pi_{\phi}^{\mathrm{RL}}$ is the network that reinforcement learning is optimizing. This term keeps the network updated by reinforcement learning as close as possible to the network learned from labeled QA data; large changes are undesirable. It is the main innovation of the PPO-ptx reinforcement-learning model.
The third term is the loss from ChatGPT pretraining. Fine-tuning should not improve question answering at the cost of degrading the generation performance learned during pretraining, so the pretraining loss is included to keep performance from dropping on other tasks.
Summary
- Put simply, ChatGPT is GPT-3 fine-tuned further on labeled data.
- ChatGPT produces strong results on many questions, but it is still fundamentally a data-driven model. Ask it questions that are not useful, and it can still go wrong.
