• The scaling of text data is likely reaching a ceiling as most of the easy web text (Common Crawl, Github, Arxiv .etc) are now used up.

• There will surely be new text data like digging harder from the internet, scanning library books and synthetic data. Yet it is quite challenging to increase another order of magnitude — more likely, they are just incremental within the current order.

• The next chapter of the game starts from multimodal, particularly unified video-language generative model, because only video data gives orders of magnitude increase.

• However, the bad news is, it seems that video data can not increase the reasoning capability of the model — recall that reasoning is the number one key capability that marks strong models.

• But the good news is, video data increase everything else, particularly grounding to real-world, and exhibit strong potential to become neural world models (instead of hard-coded physical engines like Zelda), which leads to the possibility of learning from simulated physical feedback.

• Scaling up reinforcement learning from X feedback seems to be the most promising direction to continue increase model’s reasoning capability, where X means human, AI, and environment feedback.

• Just like how AlphaGo Zero achieves super-human performance on Go, self-play and interacting with the environment could be a direction for super-human generative models. Making the model online and iteratively learn from the feedback (instead of a single-sound offline optimization) could potentially lead to continuously increased reasoning capability.

• The first chapter of the game of scale focus on scaling text data, which peaks at GPT-4 and concluded by Llama 3. The second chapter of this game would be unified video-language generative modeling and iterative reinforcement learning from X feedback.

1 - How good is Llama 3?

• It uses 15T training tokens, which is much larger than its peers.
• Particularly, mixing the code / arxiv data could improve reasoning.

• It uses benchmark-related continue pretraining data (e.g., Llemma/ MetaMath/ Mammoth) to improve/ decorate the benchmarks.
• Yet when the score becomes 80+, it would be quite difficult, though not impossible, to decorate MMLU — the dataset itself is a hard one.

Llama 3’s initial leaderboard climbing with few votes and high variance

2 - The limit of text scaling data

• CC is only part of the whole internet.

• We have not yet finished digging and crawling from CC.

• Relaxing the filtering and deduplication threshold.

• Use existing models to produce synthetic data.

• Scanning more books from libraries

• This factor is the largest undetermined factor about text scaling. We do not know how large the actual internet is.

• For companies like Microsoft / Google and Meta who can readily dump a much larger portion of the internet beyond CC, they can still dig more data.

• But the problem is, after deduplication and quality filtering, how much tokens could remain.

• The problem with this approach is, the final number of tokens we can produce of existing CC is upper bounded by the data processing pipeline, and may not change in terms of orders of magnitude

• New CC dumps increases linearly with time, still no change in orders of magnitude.

• Yet scaling law says exponentially increasing data result in linearly increased performance. So eventually we may result in another 5T new tokens on top of the 15T Llama 3 data, yet what we truly want is another 50T tokens.

• Raw data is large, we are not using all of them because of data quality and duplication. Baichuan tech report has a nice figure on the filtering influence the amount of final tokens:

• It is a research problem (see Shayne et. al., Muennighoff et. al. and Xue et. al.) to what extend you should keep your quality and deduplication standard. The general impression is, probably not too loose.

• Recently, Liu et. al. gives a nice summary of synthetic data, highlighting data sources of reasoning, tool-using, multimodality, multilingual, and alignment data.

• The key challenge remains: it seems that most existing data work cannot change the order of magnitude, such that they are mostly used for continue-pretraining and finetuning, but not directly on pretraining.

• The only exception is the Phi model series as they use GPT-4 produced data to train a much smaller model. The problem here is whether their approach can scale to larger models and break the GPT-4 upper bound.

• Clearly these are promising directions because library books are definitely of extremely high quality — much higher than the web — and can clearly improve professional knowledge benchmarks like MMLU. Below gives a list of largest libraries in the world

• The problem is not on the tech side — buying copyrights from these libraries may simply exhaust all the AI investments — and a large portion of them are not for sale. And again, if on average there is 70K tokens per book, then 200 million books is only about 14T tokens. It doubles the existing number, but not much larger.

3 - Scaling always wins, but what to scale next?

• A clear direction is multimodal data, particularly video data. Youtube and Tiktok is presumably orders of magnitude larger than text — yes that’s where the new order of magnitude come from. But there is an immediate challenge: does multimodal data improve text-based reasoning?

• The answer is hard and probably not. Then there is a follow-up realistic question: if OpenAI release GPT-5 next month, with its MMMU score increase from 56 to 70 but MMLU retains to be 86, what kind of message would it send and how would the public react to it?

MMMU leaderboard screenshot

• Yet the good news is, even if video data do not increase reasoning, it improves everything else, particularly grounding, thus enabling the model to receive feedback from the real-world.

• Specifically one may need to scale:
• The horizon of model’s exploration, e.g., deployment the model online for a year and update on a weekly basis, instead of single-step optimization
• The search space, e.g., let the model generate one million responses and pick the best of them, instead of the original InstructGPT’s best-of-7 approach.
• The source of feedback, specially gradually move from human feedback (because human feedback does not scale and the model is becoming stronger than their human annotators) to AI and environment feedback (thus the requirement for world model).

• A very unfortunate fact is that many of the existing research work is looking into tiny details of small-scale single-round optimization, e.g., adding one loss term on DPO. Yet the key here is online iterative large scale exploration and exploitation.

4 - Scaling unified video-language generative model

• Should we train on separate modalities first, then use adapters to bridge modalities, as is the current practices like LLaVA, or should we directly train on the mixture of all modalities?

• Should we use a unified transformer backbone, or some CV stuff, like UNet and CNNs, for the image / video part? What type of modifications we should make on the transformer architecture (e.g., 3D positional encoding)? How to make the best of mixture of expert layers?

• Adding new modalities should at least not harm existing modalities, yet it is common observation that adding vision may negatively influence language. How to reconcile the contradiction between different modalities?

• For the video understanding part, how to do tokenization/ representation learning? Should one consider VQ-VAE styled discrete tokens or Sora-styled continuous space-time patches? Should one use contrastive-styled objective like CLIP or reconstruction-styled objective like the original VAE?

• For the video generation part, should it be like autoregressive like VideoPoet or diffusion-based like Sora? How to train a transformer model that can simultaneously perform diffusion-styled generation and autoregressive generation?

5 - AlphaZero-styled Agents through iterative reinforcement learning from X feedback

• Use code-based feedback to verify the answer

• Use process-based reward model to verify the answer

• Use expert-level annotation to produce the feedback

6 - Conclusion: the second chapter of the game of scale