Scaling reinforcement learning with verifiable rewards for LLMs on LUMI

A supercomputer such as LUMI consists of many compute nodes that can work together to solve large computational problems. Adding nodes can help complete a task faster or make it possible to process a larger workload, but the benefit depends on how efficiently the added resources are used. This is known as scaling. 

Using a large language model (LLM) training workload as an example, this technical blog post, written by Markus Junttila, Gaurav Naithani, and Marlon Tobaben, explains different types of scaling, what can affect performance, and how to measure whether additional resources are useful. While your own AI workload might differ, many of the same principles still apply.

Reinforcement learning (RL) has become a vital method for post-training LLMs, particularly to improve their reasoning and problem-solving abilities. While scaling of LLM pre-training has previously been studied on the LUMI supercomputer, there appears to be little published work on scaling RL. We use the open-source framework verl to study how RL post-training scales from one to 16 nodes on LUMI-G, LUMI’s GPU partition. Our findings provide academic and industry users with a practical foundation for further reinforcement learning experimentation on LUMI.

Read the full blog post on the LUMI AI Factory website.