Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

Claim under test

Does capacity headroom predict where PPO helps?

The paper reports a capacity-headroom pattern: reinforcement learning gains should depend jointly on model scale and how well supervised fine-tuning already fits the task.

This is a claim-level matrix test, not a search for one favorable checkpoint. Numerical, directional, and family-wise assessments are specified in advance. The overall interpretation remains locked until all 15 Track R arms complete.

Hardware and stack provenance

Three cards, two execution profiles.

Faster hardware changes wall time, not the frozen data, seed, hyperparameters, or evaluation. The execution profile remains attached to every arm.

Deviation register

Append-only by design.

1. D-001RTX PRO 6000 arms only

   The authors' pinned PyTorch 2.5.1 stack cannot execute CUDA kernels for Blackwell sm_120. These arms use PyTorch 2.9.1 with CUDA 12.8 while retaining the other paper pins.

   Control: They are labeled COMPAT, kept out of exact-stack claims, and require exact-stack checkpoint re-evaluation before comparison.

2. D-002Wall-clock execution only

   The 15 configurations are scheduled concurrently across three locally controlled GPUs.

   Control: Data, seed, model, hyperparameters, and evaluation remain unchanged; host, GPU, stack, pause events, and commit are recorded per arm.

3. D-003Four SmolLM2 attempts

   Four immutable attempts now have terminal failure manifests. The 135M TinyStories attempt exited after SFT; the three RTX PRO 6000 attempts ended before completing the full SFT, reward, PPO, and evaluation pipeline.

   Control: Partial artifacts remain attached to their original attempt IDs and do not count as completed runs. Any recovery must create a new attempt with explicit provenance linkage; the claim-level result gate remains closed.