SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting! https://centml.ai/pricing/ Prof. Jakob Foerster (FLAIR lab, Oxford / Meta AI) and his PhD student Chris Lu make the case that deep reinforcement learning is finally winning the hardware lottery. The core thesis: RL has underperformed not because the ideas are wrong, but because running environments on CPUs while training agents on GPUs created a computational bottleneck that made experimentation slow, expensive, and brittle. JAX-based GPU-native environments now deliver ~4000x speedups, enabling the kind of rapid iteration that made supervised deep learning successful. Chris Lu explains the technical foundation -- how JAX's JIT compilation and vmap (vectorized map) allow writing a single environment instance in NumPy-like code and scaling it to millions of parallel copies on GPU. This started from necessity: their lab initially had only Google Colab free-tier compute. The constraint forced them to put environments on GPU for their Model-Free Opponent Shaping paper, and the results were surprisingly effective. The conversation then shifts to discovered policy optimization. Foerster's mirror learning framework provides theoretical grounding for why PPO works, and crucially, lets you parameterize the drift function as a neural network and meta-learn it. Evolution strategies, not gradient-based meta-learning, turned out to be the better optimizer for this -- a vindication of the bitter lesson. The learned policy optimization function shows intriguing "too good to be true" behavior: when advantages are very high, it clips more aggressively, as if it has learned skepticism. The second half covers multi-agent systems, emergent communication, and AI governance. Foerster argues forcefully for open-source AI development and democratic control, drawing analogies to CERN. His position: the biggest alignment challenge is not between AI and humans, but between those who control AI systems and the rest of the population. Concentrated AI development creates fragile single points of failure; distributed development is both safer and more innovative. --- REFERENCES: paper: [00:03:05] Deep RL Doesn't Work Yet https://www.alexirpan.com/2018/02/14/rl-hard.html [00:06:10] JaxMARL https://arxiv.org/html/2311.10090v5 [00:08:50] M-FOS: Model-Free Opponent Shaping https://arxiv.org/abs/2205.01447 [00:12:10] Kinetix Physics Simulator https://arxiv.org/abs/2410.23208 [00:14:42] Mirror Learning Framework https://arxiv.org/abs/2208.01682 [00:16:30] Discovered Policy Optimisation https://arxiv.org/abs/2210.05639 [00:28:55] AlphaGo https://arxiv.org/abs/1712.01815 [00:41:00] Open Source Generative AI https://arxiv.org/abs/2405.08597 tool: [00:09:45] JAX Library https://github.com/jax-ml/jax [00:49:51] Llama 3 https://ai.meta.com/blog/meta-llama-3/ concept: [00:24:10] Goodhart's Law https://en.wikipedia.org/wiki/Goodhart%27s_law --- LINKS: Full Transcript: https://app.rescript.info/share/04498ba49b081dbcc254c40ba7b56035 Download PDF transcript: https://app.rescript.info/api/public/sessions/8fe06defb6d2216d/pdf Prof. Jakob Foerster https://x.com/j_foerst https://www.jakobfoerster.com/ University of Oxford Profile: https://eng.ox.ac.uk/people/jakob-foerster/ REFS [[00:00:05] ARC Benchmark, Chollet https://github.com/fchollet/ARC-AGI [00:09:45] JAX Library, Google Research https://github.com/jax-ml/jax [00:25:15] LLM ARChitect, Franzen et al. https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf