Mohamed Osman from MindsAI (now Tufa Labs) joins Tim to discuss how his team achieved the highest score on the ARC Challenge 2024 — and why their approach matters for the future of deep learning. The conversation centers on test-time fine-tuning: the idea that you can treat each new problem as a fresh perceptual challenge and adapt a neural network's weights during inference, rather than relying solely on what was learned during training. Mohamed explains how they frame ARC puzzles as perception problems, pre-train a small Long T5 encoder-decoder on code and synthetic tasks to build a 'meta-model' with strong contextualisation ability, then fine-tune it at test time on each novel puzzle. The result: their model develops new abstractions on the fly, something most approaches struggle with. Tim and Mohamed dig into why VLMs are poorly suited for ARC (the frozen encoder commits to a representation before seeing the question), why beam search works well for voting (there is only one correct answer, so wrong beams disperse), and why neural networks fundamentally struggle with counting and copying due to softmax bottlenecks. They also discuss Kevin Ellis abandoning DSLs in favour of LLM-guided program synthesis, Clement Bonnet's Latent Program Networks, the relationship between code pre-training and reasoning ability, and what Chollet's forthcoming ARC v2 benchmark will demand. Mohamed shares scaling results from the hidden test set, explains why the $25k open-source incentive was not enough to justify sharing their full pipeline, and outlines Tufa Labs' research agenda: exploring compositionality in transformers, measuring contextualisation ability, and targeting 85% on ARC v1 with more compute and time. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. --- TIMESTAMPS: 00:00:00 Test-Time Fine-Tuning and the ARC Challenge 00:07:00 Neural Networks vs Programmatic Approaches to Reasoning 00:13:20 Code Pre-Training and the Meta-Model Architecture 00:20:30 Long T5 Architecture and Pre-Training Recipe 00:25:00 Voting, Beam Search, and Ensemble Methods 00:32:30 VLM Limitations and Raw Input Representation 00:40:30 ARC v2, Scaling, and the Future of the Benchmark 00:55:00 Transformer Architecture Limitations and Future Directions --- REFERENCES: paper: [00:00:00] How and Why Deep Learning for ARC https://github.com/MohamedOsman1998/deep-learning-for-arc/blob/main/deep_learning_for_arc.pdf [00:01:32] On the Measure of Intelligence https://arxiv.org/abs/1911.01547 [00:06:55] DreamCoder https://arxiv.org/abs/2006.08381 [00:13:35] Influence of Pretraining Data for Reasoning https://arxiv.org/abs/2411.12580 [00:17:50] Latent Program Networks https://arxiv.org/html/2411.08706v1 [00:20:50] T5: Exploring Transfer Learning https://arxiv.org/abs/1910.10683 [00:30:30] Combining Induction and Transduction for Abstract Reasoning https://arxiv.org/abs/2411.02272 [00:40:05] ARC Prize 2024 Technical Report https://arxiv.org/html/2412.04604v2 [00:45:20] LLM-Guided Compositional Program Synthesis https://arxiv.org/html/2503.15540 [00:59:35] ConceptARC Benchmark https://arxiv.org/abs/2305.07141 [01:02:05] Mixtape: Breaking the Softmax Bottleneck http://papers.neurips.cc/paper/9723-mixtape-breaking-the-softmax-bottleneck-efficiently.pdf resource: [00:01:06] Tufa AI Labs https://tufalabs.ai/ [00:54:25] ARC-AGI Repository https://github.com/fchollet/ARC-AGI [00:57:10] ARC Prize https://arcprize.org/ --- LINKS: Full Transcript: https://app.rescript.info/share/1dc39b77d7c23d3c7160190937f3519a Download PDF transcript: https://app.rescript.info/api/public/sessions/fe6512543cb85f5e/pdf Mohamed Osman (Tufa Labs) https://x.com/MohamedOsmanML Jack Cole (Tufa Labs) https://x.com/MindsAI_Jack How and why deep learning for ARC paper: https://github.com/MohamedOsman1998/deep-learning-for-arc/blob/main/deep_learning_for_arc.pdf [00:54:25] Abstraction and Reasoning Corpus, Francois Chollet https://github.com/fchollet/ARC-AGI