Akbir Khan, AI researcher and ICML best paper winner, discusses his work on AI alignment, debate techniques for truthful AI responses, and the future of artificial intelligence. Key points discussed: - Using debate between language models to improve truthfulness in AI responses - Scalable oversight for supervising AI models beyond human-level intelligence - The relationship between intelligence and agency in AI systems - Challenges in AI safety and alignment - The potential for a Cambrian explosion in human-like intelligent systems The discussion also explored broader topics: - The wisdom of crowds vs. expert knowledge in machine learning debates - Deceptive alignment and reward tampering in AI systems - Open-ended AI systems and their implications for development and safety - The space of possible minds and defining superintelligence - Cultural evolution and memetics in understanding intelligence Akbir Khan: https://x.com/akbirkhan https://akbir.dev/ Show notes and transcript: https://www.dropbox.com/scl/fi/sjekivbg3ok6qugsv2p1u/AkbirKhan.pdf?rlkey=ewiyvq0aq7mjvql4u7os0jos2&st=vblhp7af&dl=0 TOC (*) are best bits 00:00:00 1. Intro: AI alignment and debate techniques for truthful responses * 00:05:00 2. Scalable oversight and hidden information settings 00:10:05 3. AI agency, intelligence, and progress * 00:15:00 4. Base models, RL training, and instrumental goals 00:25:11 5. Deceptive alignment and RL challenges in AI * 00:30:12 6. Open-ended AI systems and future directions 00:35:34 7. Deception, superintelligence, and the space of possible minds * 00:40:00 8. Cultural evolution, memetics, and intelligence measurement References: 1. [00:00:40] Akbir Khan et al. ICML 2024 Best Paper: "Debating with More Persuasive LLMs Leads to More Truthful Answers" https://arxiv.org/html/2402.06782v3 2. [00:03:28] Yann LeCun on machine learning debates https://www.youtube.com/watch?v=OKkEdTchsiE 3. [00:06:05] OpenAI's Superalignment team https://openai.com/index/introducing-superalignment/ 4. [00:08:10] Sam Bowman on scalable oversight in AI systems https://arxiv.org/abs/2211.03540 5. [00:10:35] Sam Bowman on the sandwich protocol https://www.alignmentforum.org/posts/nekLYqbCEBDEfbLzF/artificial-sandwiching-when-can-we-test-scalable-alignment 6. [00:14:35] Janus' article on "Simulators" and LLMs https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators 7. [00:16:35] Thomas Suddendorf's book "The Gap: The Science of What Separates Us from Other Animals" https://www.amazon.in/GAP-Science-Separates-Other-Animals/dp/0465030149 8. [00:19:10] DeepMind on responsible AI https://deepmind.google/about/responsibility-safety/ 9. [00:20:50] Technological singularity https://en.wikipedia.org/wiki/Technological_singularity 10. [00:21:30] Eliezer Yudkowsky on FOOM (Fast takeoff) https://intelligence.org/files/AIFoomDebate.pdf 11. [00:21:45] Sammy Martin on recursive self-improvement in AI https://www.alignmentforum.org/posts/5WECpYABCT62TJrhY/will-ai-undergo-discontinuous-progress 12. [00:24:25] LessWrong community https://www.lesswrong.com/ 13. [00:24:35] Nora Belrose on AI alignment and deception https://www.lesswrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom 14. [00:25:35] Evan Hubinger on deceptive alignment in AI systems https://www.lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment 15. [00:26:50] Anthropic's article on reward tampering in language models https://www.anthropic.com/research/reward-tampering 16. [00:32:35] Kenneth Stanley's work on open-endedness in AI https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237 17. [00:34:58] Ryan Greenblatt, Buck Shlegeris et al. on AI safety protocols https://arxiv.org/pdf/2312.06942 18. [00:37:20] Aaron Sloman's concept of 'the space of possible minds' https://www.cs.bham.ac.uk/research/projects/cogaff/sloman-space-of-minds-84.pdf 19. [00:38:25] François Chollet on defining and measuring intelligence in AI https://arxiv.org/abs/1911.01547 20. [00:42:30] Richard Dawkins on memetics https://www.amazon.co.uk/Selfish-Gene-Richard-Dawkins/dp/0192860925 21. [00:42:45] Jonathan Cook et al. on Artificial Generational Intelligence https://arxiv.org/abs/2406.00392 22. [00:45:00] Peng on determinants of cryptocurrency pricing https://www.emerald.com/insight/content/doi/10.1108/CAFR-05-2023-0053/full/html