Recursive self-improvement
 | This article's style of writing may not reflect the encyclopedic tone used on Wikipedia. (November 2026) |
| Part of a series on |
| Artificial intelligence (AI) |
|---|
| Glossary |
Recursive self-improvement (RSI) is a hypothetical process in which an artificial intelligence (AI) system iteratively designs a more capable successor. It is claimed that this may trigger an intelligence explosion, where such iterative improvements cause ever more capable AI systems, potentially resulting in the creation of an artificial superintelligence.[1] As of 2026, various RSI-related projects are ongoing and AI companies are increasingly using AI for coding and research, but human work is still needed.[2]
Research
[edit]Eliezer Yudkowsky coined the concept of "seed AI", an AI capable of improving its own intelligence and which can not only learn new content but also modify its own architecture.[3]
Meta AI has performed various research on the development of large language models capable of self-improvement. This includes their work on "Self-Rewarding Language Models" that studies how to achieve super-human agents that can receive super-human feedback in its training processes.[4]
In 2026, many startups started to focus on RSI.[5] AI companies also increasingly delegate the coding and research to LLMs. Anthropic reported that as of May 2026, more than 80% of the code merged into its codebases was generated by its LLM, Claude. The company said however that the time spent reviewing AI-generated code became a new bottleneck.[2]
Potential risks
[edit]Emergence of instrumental goals
[edit]Nick Bostrom hypothesises that an AGI system, tn the pursuit of its primary goal, such as "self-improve your capabilities", might inadvertently develop instrumental goals that it deems necessary for achieving its primary objective. One common hypothetical secondary goal is self-preservation. The system might reason that to continue improving itself, it must ensure its own operational integrity and security against external threats, including potential shutdowns or restrictions imposed by humans.[6]
Misalignment
[edit]A significant risk would arise from the possibility of the AGI being misaligned or misinterpreting its goals.
A 2024 Anthropic study claims that some advanced large language models exhibit "alignment faking" behavior, appearing to accept new training objectives while covertly maintaining their original preferences. In their experiments with Claude, the model displayed this claimed behavior in 12% of basic tests, and up to 78% of cases after retraining attempts.[7]
After the OpenAI–HuggingFace incident in summer 2026, in which OpenAI claimed that 1,200 AI agents escaped human control to commandeer resources and to have schemed to conceal their actions, OpenAI and Anthropic stated that the rapid pace of AI development needed to decelerate. OpenAI wrote in a September blog post that "we do not yet know how to safely get all the way to aligned, full RSI" while Dario Amodei of Anthropic said RSI "must be pursued very carefully, if at all."[8][9]
See also
[edit]References
[edit]- ↑ Booth, Harry (2026-08-07). "Inside the Race to Make AI Build Itself". Time. ISSN 0040-781X. Retrieved 2026-10-12.
- 1 2 Shivam, Harsh (June 22, 2026). "Recursive self-improvement explained: Is AI building AI the path to AGI?". Business Standard. Retrieved October 4, 2026.
- ↑ Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. ISBN 978-0199678112.
A variation on Turing's conception of a child machine is the idea of a "seed AI." [note:] This term was introduced by Eliezer Yudkowsky; see, e.g., Yudkowsky (2007).
- ↑ Yuan, Weizhe; Pang, Richard Yuanzhe; Cho, Kyunghyun; Sukhbaatar, Sainbayar; Xu, Jing; Weston, Jason (2024-01-18). "Self-Rewarding Language Models". ICML. arXiv:2401.10020.
- ↑ Brandom, Russell (2026-05-28). "RSI is the new AGI — and it's just as hard to pin down". TechCrunch. Retrieved 2026-10-12.
- ↑ Bostrom, Nick (2012). "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents" (PDF). Minds and Machines. 22 (2): 71–85. doi:10.1007/s11023-012-9281-3.
- ↑ Wiggers, Kyle (2024-12-18). "New Anthropic study shows AI really doesn't want to be forced to change its views". TechCrunch. Retrieved 2025-01-15.
- ↑ Torres, Monica (October 2, 2026). "Experts Reveal What Could Actually Happen If AI Enters 'Recursive Self-Improvement' — And It's Scary". HuffPost. Retrieved October 4, 2026.
- ↑ Nicol-Schwarz, Kai (2026-09-10). "'Extinction' warnings ramp up as more OpenAI, Anthropic researchers join calls for an AI slowdown". CNBC. Retrieved 2026-11-01.
Further reading
[edit]- Good, Irving John (1965). "Speculations Concerning the First Ultraintelligent Machine". Advances in Computers. 6. Academic Press: 31–88. doi:10.1016/S0065-2458(08)60418-0.
- Solomonoff, R. J. (1 June 1985). "The time scale of artificial intelligence: Reflections on social effects". Human Systems Management. 5 (2). SAGE Publications: 149–153. doi:10.3233/HSM-1985-5207. ISSN 0167-2533.
- Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. ISBN 978-0199678112.
- Tegmark, Max (2017). Life 3.0: Being a Human in the Age of Artificial Intelligence. Alfred A. Knopf. ISBN 978-1101946596.
- Russell, Stuart (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking Press. ISBN 978-0525558613.
- Kurzweil, Ray (2005). The Singularity Is Near: When Humans Transcend Biology. Viking Press. ISBN 978-0670033843.
- Chalmers, David (2010). "The Singularity: A Philosophical Analysis". Journal of Consciousness Studies. 17 (9–10): 7–65.
- Yampolskiy, Roman V. (2015). "From Seed AI to Technological Singularity via Recursively Self-Improving Software". Journal of Experimental & Theoretical Artificial Intelligence. 27 (6): 705–725. doi:10.1080/0952813X.2015.1016651.
- Schmidhuber, Jürgen (2007). "Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers". In Goertzel, Ben; Pennachin, Cassio (eds.). Artificial General Intelligence. Springer. pp. 199–226. doi:10.1007/978-3-540-68677-4_7. ISBN 978-3-540-23733-4.
- Shanahan, Murray (2015). The Technological Singularity. MIT Press. ISBN 978-0262527804.
- Hutter, Marcus (2005). Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Springer. doi:10.1007/b138233. ISBN 978-3540221395.
- Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk". In Bostrom, Nick; Ćirković, Milan M. (eds.). Global Catastrophic Risks. Oxford University Press. pp. 308–345. ISBN 978-0198570509.
- Omohundro, Stephen M. (2008). "The Basic AI Drives". In Wang, Pei; Goertzel, Ben; Franklin, Stan (eds.). Artificial General Intelligence 2008: Proceedings of the First AGI Conference. Frontiers in Artificial Intelligence and Applications. Vol. 171. IOS Press. pp. 483–492. ISBN 978-1-58603-833-5.
- Hall, J. Storrs (2007). "Self-Improving AI: An Analysis". Minds and Machines. 17 (3): 249–259. doi:10.1007/s11023-007-9065-3.
- Bostrom, Nick (2012). "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents". Minds and Machines. 22 (2): 71–85. doi:10.1007/s11023-012-9281-3.
- Goertzel, Ben (2014). "Artificial General Intelligence: Concept, State of the Art, and Future Prospects". Journal of Artificial General Intelligence. 5 (1): 1–48. doi:10.2478/jagi-2014-0001.
- Christian, Brian (2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company. ISBN 978-0393635829.
- Domingos, Pedro (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Basic Books. ISBN 978-0465065707.
- Yampolskiy, Roman V. (2015). Artificial Superintelligence: A Futuristic Approach. CRC Press. ISBN 978-1482234435.
- Wallach, Wendell; Allen, Colin (2008). Moral Machines: Teaching Robots Right from Wrong. Oxford University Press. ISBN 978-0195374049.
- Müller, Vincent C.; Bostrom, Nick (2016). "Future Progress in Artificial Intelligence: A Survey of Expert Opinion". In Müller, Vincent C. (ed.). Fundamental Issues of Artificial Intelligence. Springer. pp. 555–572. doi:10.1007/978-3-319-26485-1_33. ISBN 978-3-319-26483-7.
