Introduction
In a recent talk by Neil Thompson, he discussed the next steps in AI/ML and the challenges and potential solutions that lie ahead. Thompson emphasized the importance of understanding the context of modern progress and its implications for technologies like large language models. He also highlighted the emergence of generative AI, exemplified by technologies such as GPT and Stable Diffusion. This article provides an overview of Thompson’s key points and offers insights into the future of AI/ML.
Computer Vision and Exponential Growth
Thompson started by analyzing computer vision, an older technology that can guide us in predicting future trends. He showcased the rapid progression of computer vision, citing the example of the ImageNet database. He noted that AI improvements feel exponential because of the smooth transitions observed in computer vision’s exponential scale.
Thompson pointed out that the growth of AI models comes at a cost. The increased size of models consumes more computing power, resulting in higher expenses and a significant carbon footprint. He expressed concerns about resource scarcity in larger AI systems, which could potentially lead to a lack of model diversity.
The Moore’s Law Conundrum
Thompson drew parallels between AI progress and Moore’s Law, which states that computing power doubles approximately every two years. However, he highlighted that Moore’s Law is reaching its limits. To continue advancing AI, researchers need to explore alternatives such as hardware accelerators, quantum computing, and algorithmic improvements.
Thompson acknowledged that these alternatives are still in the theory stage and may lack consistency. Finding the necessary performance for next-generation AI systems is a significant challenge. He emphasized the importance of ongoing research to identify how to achieve the desired performance while moving up the AI curve.
Conclusion
The presentation by Neil Thompson shed light on the future of AI/ML, highlighting the need for thoughtful consideration of the challenges and potential solutions. As AI models continue to grow in size and complexity, it is crucial to address resource scarcity and carbon footprint issues. Exploring alternative technologies and making algorithmic improvements are vital to overcome the limits of Moore’s Law. Researchers and engineers must strive to find innovative solutions to ensure the continued advancement of AI/ML.
