Transitioning from a creator to a user of CUDA provides critical insights and humility by revealing the incorrect assumptions made during development.
CUDA has evolved from a low-level language into a comprehensive platform of compilers, libraries, and SDKs that enable GPU access for multiple languages.
CUDA supports languages like Fortran to accelerate existing codebases in supercomputing for fields such as physics and weather forecasting.
Python's large ecosystem, developer productivity, and vast talent pool made it the de facto language for AI, creating new challenges for parallel computing platforms.
Developing new chips takes years of predictive work, creating a challenge to meet the rapidly changing demands of software, especially in the AI space.
Technology evolves organically as users find novel applications for existing tools, such as using gaming GPUs for scientific computing and AI.
Advances that make AI models cheaper or more efficient don't reduce overall compute demand; instead, they enable the creation of even larger and more powerful models.
While transistor density still doubles, the power per transistor is not halving, creating a thermal and power delivery bottleneck for chip performance.
Overcoming power limitations requires moving from single-chip optimization to building large, networked, data-center-scale systems with specialized hardware.
The future of computing will be a hybrid model combining classical, neural, and quantum approaches to solve complex problems using the best tool for each task.
While low-level drivers are proprietary, the vast majority of CUDA's higher-level libraries like Rapids and Cutlass are open source and welcome community contributions.
21:02Andy Terrel
59:43Paul Graham
22:07Ankit Patel
1:09:10Chris Heilmann, Daniel Cranney & Nicole Jeschko
24:39Paul Graham
22:18Paul Graham
29:52Anshul Jindal & Martin Piercy
34:48Maria Apazoglou
Jobs that call for the skills explored in this talk.
