The CCC held the Systems and Applications Challenges for the Emerging Bazaar of Accelerators workshop in August of 2023 in Washington, DC. The workshop was organized by Catherine Schuman (University of Tennessee, Knoxville and CCC Full Stack Task Force member) and John Shalf (Lawrence Berkeley National Laboratory), and was supported by Thomas Conte (Georgia Institute of Technology and previous CCC Council member). The workshop, which was attended by about 25 participants from industry, academia, and government, focused on the challenges associated with a future of computing with ubiquitous heterogeneous accelerators.
As the end of Moore’s Law has come and gone, future performance gains are increasingly dependent on specialized computer architecture. This paradigm shift, however, does not simply necessitate plugging a few more diverse processors into existing systems. Incorporating diverse accelerators in new systems results in an incredible increase in complexity for developers on every layer of the full stack. On the hardware side, researchers are exploring ways to integrate emerging computing technologies (neuromorphic, quantum analog, probabilistic) into systems already rife with diverse accelerators. As each of these components function very differently, it is an enormous challenge to communicate data across components and measure overall performance. On the software side, heterogeneous accelerators function differently enough that they often require developers to program each component using different languages, and sometimes require new programming languages and abstractions to be developed. This complexity scales with each heterogeneous accelerator that is added to the system.
“The move towards specialization and heterogeneous computing is already afoot, and is accelerating”, says John Shalf. “It cost us $25 Million per application in the Exascale Computing Project to update our codes just to handle CPU+GPU heterogeneous acceleration, but now we are seeing even more variety in accelerators in the marketplace. We have to do something before the complexity grows overwhelming.”
Throughout the workshop, attendees grappled with these emerging challenges and discussed tools and solutions to remedy this growing problem. Some of the research directions attendees identified include:
- Re-evaluating and re-thinking benchmarking and performance metrics for heterogeneous systems, because existing metrics are formulated for systems with universal/general-purpose instruction processors, an architecture that heterogeneous systems do not necessarily follow.
- Developing new programming abstractions that take into account non-Von-Neumann heterogeneous accelerators
- Creating new funding avenues and projects for cross-disciplinary research that spans across different levels of the compute stack.
- Re-thinking how we train students with a greater focus on training generalists, who understand how to work across disciplines and can enable interdisciplinary collaborations.
“We will need to fundamentally rethink how we train the new generation to think about challenges across the full stack, especially as these more and more diverse systems emerge”, says Catherine Schuman. “It will be critical to have researchers who understand the challenges and opportunities for these systems, so that we can fully leverage the computational benefits they will provide.”
Thank you for reading, and please check out the full report for more details about the findings and recommendations from this workshop.
