The Computing Community Consortium’s (CCC) latest report, Beyond Code: Engineering Trustworthy Software Systems with AI at Scale, examines what happens to software engineering as AI takes over most of the work of writing code. Between 84% and 90% of developers now use these tools. The report is primarily addressed to researchers, but several of its findings describe conditions that research alone cannot resolve.

Beyond Code: Engineering Trustworthy Software Systems with AI at Scale was authored by Randal Burns (Johns Hopkins University), Sebastian Elbaum (University of Virginia), Gabrielle Allen (University of Wyoming), Nils Aschenbruck (Osnabrück University), Terry Benzel (University of Southern California), William Gropp (University of Illinois Urbana-Champaign), Rick Kazman (University of Hawaii), and Manish Parashar (University of Utah).

Read the Report Here

Avoiding Security Breaches in AI-Generated Software

Software built using AI is entering systems that people depend on. But AI now generates code faster than anyone can review it, and the report finds that the resulting systems carry unclear provenance, configuration errors, unvetted third-party components, and expanded attack surfaces. Faults missed in AI-generated code are already causing security breaches.

Organizations also cannot fully tell what is in the code they receive. Because these systems are black boxes with respect to their training data, users are unable to determine whether generated code contains copyrighted material, infringes patents, or reproduces proprietary work belonging to someone else. Major providers have responded by offering their customers legal indemnification. But interestingly, the market is developing ways to allocate this risk faster than the technical means to actually measure it.

Meanwhile, the checks that would normally catch these problems are weakening:

• Reviewer fatigue. People facing large volumes of machine-generated code miss subtle faults and tend to defer to the machine
• Misleading test results. Automated testing can produce results that look thorough while confirming the wrong behavior
• Concentration risk. When most organizations rely on the same dominant model, they inherit the same blind spots at once

Because of this, verifying code has to scale the way generating code has; and it needs research funding to do so. The report calls for automated checking that runs at machine speed, including formal verification methods that prove a system meets its specification rather than testing samples of its behavior. It also calls for tools that trace generated code back to its sources and for auditing of training data that establishes clear licensing, so organizations can assess what they are deploying before they deploy it. Against concentration risk, it proposes using different AI models to probe each other’s output for vulnerabilities.

Who Can Afford to Study These Systems

Evaluating whether AI systems are safe requires running them at the scale they actually operate. The report finds that the ability to do that now sits almost entirely with industry, which controls the computing power, the training data, and the engineering talent, along with the deployment environments where these systems run in production. Universities increasingly study commercial products after release rather than building the next generation of systems themselves. The report describes academic researchers:

• Evaluating industry outputs rather than developing systems of their own
• Auditing systems they cannot see inside for bias and other properties
• Reproducing findings at a fraction of the scale available to industry labs
• Leaving for industry altogether

University labs still lead on safety questions, including the social and ethical effects of AI, but the report finds academic relevance declining elsewhere. It identifies one fix as fast-acting: giving academic researchers access to frontier computing infrastructure, through a combination of industry funding and national compute initiatives.

A Workforce Built for Different Work

The report finds that the job of a software developer is changing, from writing code toward specifying what a system should do and supervising the AI tools that produce it. Industry has already started hiring on that basis, evaluating engineers on system design and their ability to direct and verify AI output. That shift outpaces the training pipeline. In higher education, the report calls for:

• Curricula centered on requirements, specification, and system design rather than isolated programming exercises
• New teaching approaches, since students will no longer build their understanding of systems through the act of programming
• Retraining beyond the university, reaching the current workforce whose implementation-focused roles are disappearing

Read the Full Report

The findings of this report are informed by discussions at the CCC Beyond Code: Engineering Trustworthy Software Systems with AI at Scale visioning workshop held February 25-26, 2026 in San Francisco, California, generously supported by the National Science Foundation (NSF) and the IEEE Computer Society. The workshop convened 41 experts across academia, industry, and government in artificial intelligence, software engineering, programming languages, cybersecurity, and systems engineering.

Read the Full Report

Tune in to the CCC LinkedIn Showcase Page for updates and more reports like this. Stay connected with CCC for the latest insights, publications, and opportunities to engage by subscribing here.

This material is based upon work supported by the U.S. National Science Foundation (NSF) under Award Nos. 2300842 and 2619366. These awards support the Computing Community Consortium (CCC), a programmatic committee of the Computing Research Association (CRA). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.