Blog

The mission of the hackathon: to write a program that can scan millions of lines of open source code, identify security flaws and fix them, all without human intervention. Success would mean winning millions of dollars in a two-year competition sponsored by DARPA, the Defense Advanced Research Projects Agency.

The competition is one of the clearest signs yet that the government sees flaws in open source software as one of the country’s biggest security risks, and sees artificial intelligence as vital to tackling it.

Free and open source programs, such as the Linux operating system, help run everything from websites to power plants. The code is inherently no worse than what’s in the proprietary programs of companies like Microsoft and Oracle, but there aren’t enough skilled engineers tasked with testing it.

As a result, poorly maintained free code is at the root of some of the costliest cybersecurity breaches of all time, including the 2017 Equifax disaster that exposed the personal information of half of all Americans. The incident, which led to the largest data breach settlement, cost the company more than $1 billion in improvements and penalties.

If humans can’t handle all the code woven into every industrial sector, DARPA hopes machines can.

“The goal is to have a complete ‘cyber reasoning system’ that uses large language models to detect vulnerabilities, prove that they are vulnerabilities, and fix them,” explained one of the consulting professors, Jan Shoshitaishvili of Arizona State.

To get there, the team grapples with the often grim reality behind AI’s lofty aspirations. Students do things like enforce “sanity checks” to catch hallucinations, verify that patches actually fix the problems they’re supposed to, and have two AI systems debate each other over the best fixes—with a third AI deciding the winner.

“AI is like a 3-year-old with infinite knowledge,” said UC-Santa Barbara student and team co-captain Lucas Dressel. “You need to give him useful feedback.”

Team Shellphish is one of about 40 competitors in a competition known as AIxCC, an artificial intelligence cyber challenge run by DARPA, the Pentagon’s research arm charged with developing secret weapons and defending against them.

“We want to redefine how we provide widely used, critical codebases because of how ubiquitous open source is in critical infrastructure sectors,” said Andrew Carney, DARPA’s project manager for the competition.

While DARPA helped birth the Internet to survive communication failures, it has become painfully obvious that the network has also introduced massive weaknesses.

With no built-in security, massive interconnections allow anyone or anything to start anywhere and find their way into the machines that power the modern world. Once inside, users can impersonate employees or system administrators, steal national or trade secrets, and shut down the site or hold it for ransom.

Hackers take more victims than ever: The number of data breaches reported to FBI-run US Internet Crime Complaint Center triples between 2021 and 2023 Government agents snoop in rival nations electric and water plants. Criminal gangs, consumed by illegal profits, think nothing of eliminating hospitals and sending desperate patients elsewhere.

Open source software, whether written by students or visionary geniuses, is almost as ubiquitous as the Internet itself, found in 90% of commercial software by some estimates.

Like any software, it has bugs, some of which can be used to seize control of a machine.

Some large open source projects are run by volunteer armies close to the size of Wikipedia, and are generally in good shape. Some have backers who get grants from large corporate users who make it work.

And then there’s everything else, including programs written as homework by authors who barely remember them.

“Open source has always been ‘use at your own risk,'” said Brian Bellendorff, who founded the Open Source Security Foundation after decades of maintaining pioneered free server software, Apache, and other Apache Software Foundation projects.

“It’s not free like speech or even free like beer,” he said. “It’s loose as a puppy and needs care and feeding.”

The risks were highlighted recently by two very different incidents.

The first was a vulnerability in a small system activity tracker known as Log4j, used by thousands of software developers and installed on millions of machines.

In 2013, a user suggested adding some code to Log4j, and the small Apache Foundation team supporting Log4j approved it. In November 2021, a Chinese engineer saw that the added section contained a huge design flaw that would allow a system takeover, and he noted the problem to the Apache group.

While Apache was working on a patch to fix the problem, an unidentified researcher discovered the upcoming changes and developed a malicious tool to control computers running Log4j. Apache pushed the patch, setting off a race between thousands of defenders and those trying to exploit the flaw before it was patched.

Many instances of Log4j are still unfixed. On Thursday, the National Security Agency and others warned that North Korean spies were still breaking into US web servers running old versions.

The White House Cybersecurity Review Board concluded that only better coding and thorough audits could stop the spread of the Log4j flaw, and that open source efforts like Apache’s “will need continued financial support and expertise.”

The Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) has responded with small grants for startups and insists companies declare what’s in their software. But these are slow initiatives.

The latest reminder of the vulnerability came in March. Then a Microsoft engineer tracked a slight increase in CPU usage for open-source Linux tools that had just been updated. He discovered that a spying backdoor had been inserted by the official maintainer of the tools and blew the whistle in time to stop its delivery in the most popular versions of Linux.

In a nightmare scenario for security professionals, the anonymous maintainer won control of the project after contributing for years, aided by secret allies who lobbied the previous manager to cede control.

As open source security rose to a top priority for CISA and national security institutions, OpenAI and Microsoft unleashed ChatGPT and generative artificial intelligence on the world.

By democratizing programming, new tools allowed non-coders to create software. AI also helped existing programmers, including criminal hackers, who could more quickly incorporate tricks to exploit vulnerabilities and deliver more convincing decoys, such as emails that appeared to come from regular contacts with shared interests.

AI also enhances defensive efforts, such as analyzing reams of log files for unusual behavior and summarizing security incidents. It can also flag security bugs in programs as they are written.

But finding where the holes are in open source programs before attackers find them is a holy grail for DARPA and the AIxxCC contestants.

DARPA held a cyber challenge at the Def Con 2016 hacking convention, where programs competed in a “capture the flag” competition to hack each other in an artificial environment.

In this year’s competition, teams use their AI-enhanced programs to digest and improve millions of lines of real-world code.

Shellphish is one of seven teams that wrote papers outlining their approach well enough to receive $1 million in funding for the steps, which will culminate in the semifinals in August at Def Con, which attracted 40 entries. The winner will receive another $2 million in 2025.

Some of Shellphish’s first million dollars went to the Brea home listed on Airbnb, which hosted hackers for three weeks in June and another two in July. More went for a huge test environment that uses 5000 CPU cores.

Shellphish is not a random group of hackers. Although heavily affiliated with two state universities with changing populations, the team has been around for 20 years and its founders are still involved.

Italian Giovanni Vigna was teaching computer security at UC-Santa Barbara, including offensive and defensive techniques, when he founded a capture-the-flag team in 2003 to engage students more and expand their capabilities. It won the Def Con competition in 2005, and a spin-off of the team hosted the competition later for four years.

While his students graduated and spread to Arizona and elsewhere, some stayed involved or had their own students become involved.

Shellphish competed in the original 2016 Cyber ​​Grand Challenge, but were knocked out in the finals.

“We had all these great tools, but we ran out of time to integrate them,” Shoshitaishvili recalls. “So ‘Don’t fall for a nerd’ was my #1 tip.” (Need sniping refers to distracting a techie with an interesting problem.)

At the heart of the effort are tools known in the security field as “fuzzers.” They run all kinds of data into a program to see how it handles the unexpected.

Team members admit that even professionals cannot find the most obscure flaws or deliberate backdoors. At best, Shellphish’s main program and others will be able to quickly find a lot of low-hanging fruit and get rid of them before malicious hackers can exploit them.

“AI will be able to solve things that take humans months,” Dressel said.

Under the terms of the DARPA competition, all finalists must release their programs as open source so that software vendors and users can run them.

Yang compared the expected progress to security milestones such as forced software updates and browser “sandboxes” that prevent web programs from escaping the browser and executing elsewhere on the user’s device.

AI won’t be able to make all software safe, he said. But it will give people more time to try.

After a final, almost sleepless night of debugging and panicked last-minute fixes, Shellphish submitted its program by the 9am deadline. In a few weeks, at the next Def Con in Las Vegas, they’ll find out if they’re finalists. Win or lose, their AI-assisted code will be available for others to upgrade, improving security for all.