In high school, I found vulnerabilities in:
Nowadays I'm working on finding all vulnerabilities on the internet. That means you need to make machines exceptional at reverse engineering, as other people's web applications usually don't have their source code available.
Reverse engineering is difficult, but turns out the more difficult problem is verifying vulnerabilities.
A vulnerability is a behavior reachable under the implementation that should be unreachable under the intended specification. To verify one, you need the intended specification. For most vulnerability classes, this specification is application-specific and never written down. There is no general oracle.
For certain vulnerability classes, it's easy to build universal, deterministic oracles. Examples are memory corruptions (crash-based, sanitizer-based...), SQL injections (SLEEP 5 vs SLEEP 10) and cross-site scripting (alert(canary)).
Almost nothing else has a universal oracle. All questions concerning access control, such as whether a particular object access is permitted, depends on the application's semantics, and by Rice's theorem you cannot recover these from the implementation alone in general.
Many think that LLMs are good solutions to the verification problem. Unfortunately, this is not the case as they will always yield a nontrivial amount of false positives.
Suppose your verifier has a 1% false positive rate. You test 50,000 candidate inputs against an application with 5 real vulnerabilities. The prior P(vulnerability | input) is on the order of 10-4.
You report 5 true positives and 500 false positives. Precision is 1%, and the user learns to treat every finding as noise. If you have a 1% false positive rate, you will neither be able to find all vulnerabilities on the internet nor build a useful enterprise security product.
Whoever solves verification first finds all vulnerabilities on the internet. This is a prerequisite for AGI. If DPRK can exfiltrate your weights or find vulnerabilities faster than you can patch them, it doesn't matter who built AGI.
We are hiring.
