I’ve been experimenting with using GPT-4o and Claude 3.5 to audit my Solidity code. While they catch basic reentrancy issues, they often miss deep logical flaws or flag "bugs" that aren't actually there. Is there a specific prompt framework or a "Multi-Agent" setup you’re using to make AI audits more reliable before sending code to a human firm like OpenZeppelin? I'm worried about the legal liability of relying too much on AI.
3 answers
You cannot rely on a single LLM pass for something as critical as a smart contract. We built a three-step pipeline: First, a "Static Analysis" agent runs tools like Slither and Mythril. Second, a "Vulnerability Scanner" agent (using Claude) looks at the output and explains the risks. Third, a "Critic" agent (using GPT-4) tries to find reasons why the second agent is wrong. This "Adversarial" approach has caught logic errors that a single prompt never would have found. However, we still treat this as a "pre-audit" step; we never deploy to Mainnet without a human senior auditor signing off.
Deborah, how do you handle the high costs of running three different high-end models for every minor code change in your repository?
I find that the "System Prompt" is the most important part. You have to tell the AI to "think step-by-step" and specifically act as a malicious black-hat hacker.
Good point, Sandra. "Thinking step-by-step" or Chain of Thought (CoT) is essential. It forces the model to trace the state changes in the contract more accurately.
Gary, we don't run the full "Adversarial" pipeline for every commit. We only trigger it when a PR is labeled "High Risk" or involves changes to the tokenomics logic. For daily linting, we use a much lighter fine-tuned model that only looks for syntax errors. This keeps our developer productivity high without burning through our OpenAI credits by noon every day.