When I first began participating in public competitions in early 2025, I, like many beginners, would simply ask an LLM what mistakes and vulnerabilities it saw in a particular function or contract. This is fundamentally the wrong way to use an LLM.
LLMs very often hallucinate when you ask them to draw conclusions or analyze code for vulnerabilities. Any answers that involve reasoning and analysis should be treated with skepticism. In the vast majority of cases, such answers will be incorrect.
I remember an LLM claiming that if, in some function, we transfer tokens to an external address but later during the function’s execution a revert occurs, then the transferred tokens would get stuck in the external contract. Of course, if you already have some experience and understand that a function does not execute in parts and that if a revert happens no changes are applied, no tokens are transferred, and nothing gets stuck, then you can see that the LLM is hallucinating.
But you need to already have some experience in how the blockchain works and at least a basic understanding of Web3 security to work with LLMs.
Use them as a sort of protocol developer, because they do a good job understanding what code does.
And don’t use them as security researchers, because they have a limited grasp of smart contract security and can only find the most obvious vulnerabilities (at least given the current state of LLM development).