Large language models for program verification

Abstract

Program verification is a crucial process in software engineering, ensuring that programs func- tion correctly according to their specifications. Traditional methods of program verification, such as model checking and symbolic execution, often require significant manual effort and com- putational resources. This report explores the potential of Large Language Models to address challenges in program verification by automating critical tasks such as generating loop invariants, preconditions, and post conditions, as well as directly performing verification. The research is divided into two primary objectives. The first focuses on leveraging LLMs for the generation of loop invariants, a complex yet essential component of program verification. By utilizing LLMs’ generative capabilities, we aim to reduce the manual effort involved in formal specification creation. The second objective evaluates the reliability of LLMs as standalone verifiers, comparing their performance against state-of-the-art verification tools using the SV- COMP benchmark, which includes diverse C programs and verification properties. The experiments involve testing LLMs, such as LLaMA 3.1 and 3.2, on real-world verification tasks. Results indicate promising capabilities of LLMs in generating verification components and reasoning about program correctness. However, challenges remain in fully replacing traditional techniques. This study highlights the potential of integrating LLMs into formal verification workflows, paving the way for more efficient and scalable solutions in software verification.