“Artificial Incorrectness: SMT and LLMs in Hardware Synthesis” by Edward Wang, Joe Walston, Luca Daniel, Tony Tan, Yoni Zohar, and Clark W. Barrett. In NASA Formal Methods - 18th International Symposium, NFM 2026, Los Angeles, CA, USA, May 5-7, 2026, Proceedings, (Jyotirmoy V. Deshmukh, Klaus Havelund, and Alessandro Pinto, eds.), 2026, pp. 261-284. Los Angeles, CA.
The adoption of large language models (LLMs) in hardware design automation poses correctness risks for safety-critical applications. We systematically evaluate LLMs against Satisfiability Modulo Theories (SMT) solvers across three hardware synthesis tasks, revealing that LLMs achieve lower levels of functional correctness compared to SMT approaches in our benchmarks. Our findings reveal a crucial distinction: whilst SMT solvers can excel at direct synthesis and can exhaustively validate LLM outputs, their counterexample feedback fails to improve LLM performance. This demonstrates that effective validation does not translate to effective improvement guidance for LLMs, establishing formal methods as essential for direct synthesis and a need for better iterative refinement methods in reliable AI-assisted hardware design.
BibTeX entry:
@inproceedings{WWD+26,
author = {Edward Wang and Joe Walston and Luca Daniel and Tony Tan and
Yoni Zohar and Clark W. Barrett},
editor = {Jyotirmoy V. Deshmukh and Klaus Havelund and Alessandro Pinto},
title = {Artificial Incorrectness: {SMT} and LLMs in Hardware Synthesis},
booktitle = {{NASA} Formal Methods - 18th International Symposium,
{NFM} 2026, Los Angeles, CA, USA, May 5-7, 2026, Proceedings},
series = {Lecture Notes in Computer Science},
volume = {16622},
pages = {261--284},
publisher = {Springer},
year = {2026},
doi = {10.1007/978-3-032-28079-4_12},
note = {Los Angeles, CA},
url = {https://doi.org/10.1007/978-3-032-28079-4_12}
}
(This webpage was created with bibtex2web.)