“Parameterized Abstract Interpretation for Transformer Verification” by Pei Huang, Dennis Wei, Omri Isac, Haoze Wu, Min Wu, and Clark W. Barrett. In Fortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20-27, 2026, (Sven Koenig, Chad Jenkins, and Matthew E. Taylor, eds.), Jan. 2026, pp. 35500-35508.
Transformers based on the self-attention mechanism have become foundational models across a wide range of domains, thereby creating an urgent need for effective formal verification techniques to better understand their behavior and ensure safety guarantees. In this paper, we propose two parameterized linear abstract domains for the inner products in the self-attention module, aiming to improve verification precision. The first one constructs symbolic quadratic upper and lower bounds for the product of two scalars, and then derives parameterized affine bounds using tangents. The other one constructs parameterized bounds by interpolating affine bounds proposed in prior work. We evaluate these two parameterization methods and demonstrate that both of them outperform the state-of-the-art approach which is regarded as optimal with respect to a certain mean gap. Experimental results show that, in the context of robustness verification, our approach is able to verify many instances that cannot be verified by existing methods. In the interval analysis, our method achieves tighter results compared to the SOTA, with the strength becoming more pronounced as the network depth increases.
BibTeX entry:
@inproceedings{HWI+26,
author = {Pei Huang and Dennis Wei and Omri Isac and Haoze Wu and Min
Wu and Clark W. Barrett},
editor = {Sven Koenig and Chad Jenkins and Matthew E. Taylor},
title = {Parameterized Abstract Interpretation for Transformer
Verification},
booktitle = {Fortieth {AAAI} Conference on Artificial Intelligence,
Thirty-Eighth Conference on Innovative Applications of Artificial
Intelligence, Sixteenth Symposium on Educational Advances in
Artificial Intelligence, {AAAI} 2026, Singapore, January 20-27,
2026},
pages = {35500--35508},
publisher = {{AAAI} Press},
month = jan,
year = {2026},
doi = {10.1609/AAAI.V40I42.40860},
url = {https://doi.org/10.1609/aaai.v40i42.40860}
}
(This webpage was created with bibtex2web.)