“Convex Bounds on the Softmax Function with Applications to Robustness Verification” by Dennis Wei, Haoze Wu, Min Wu, Pin-Yu Chen, Clark Barrett, and Eitan Farchi. In Proceedings of The 26^th International Conference on Artificial Intelligence and Statistics (AISTATS '23), (Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, eds.), Apr. 2023, pp. 6853-6878. Valencia, Spain.
The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.
BibTeX entry:
@inproceedings{WWW+23,
author = {Dennis Wei and Haoze Wu and Min Wu and Pin-Yu Chen and Clark
Barrett and Eitan Farchi},
editor = {Francisco Ruiz and Jennifer Dy and Jan-Willem van de Meent},
title = {Convex Bounds on the Softmax Function with Applications to
Robustness Verification},
booktitle = {Proceedings of The {\it 26^{th}} International Conference
on Artificial Intelligence and Statistics (AISTATS '23)},
series = {Proceedings of Machine Learning Research},
volume = {206},
pages = {6853--6878},
publisher = {PMLR},
month = apr,
year = {2023},
note = {Valencia, Spain},
url = {https://proceedings.mlr.press/v206/wei23c.html}
}
(This webpage was created with bibtex2web.)