SILO: Optimization over Trained Neural Networks: Going Large with Gradient-Based Algorithms

Abstract

When optimizing a nonlinear objective, one can employ a neural network as a surrogate for the nonlinear function. However, the resulting optimization model can be time-consuming to solve globally with exact methods. As a result, local search that exploits the neural-network structure has been employed to find good solutions within a reasonable time limit. For such methods, a lower per-iteration cost is advantageous when solving larger models. The contribution of this paper is two-fold. First, we propose a gradient-based algorithm with lower per-iteration cost than existing methods. Second, we further adapt this algorithm to exploit the piecewise-linear structure of neural networks that use Rectified Linear Units (ReLUs). In line with prior research, our methods become competitive with — and then dominant over — other local search methods as the optimization models become larger.

Joint work with Jiatai Tong (Northwestern), Yilin Zhu and Sam Burer (University of Iowa)

Bio

Thiago Serra is an assistant professor of business analytics at the University of Iowa. His scholarship focuses on the theory, practice, and integration of machine learning and mathematical optimization. Previously, he was an assistant professor at Bucknell University, a visiting research scientist at Mitsubishi Electric Research Labs, and an operations research analyst at Petrobras. He has a Ph.D. in operations research from Carnegie Mellon University, from which he received the Gerald L. Thompson Doctoral Dissertation Award in Management Science. He also has a master’s degree in computer science from the University of Sao Paulo (USP) and a computer engineering degree from the University of Campinas (Unicamp). He has served the INFORMS Computing Society as vice chair (2022-2023) and chair (2024-2025). He also serves as an associate editor for the journals INFORMS Journal on Computing and INFORMS Journal on Data Science.