Weight Uncertainty in Transformer Network for the Traveling Salesman Problem

Traveling Salesman Problem (TSP) and its variations have many applications, especially for the case of circuit design where the ordering is crucial for the final design quality. Recently the deep learning (DL) based solvers were found successful in solving TSP, and have been applied to solve practical circuit design problems. However as the problem size increases, the gap between the optimal solution and the one predicted by DL based solver increases. For practical applications where the following decision making strongly relies on the solver quality, knowing the model uncertainly is very important. In the present work, a generalized spike and slab distribution (gSaS) was introduced to model the weight uncertainty for the attention based neural network TSP solver. Instead of minimizing the variational free energy or the expected lower bound on the marginal likelihood in the traditional variational inference, we directly minimize the loss function of the original model with respected to the hyper parameters of gSaS. Numerical experiments indicate that our proposed method can predict reasonable model uncertainty without sacrificing any test accuracy.