MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
CoRR(2024)
摘要
This paper addresses the challenge of optimizing meta-parameters (i.e.,
hyperparameters) in machine learning algorithms, a critical factor influencing
training efficiency and model performance. Moving away from the computationally
expensive traditional meta-parameter search methods, we introduce MetaOptimize
framework that dynamically adjusts meta-parameters, particularly step sizes
(also known as learning rates), during training. More specifically,
MetaOptimize can wrap around any first-order optimization algorithm, tuning
step sizes on the fly to minimize a specific form of regret that accounts for
long-term effect of step sizes on training, through a discounted sum of future
losses. We also introduce low complexity variants of MetaOptimize that, in
conjunction with its adaptability to multiple optimization algorithms,
demonstrate performance competitive to those of best hand-crafted learning rate
schedules across various machine learning applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要