The importance of feature preprocessing for differentially private linear optimization
ICLR 2024(2023)
摘要
Training machine learning models with differential privacy (DP) has received
increasing interest in recent years. One of the most popular algorithms for
training differentially private models is differentially private stochastic
gradient descent (DPSGD) and its variants, where at each step gradients are
clipped and combined with some noise. Given the increasing usage of DPSGD, we
ask the question: is DPSGD alone sufficient to find a good minimizer for every
dataset under privacy constraints? Towards answering this question, we show
that even for the simple case of linear classification, unlike non-private
optimization, (private) feature preprocessing is vital for differentially
private optimization. In detail, we first show theoretically that there exists
an example where without feature preprocessing, DPSGD incurs an optimality gap
proportional to the maximum Euclidean norm of features over all samples. We
then propose an algorithm called DPSGD-F, which combines DPSGD with feature
preprocessing and prove that for classification tasks, it incurs an optimality
gap proportional to the diameter of the features max_x, x' ∈ Dx -
x'_2. We finally demonstrate the practicality of our algorithm on image
classification benchmarks.
更多查看译文
关键词
private optimization,feature preprocessing,differential privacy
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要