Approximating Probability Distributions With Short Vectors, Via Information Theoretic Distance Measures

Given a probability distribution p = (p(1),...,p(n)) and an integer m < n, what is the probability distribution q = (q(1),...,q(m)) that is "the closest" to p, that is, that best approximates p ? It is clear that the answer depends on the function one chooses to evaluate the goodness of the approximation. In this paper we provide a general criterion to approximate p with a shorter vector q by using ideas from majorization theory. We evaluate the goodness of our approximation by means of a variety of information theoretic distance measures.