The Complexity of the Co-occurrence Problem

Let S be a string of length n over an alphabet $$\varSigma $$ and let Q be a subset of $$\varSigma $$ of size $$q \ge 2$$ . The co-occurrence problem is to construct a compact data structure that supports the following query: given an integer w return the number of length-w substrings of S that contain each character of Q at least once. This is a natural string problem with applications to, e.g., data mining, natural language processing, and DNA analysis. The state of the art is an $$O(\sqrt{nq})$$ space data structure that—with some minor additions—supports queries in $$O(\log \log n)$$ time [CPM 2021]. Our contributions are as follows. Firstly, we analyze the problem in terms of a new, natural parameter d, giving a simple data structure that uses O(d) space and supports queries in $$O(\log \log n)$$ time. The preprocessing algorithm does a single pass over S, runs in expected O(n) time, and uses $$O(d + q)$$ space in addition to the input. Furthermore, we show that O(d) space is optimal and that $$O(\log \log n)$$ -time queries are optimal given optimal space. Secondly, we bound $$d = O(\sqrt{nq})$$ , giving clean bounds in terms of n and q that match the state of the art. Furthermore, we prove that $$\varOmega (\sqrt{nq})$$ bits of space is necessary in the worst case, meaning that the $$O(\sqrt{nq})$$ upper bound is tight to within polylogarithmic factors. All of our results are based on simple and intuitive combinatorial ideas that simplify the state of the art.