Efficient Discovery of Functional Dependencies on Massive Data

Functional dependencies are the most common constraints in the design theory for relational databases, which have very important practical applications in many areas. Different kinds of algorithms are proposed to discover the functional dependencies. However, it is found in this paper that the existing algorithms cannot deal well with massive data which cannot be held entirely in memory due to high memory consumption and high computation cost. In this paper, a novel algorithm FSC is presented to compute functional dependencies on massive data. The two-step execution of FSC relies on a pre-computed update-friendly assistant structure of comparable pairs which reflect the identifier pairs for tuples with at least one equal attribute. In step 1, FSC determines the violated functional dependencies and introduces the selective comparison by comparable pairs to reduce the required pairwise comparison significantly. The direct value-combination compression strategy is devised to process attributes of small cardinality. In step 2, FSC induces the required functional dependencies by the results in step 1. The extensive experimental results, conducted on synthetic and real-life data sets, show that FSC can discover functional dependencies on massive data efficiently.