Leveraging Documentation to Test Deep Learning Library Functions

It is integral to test API functions of widely-used deep learning (DL) libraries. The effectiveness of such testing requires DL-specific input constraints of these API functions. Such constraints enable the generation of valid inputs, i.e., inputs that follow these DL-specific constraints, to explore deep to test the core functionality of API functions. Existing fuzzers have no knowledge of such constraints, and existing constraint-extraction techniques are ineffective for extracting DL-specific input constraints. To fill this gap, we design and implement a document-guided fuzzing technique—D2C—for API functions of DL libraries. D2C leverages sequential pattern mining to generate rules for extracting DL-specific constraints from API documents and uses these constraints to guide the fuzzing to generate valid inputs automatically. D2C also generates inputs that violate these constraints to test the input validity checking code. In addition, D2C uses the constraints to generate boundary inputs to detect more bugs. Our evaluation of three popular DL libraries (TensorFlow, PyTorch, and MXNet) shows that D2C’s accuracy in extracting input constraints is 83.3–90.0%. D2C detects 121 bugs, while a baseline fuzzer without input constraints detects only 68 bugs. Most (89) of the 121 bugs are previously unknown, 54 of which have been fixed or confirmed by developers after we report them. In addition, D2C detects 38 inconsistencies within documents, including 28 that are fixed or confirmed after we report them.