On Comprehension of Genetic Programming Solutions: A Controlled Experiment on Semantic Inference

Applied to the problem of automatic program generation, Genetic Programming often produces code bloat, or unexpected solutions that are, according to common belief, difficult to comprehend. To study the comprehensibility of the code produced by Genetic Programming, attribute grammars obtained by Genetic Programming-based semantic inference were compared to manually written ones. According to the established procedure, the research was carried out as a controlled classroom experiment that involved two groups of students from two universities, and consisted of a background questionnaire, two tests and a feedback questionnaire after each test. The tasks included in the tests required the identification of various properties of attributes and grammars, the identification of the correct attribute grammar from a list of choices, or correcting a semantic rule in an attribute grammar. It was established that solutions automatically generated by Genetic Programming in the field of semantic inference, in this study attribute grammars, are indeed significantly harder to comprehend than manually written ones. This finding holds, regardless of whether comprehension correctness, i.e., how many attribute grammars were correctly comprehended, or comprehension efficiency is considered, i.e., how quickly attribute grammars were correctly comprehended.