An eye tracking study assessing the impact of background styling in code editors on novice programmers' code understanding

Background and Context: The designers of programming editors aimed at learners have long experimented with different styles of code presentation. The idea of syntax highlighting - coloring specific words - is very old. More recently, some editors (including text-, frame- and block-based editors) have added forms of scope highlighting - colored rectangles to represent programming scope - but there have been few studies to investigate whether this is beneficial for novices when reading and understanding program code. Objectives: We investigated whether the use of scope highlighting during code comprehension tasks (a) has an impact on where users focus their gaze, (b) affects the accuracy of user's responses to tasks, and/or (c) affects the speed of user's correct responses to the tasks. Method: We conducted a controlled trial with a crossover design, where all participants completed the same twelve code comprehension tasks, each performed in one of three scope highlighting conditions. The conditions were (a) "plain" Java with no scope highlighting, (b) Java with BlueJ's scope highlighting, and (c) Stride, a frame-based language very similar to Java. We used a combination of eye-tracking hardware and video observation to record where users were looking in the code, the time it took them to answer the task, and the graded correctness of their answer. The data comes from students at two institutions in different countries. Findings: We found that there was no difference between plain Java, highlighted Java or Stride in terms of correctness, or speed to produce correct answers - and this was unaffected by whether users had seen the latter two interfaces before. There was a difference in eye gaze behavior, especially between Stride and the two Java interfaces. Participants' gaze moved around less, they had shorter saccade lengths, longer fixation durations, fewer regressions, and less line coverage in Stride, which could imply a higher cognitive load. Regardless of condition, all participants who answered correctly read buggy lines toward the end of the task. Implications: The highlighting conditions showed a difference in low-level gaze behavior, but this did not translate into a difference at the higher level of program comprehension task performance. The gaze behavior difference could be due to the novelty of the design of the Stride interface other than the scope highlighting (which showed no such difference when present/not present in the Java display). However, it seems that large changes in the presentation of the program code have only a small (if any) effect on task performance.