Is the SAFE Approach Too Simple for App Feature Extraction? A Replication Study.

[Context and motivation] Automatic extraction and analysis of app features from user reviews is helpful for software developers to better understand users perceptions of delivered app features. Recently, a rule-based approach called safe was proposed to automatically extract app features from user reviews. safe was reported to obtain superior performance in terms of precision and recall over previously proposed techniques. However, the procedure used to evaluate safe was in part subjective and not repeatable and thus the whole evaluation might not be reliable. [Question/problem] The goal of our study is to perform an external replication of the safe evaluation using an objective and repeatable approach. [Principal ideas/results] To this end, we first implemented safe and checked the correctness of our implementation on the set of app descriptions that were used and published by the authors of the original study. We applied our safe implementation to eight review datasets (six app review datasets, one laptop review dataset, one restaurant review dataset) and evaluated its performance against manually annotated feature terms. Our results suggest that the precision of the safe approach is strongly influenced by the density of the annotated app features in a review dataset. Overall, we obtained an average precision and recall of 0.120 and 0.539, respectively which is lower than the performance reported in the original safe study. [Contribution] We performed an unbiased and reproducible evaluation of the safe approach for user reviews. We make our implementation and all datasets used for the evaluation available for replication by others.