A camera-based solution for customer behavior identification

Nowaday, automatic or semi-automatic sale becomes the competitive advantage of retail or producer. In particular, understanding customer behaviors can help improve any vending system. In this work, we focus on finding a solution supporting automatic sales from shelves, that can detect what customers want. We try to set up a hypothetical system and propose a camera-based solution that helps identify customer activity. First, the layout of shelf is determined. Then, our system detects and tracks customer hands. While tracking, hand status (holding or non-holding) are also verified to see if the customer take something. Finally, based on hand's status changes, the system know if the customer takes or returns a product. Further, with a camera-based solution, we would like the proposed solution can run on a machine with a modest configure. In our solution, we choose tiny-YoloV3 for hand detection and MobileNetV2 for hand status classification. We evaluated each sub-module on available datasets. The overall performance of the system is manually evaluated on a set of sessions from different customers. We obtained promising results (84% in terms of accuracy) on the test set.