Data provenance in multi-agent systems: relevance, benefits and research opportunities.

The popularity of applications based on artificial intelligence creates the need for making them able to explain their behaviour and be accountable for their decisions. This is a challenge mainly if applications are distributed, being composed of multiple autonomous agents, forming a Multi-Agent System (MAS). A key means of making these systems explainable is to track agent behaviour, that is, to record the provenance of their actions and reasoning. Although the idea of provenance has been explored in some contexts, it has been little explored in the context of MAS, leaving many open issues that must be understood and addressed. Our goal in this paper is to make a case for the importance of the data provenance to MAS, discussing what questions can be answered regarding MAS behaviour using provenance and, with a case study, demonstrating the benefits that provenance provides to answer these questions. This study involves the use of a framework, namely FProvW3C, which collects and stores the provenance of d...