Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies

We consider off-policy evaluation (OPE) of deterministic target policies forreinforcement learning (RL) in environments with continuous action spaces.While it is common to use importance sampling for OPE, it suffers from highvariance when the behavior policy deviates significantly from the targetpolicy. In order to address this issue, some recent works on OPE proposedin-sample learning with importance resampling. Yet, these approaches are notapplicable to deterministic target policies for continuous action spaces. Toaddress this limitation, we propose to relax the deterministic target policyusing a kernel and learn the kernel metrics that minimize the overall meansquared error of the estimated temporal difference update vector of an actionvalue function, where the action value function is used for policy evaluation.We derive the bias and variance of the estimation error due to this relaxationand provide analytic solutions for the optimal kernel metric. In empiricalstudies using various test domains, we show that the OPE with in-samplelearning using the kernel with optimized metric achieves significantly improvedaccuracy than other baselines.