text: 363,85,1171385645920595968,180451291,Uri Shalit,"In a new paper with Shie Mannor and led by @guytenn, we work at the intersection of RL and causal inference. We give a new method to perform off-policy evaluation in the presence of dynamic unmeasured confounders, or a POMDP to use RL terminology.  <LINK> <LINK> Imagine learning a policy offline from hospital data. We know the actions we learn from were conducted by doctors who have some information our RL agent can’t see: unmeasured confounders. Gottesman et al. <LINK> showed how bad things can go in this scenario. In our paper, we give a way to evaluate agent policies in this difficult scenario. Main assumption: the unknown cond. prob. matrix between unmeasured and measured is invertible, as are the transition matrices btw unmeasured. Inspired by Miao et al. <LINK> While we believe this is an important theoretical advance, the resulting method requires estimating inverses of big matrices. So we introduce a “decoupled POMDP” model which we consider sensible in its causal assumptions and leads to an easier estimation problem. <LINK> In the figures: u_t are unobserved confounders/states, z_t and o_t are observations, a_t are actions, r_t rewards, π_b behavior policy (e.g. human doctor), π_e agent policy we want to evaluate",https://arxiv.org/abs/1909.03739,"This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Markov Decision Processes (POMDPs) and establish what we believe is the first off-policy evaluation result for POMDPs. In addition, we formulate a model in which observed and unobserved variables are decoupled into two dynamic processes, called a Decoupled POMDP. We show how off-policy evaluation can be performed under this new model, mitigating estimation errors inherent to general POMDPs. We demonstrate the pitfalls of off-policy evaluation in POMDPs using a well-known off-policy method, Importance Sampling, and compare it with our result on synthetic medical data. ",Off-Policy Evaluation in Partially Observable Environments,5,"['In a new paper with Shie Mannor and led by @guytenn, we work at the intersection of RL and causal inference. We give a new method to perform off-policy evaluation in the presence of dynamic unmeasured confounders, or a POMDP to use RL terminology.  \n<LINK> <LINK>', 'Imagine learning a policy offline from hospital data. We know the actions we learn from were conducted by doctors who have some information our RL agent can’t see: unmeasured confounders. Gottesman et al. https://t.co/IIEH1xZrcd showed how bad things can go in this scenario.', 'In our paper, we give a way to evaluate agent policies in this difficult scenario. \nMain assumption: the unknown cond. prob. matrix between unmeasured and measured is invertible, as are the transition matrices btw unmeasured. Inspired by Miao et al. \nhttps://t.co/mgz65P9uxM', 'While we believe this is an important theoretical advance, the resulting method requires estimating inverses of big matrices. So we introduce a “decoupled POMDP” model which we consider sensible in its causal assumptions and leads to an easier estimation problem. https://t.co/52snAPT6h3', 'In the figures: u_t are unobserved confounders/states, z_t and o_t are observations, a_t are actions, r_t rewards, π_b behavior policy (e.g. human doctor), π_e agent policy we want to evaluate']",19,09,1241