In partially observable Markov decision processes (POMDPs), the optimal policy generally depends on the history of observations and past actions. Asymmetric actor-critic methods have become a popular method to learn such policies when additional information, such as the state, is available during training. The critic, which is not used at execution, has access to the state during training. In a partially observable environment, a critic taking only the state as input is generally ill-defined or provides biased policy gradients. Alternatively, a critic taking both the state and the history as input is well-defined and provides unbiased policy gradients. In this work, we show that a critic taking the state and the hidden state of the recurrent actor is sufficient to be well-defined and to provide unbiased policy gradients, while preventing the use of a second recurrent approximator of the history. We call it the memory-state critic. Even if it is a lossy representation of the history, we show that we do not need to backpropagate the critic loss past the hidden state. We evaluate the memory-state critic on a vision-based pursuit-evasion environment between two quadrotors with two types of arenas. The pursuer is the learning agent, and the evader is sampled per episode from a fixed pool of heuristic behaviors. The results show that the memory-state critic outperforms the history-state critic and converges faster. In addition to being unbiased compared to the state-only critic, it also marginally outperforms it in the wall arena, where the actor’s history carries information that the privileged state alone does not.