In this post, I will be talking about policy optimization methods from the papers Trust Region Policy Optimization (Schulman et al. 2015) and Proximal Policy Optimization Algorithms (Schulman et al. 2017). …

In this post, I will be exploring the concepts following the paper Deterministic Policy Gradient Algorithms (Silver et al.), implementing the algorithm COPDAC (Compatible Off-Policy Deterministic Actor-Critic) proposed in the paper, and training the agent on the continuous-control environment MountainCar. Introduction In the paper Deterministic Policy Gradient Algorithms, Silver proposes a…

In this post, I will be exploring the ideas behind the paper Off-Policy Actor-Critic (Degris et al.) submitted to the ICML 2012. The paper presents an Actor-Critic method that is fully online and uses a behavior policy to learn the target policy. This off-policy learning method takes advantage of action…

In the past few Policy Gradient and Actor-Critic algorithms I’ve implemented, I’ve been using the classical control environment, CartPole, as my experiment benchmark. The CartPole environment has a simple discrete action space: pushing the cart left or right. …

In this post, I’ll be implementing some Actor-Critic methods using the policy gradients methods and value function approximations from my previous posts. I won’t focus too much on the theory and details but more so on combining previous ideas of Actor and Critic. Actor-Critic Algorithms First, let’s look at the One-step Actor-Critic…

In my previous post, I discussed a way to reduce variance by using the generalized policy update equation, which is derived from the policy gradient theorem, and added a learned value function as a baseline. …

After an introduction to the REINFORCE algorithm, I wanted to explore a little bit further this simple algorithm derived from the policy gradient theorem. In this post, I will be implementing REINFORCE with baseline and some small modifications and testing it out on the CartPole environment. …

In the previous posts, I have been working on a form of Reinforcement learning, Q learning, where the agent finds an optimal policy that maximizes the total reward over a trajectory of states. I later expanded on Q learning with deep neural networks and explored the implementation of DQN. In…

In my previous post A First Look at Reinforcement Learning, I attempted to use Deep Q learning to solve the CartPole problem. In this post, I will be further exploring Deep Q learning but in the context of Atari games. In 2013, the paper by the Deepmind team Playing Atari…