Continuous Control¶
In this project, I use DDPG agent to solve the second version of the environment which contains 20 identical agents
Algorithm¶
Briefly speaking, DDPG is similar to DQN in many parts but better than it in continuous problem. Because DQN relies on action value function to find the best action, and hard to be applied in continuous cases since there are too many actions. Unlike DQN only use Q function, DDPG is a Actor-Critic method. It have an actor to performed the predicted best action, and a critic to tell whether the action performed by the actor is good.
Besides using Replay buffer and Local target network architecture which are adapted from DQN. DDPG also use other technics such as Soft update, rather than directly update target network at once, it update target network a little bit every time. And also adding noise to action, by doing so can encourage actor to explore.
I also use the training tricks suggested in the Benchmark Implementation like update the networks 10 times after every 20 timesteps. Although I tried gradient clipping too, the agent which did not using gradient clipping can solve the environment faster. So I decide not to use this.
Actor and Critic Model¶
The input layer of the actor model is the state size, and it return actions corresponding to the state in the output layer.
Actor(
(fc1): Linear(in_features=33, out_features=256, bias=True)
(fc2): Linear(in_features=256, out_features=128, bias=True)
(fc3): Linear(in_features=128, out_features=4, bias=True)
)The structure of the critic model is below, the input layer size is the state size; in the second hidden layer, we concatenate the output of the first hidden layer with the action performed by actor. And the return a Q-value in the output layer.
Critic(
(fcs1): Linear(in_features=33, out_features=400, bias=True)
(fc2): Linear(in_features=404, out_features=300, bias=True)
(fc3): Linear(in_features=300, out_features=1, bias=True)
)Hyper parameters¶
The hyperparameters I use are list below.
BUFFER_SIZE = int(1e5):replay buffer sizeBATCH_SIZE = 128:minibatch sizeGAMMA = 0.99:discount factorTAU = 1e-3:for soft update of target parametersLR_ACTOR = 1e-4:learning rate of the actorLR_CRITIC = 1e-4:learning rate of the criticWEIGHT_DECAY = 0:L2 weight decayLEARN_EVERY = 20:how often to learn from the experienceUPDATE_EVERY = 10:how often to update the target network
I found that in this case, the agent is really sensitive to the hyper parameters. For example, I set WEIGHT_DECAY to 0.0001 as the same to the DDPG in bipedal environment. But the agent turns out not able to learn, it even can't not get score higher than 2.0 over hundreds of episodes. So I think hyper parameters is one of the crucial points in this environment.
Another interesting point is in the OUNoise part. In the sample function, the original version is using random.random() to generate noise. However, when using this version, my agent cannot learn. It will stuck at the score around 1.0, like the situation mentioned above.
dx = self.theta * (self.mu - x) + self.sigma * np.array([random.random() for i in range(len(x))])
Like this plot
After trying other possible modifications, I found that switching to standard normal distribution can solve this problem. The possible explanation is that random.random() may generate some very big (close to 1) or very small (close to 0) value and cause the value of actions are unstable.
dx = self.theta * (self.mu - x) + self.sigma * np.random.standard_normal(self.size)
Result¶
the agent solve the environment at Episode 159. It sometimes float in a range from 120 ~ 160, I choose the most recent result to show here.
Watch the agent play.
Future Ideas¶
There are many other great algorithms to try, such as PPO, TRPO and D4PG... these algorithms are designed to fit continuous tasks. While implementing these algorithms, I can definitely learn a lot from it. So I plan to try them after I graduate this nanodegree.
1. Start the Environment¶
We begin by importing the necessary packages. If the code cell below returns an error, please revisit the project instructions to double-check that you have installed Unity ML-Agents and NumPy.
from unityagents import UnityEnvironment
import numpy as np
import matplotlib.pyplot as plt
Next, we will start the environment! Before running the code cell below, change the file_name parameter to match the location of the Unity environment that you downloaded.
- Mac:
"path/to/Reacher.app" - Windows (x86):
"path/to/Reacher_Windows_x86/Reacher.exe" - Windows (x86_64):
"path/to/Reacher_Windows_x86_64/Reacher.exe" - Linux (x86):
"path/to/Reacher_Linux/Reacher.x86" - Linux (x86_64):
"path/to/Reacher_Linux/Reacher.x86_64" - Linux (x86, headless):
"path/to/Reacher_Linux_NoVis/Reacher.x86" - Linux (x86_64, headless):
"path/to/Reacher_Linux_NoVis/Reacher.x86_64"
For instance, if you are using a Mac, then you downloaded Reacher.app. If this file is in the same folder as the notebook, then the line below should appear as follows:
env = UnityEnvironment(file_name="Reacher.app")env = UnityEnvironment(file_name='Reacher_Windows_x86_64/Reacher.exe')
Environments contain brains which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python.
# get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]
2. Examine the State and Action Spaces¶
In this environment, a double-jointed arm can move to target locations. A reward of +0.1 is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.
The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector must be a number between -1 and 1.
Run the code cell below to print some information about the environment.
# reset the environment
env_info = env.reset(train_mode=True)[brain_name]
# number of agents
num_agents = len(env_info.agents)
print('Number of agents:', num_agents)
# size of each action
action_size = brain.vector_action_space_size
print('Size of each action:', action_size)
# examine the state space
states = env_info.vector_observations
state_size = states.shape[1]
print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))
print('The state for the first agent looks like:', states[0])
3. Take Random Actions in the Environment¶
In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.
Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step. A window should pop up that allows you to observe the agent, as it moves through the environment.
Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!
env_info = env.reset(train_mode=False)[brain_name] # reset the environment
states = env_info.vector_observations # get the current state (for each agent)
scores = np.zeros(num_agents) # initialize the score (for each agent)
for t in range(10):
actions = np.random.randn(num_agents, action_size) # select an action (for each agent)
actions = np.clip(actions, -1, 1) # all actions between -1 and 1
env_info = env.step(actions)[brain_name] # send all actions to tne environment
next_states = env_info.vector_observations # get next state (for each agent)
rewards = env_info.rewards # get reward (for each agent)
dones = env_info.local_done # see if episode finished
scores += env_info.rewards # update the score (for each agent)
states = next_states # roll over states to next time step
if np.any(dones): # exit loop if episode finished
break
print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))
4. It's Your Turn!¶
Now it's your turn to train your own agent to solve the environment! When training the environment, set train_mode=True, so that the line for resetting the environment looks like the following:
env_info = env.reset(train_mode=True)[brain_name]
import torch
import torch.nn as nn
import torch.nn.functional as F
def hidden_init(layer):
fan_in = layer.weight.data.size()[0]
lim = 1. / np.sqrt(fan_in)
return (-lim, lim)
class Actor(nn.Module):
"""Actor (Policy) Model."""
def __init__(self, state_size, action_size, seed, fc1_units=256, fc2_units=128):
"""Initialize parameters and build model.
Params
======
state_size (int): Dimension of each state
action_size (int): Dimension of each action
seed (int): Random seed
fc1_units (int): Number of nodes in first hidden layer
fc2_units (int): Number of nodes in second hidden layer
"""
super(Actor, self).__init__()
self.seed = torch.manual_seed(seed)
self.fc1 = nn.Linear(state_size, fc1_units)
self.fc2 = nn.Linear(fc1_units, fc2_units)
self.fc3 = nn.Linear(fc2_units, action_size)
self.reset_parameters()
def reset_parameters(self):
self.fc1.weight.data.uniform_(*hidden_init(self.fc1))
self.fc2.weight.data.uniform_(*hidden_init(self.fc2))
self.fc3.weight.data.uniform_(-3e-3, 3e-3)
def forward(self, state):
"""Build an actor (policy) network that maps states -> actions."""
x = F.relu(self.fc1(state))
x = F.relu(self.fc2(x))
return F.tanh(self.fc3(x))
class Critic(nn.Module):
"""Critic (Value) Model."""
def __init__(self, state_size, action_size, seed, fcs1_units=256, fc2_units=128):
"""Initialize parameters and build model.
Params
======
state_size (int): Dimension of each state
action_size (int): Dimension of each action
seed (int): Random seed
fcs1_units (int): Number of nodes in the first hidden layer
fc2_units (int): Number of nodes in the second hidden layer
"""
super(Critic, self).__init__()
self.seed = torch.manual_seed(seed)
self.fcs1 = nn.Linear(state_size, fcs1_units)
self.fc2 = nn.Linear(fcs1_units+action_size, fc2_units)
self.fc3 = nn.Linear(fc2_units, 1)
self.reset_parameters()
def reset_parameters(self):
self.fcs1.weight.data.uniform_(*hidden_init(self.fcs1))
self.fc2.weight.data.uniform_(*hidden_init(self.fc2))
self.fc3.weight.data.uniform_(-3e-3, 3e-3)
def forward(self, state, action):
"""Build a critic (value) network that maps (state, action) pairs -> Q-values."""
xs = F.relu(self.fcs1(state))
x = torch.cat((xs, action), dim=1)
x = F.relu(self.fc2(x))
return self.fc3(x)
import random
import copy
from collections import namedtuple, deque
import torch.optim as optim
BUFFER_SIZE = int(1e5) # replay buffer size
BATCH_SIZE = 128 # minibatch size
GAMMA = 0.99 # discount factor
TAU = 1e-3 # for soft update of target parameters
LR_ACTOR = 1e-4 # learning rate of the actor
LR_CRITIC = 1e-4 # learning rate of the critic
WEIGHT_DECAY = 0 # L2 weight decay
LEARN_EVERY = 20 # how often to learn from the experience
UPDATE_EVERY = 10 # how often to update the target network
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
class Agent():
"""Interacts with and learns from the environment."""
def __init__(self, state_size, action_size, random_seed):
"""Initialize an Agent object.
Params
======
state_size (int): dimension of each state
action_size (int): dimension of each action
random_seed (int): random seed
"""
self.state_size = state_size
self.action_size = action_size
self.seed = random.seed(random_seed)
# Actor Network (w/ Target Network)
self.actor_local = Actor(state_size, action_size, random_seed).to(device)
self.actor_target = Actor(state_size, action_size, random_seed).to(device)
self.actor_optimizer = optim.Adam(self.actor_local.parameters(), lr=LR_ACTOR)
# Critic Network (w/ Target Network)
self.critic_local = Critic(state_size, action_size, random_seed).to(device)
self.critic_target = Critic(state_size, action_size, random_seed).to(device)
self.critic_optimizer = optim.Adam(self.critic_local.parameters(), lr=LR_CRITIC, weight_decay=WEIGHT_DECAY)
# Noise process
self.noise = OUNoise(action_size, random_seed)
# Replay memory
self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, random_seed)
self.lt_step = 0
def step(self, states, actions, rewards, next_states, dones):
"""Save experience in replay memory, and use random sample from buffer to learn."""
# Save experience / reward
for state, action, reward, next_state, done in zip(states, actions, rewards, next_states, dones):
self.memory.add(state, action, reward, next_state, done)
# Learn every LEARN_EVERY times
self.lt_step = (self.lt_step + 1) % LEARN_EVERY
if self.lt_step == 0:
# Learn, if enough samples are available in memory
if len(self.memory) > BATCH_SIZE:
# update network UPDATE_EVERY times
for t in range(UPDATE_EVERY):
experiences = self.memory.sample()
self.learn(experiences, GAMMA)
def act(self, state, add_noise=True):
"""Returns actions for given state as per current policy."""
state = torch.from_numpy(state).float().to(device)
self.actor_local.eval()
with torch.no_grad():
action = self.actor_local(state).cpu().data.numpy()
self.actor_local.train()
if add_noise:
action += self.noise.sample()
return np.clip(action, -1, 1)
def reset(self):
self.noise.reset()
def learn(self, experiences, gamma):
"""Update policy and value parameters using given batch of experience tuples.
Q_targets = r + γ * critic_target(next_state, actor_target(next_state))
where:
actor_target(state) -> action
critic_target(state, action) -> Q-value
Params
======
experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples
gamma (float): discount factor
"""
states, actions, rewards, next_states, dones = experiences
# ---------------------------- update critic ---------------------------- #
# Get predicted next-state actions and Q values from target models
actions_next = self.actor_target(next_states)
Q_targets_next = self.critic_target(next_states, actions_next)
# Compute Q targets for current states (y_i)
Q_targets = rewards + (gamma * Q_targets_next * (1 - dones))
# Compute critic loss
Q_expected = self.critic_local(states, actions)
critic_loss = F.mse_loss(Q_expected, Q_targets)
# Minimize the loss
self.critic_optimizer.zero_grad()
critic_loss.backward()
self.critic_optimizer.step()
# ---------------------------- update actor ---------------------------- #
# Compute actor loss
actions_pred = self.actor_local(states)
actor_loss = -self.critic_local(states, actions_pred).mean()
# Minimize the loss
self.actor_optimizer.zero_grad()
actor_loss.backward()
self.actor_optimizer.step()
# ----------------------- update target networks ----------------------- #
self.soft_update(self.critic_local, self.critic_target, TAU)
self.soft_update(self.actor_local, self.actor_target, TAU)
def soft_update(self, local_model, target_model, tau):
"""Soft update model parameters.
θ_target = τ*θ_local + (1 - τ)*θ_target
Params
======
local_model: PyTorch model (weights will be copied from)
target_model: PyTorch model (weights will be copied to)
tau (float): interpolation parameter
"""
for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):
target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)
class OUNoise:
"""Ornstein-Uhlenbeck process."""
def __init__(self, size, seed, mu=0., theta=0.15, sigma=0.2):
"""Initialize parameters and noise process."""
self.mu = mu * np.ones(size)
self.theta = theta
self.sigma = sigma
self.seed = random.seed(seed)
self.size = size
self.reset()
def reset(self):
"""Reset the internal state (= noise) to mean (mu)."""
self.state = copy.copy(self.mu)
def sample(self):
"""Update internal state and return it as a noise sample."""
x = self.state
#dx = self.theta * (self.mu - x) + self.sigma * np.array([random.random() for i in range(len(x))])
dx = self.theta * (self.mu - x) + self.sigma * np.random.standard_normal(self.size)
self.state = x + dx
return self.state
class ReplayBuffer:
"""Fixed-size buffer to store experience tuples."""
def __init__(self, action_size, buffer_size, batch_size, seed):
"""Initialize a ReplayBuffer object.
Params
======
buffer_size (int): maximum size of buffer
batch_size (int): size of each training batch
"""
self.action_size = action_size
self.memory = deque(maxlen=buffer_size) # internal memory (deque)
self.batch_size = batch_size
self.experience = namedtuple("Experience", field_names=["state", "action", "reward", "next_state", "done"])
self.seed = random.seed(seed)
def add(self, state, action, reward, next_state, done):
"""Add a new experience to memory."""
e = self.experience(state, action, reward, next_state, done)
self.memory.append(e)
def sample(self):
"""Randomly sample a batch of experiences from memory."""
experiences = random.sample(self.memory, k=self.batch_size)
states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)
actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).float().to(device)
rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)
next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)
dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)
return (states, actions, rewards, next_states, dones)
def __len__(self):
"""Return the current size of internal memory."""
return len(self.memory)
agent = Agent(state_size=state_size, action_size=action_size, random_seed=10)
def run_training(n_episodes=300, max_t=1400):
scores_deque = deque(maxlen=100)
scores = []
solved = False
for i_episode in range(1, n_episodes+1):
env_info = env.reset(train_mode=True)[brain_name]
states = env_info.vector_observations
score = 0
agent.reset()
while True:
actions = agent.act(states)
env_info = env.step(actions)[brain_name] # send all actions to the environment
next_states = env_info.vector_observations # get next state (for each agent)
rewards = np.array(env_info.rewards) # get reward (for each agent)
dones = env_info.local_done
agent.step(states, actions, rewards, next_states, dones)
states = next_states
score += np.mean(rewards)
if any(dones):
break
scores_deque.append(score)
scores.append(score)
print('\rEpisode {}\tAverage Score: {:.2f}\tScore: {:.2f}'.format(i_episode, np.mean(scores_deque), score), end="")
if i_episode % 100 == 0:
torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')
torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')
print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))
if np.mean(scores_deque) >= 30 and solved == False:
solved = True
print('\rSolved at Episode {} !\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))
return scores
scores = run_training()
import pandas as pd
fig = plt.figure(figsize=(16,5))
ax = fig.add_subplot(121)
plt.plot(np.arange(1, len(scores)+1), scores)
plt.axhline(y=30.0, color='r', linestyle='-')
plt.ylabel('Score')
plt.xlabel('Episode #')
ay = fig.add_subplot(122)
plt.plot(np.arange(1, len(scores)+1), pd.Series(scores).rolling(100).mean())
plt.axhline(y=30.0, color='r', linestyle='-')
plt.ylabel('Average Score')
plt.xlabel('Episode #')
plt.show()
Watch Agent¶
agent = Agent(state_size=state_size, action_size=action_size, random_seed=10)
agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))
agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))
for t in range(1):
env_info = env.reset(train_mode=False)[brain_name] # reset the environment
states = env_info.vector_observations # get the current state (for each agent)
scores = np.zeros(num_agents) # initialize the score (for each agent)
while True:
actions = agent.act(states)
env_info = env.step(actions)[brain_name] # send all actions to tne environment
next_states = env_info.vector_observations # get next state (for each agent)
rewards = env_info.rewards # get reward (for each agent)
dones = env_info.local_done # see if episode finished
scores += env_info.rewards # update the score (for each agent)
states = next_states # roll over states to next time step
if np.any(dones): # exit loop if episode finished
break
print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))
env.close()