Deep Deterministic Policy Gradient (DDPG)

by ComputingNotes
Reinforcement Learning

DDPG learns Q-function and a policy concurrently. Off-policy data and bellman equation are used to learn Q-function and then Q-function to lean policy.

Q-function
A Policy
Learn
concurrently
Deep Deterministic Policy Gradient
(DDPG)
off-policy
data
+
Bellman
Equation
learn
Q-function
Learn
Policy

DDGA is considered off-policy algorithm and is specially for continuous action spaces.

Off-Policy Algorithm Continuous Action Spaces
  • Learns a Q-function and a policy concurrently.
  • Uses off-policy data and the Bellman equation to learn the Q-function.
  • The learned Q-function is subsequently used to learn and optimize the policy.

Key Equations: DDPG

(Learning a Q-function & learning a policy)

Bellman Equation
optimal action value function $Q^*(s,a)$
$$ \mathbb{E}_{s’ \sim P}\left[r(s,a) + \gamma \max_{a’} Q^*(s’, a’)\right] $$
↳ $s’ \sim P \Rightarrow$ next state, $s’$, sampled from environment distribution $P(\cdot | s, a)$
Mean-squared Bellman error (MSBE)
$$ L(\phi, D) = \mathbb{E}_{(s,a,r,s’,d) \sim D} \left[ \left( Q_{\phi}(s,a) – \left( r + \gamma (1-d) \max_{a’} Q_{\phi}(s’, a’) \right) \right)^2 \right] $$
↓ This function tells how closely $Q_\phi$ satisfies Bellman Equation.
↓ neural network $Q_\phi(s,a)$ with parameters $\phi$
• $D \rightarrow$ set of previous experiences
↘ $(r + \gamma(1-d)…) \rightarrow$ target
Introducing target networks
(delayed target network with parameters $\phi_{\text{targ}}$)
$$ \text{target} \Rightarrow y(r,s’,d) = r + \gamma(1-d) \max_{a’} Q_{\phi_{\text{targ}}}(s’,a’) $$
$$ \phi_{\text{targ}} \leftarrow \rho \phi_{\text{targ}} + (1-\rho)\phi $$
↱ updated smoothly over time via polyak averaging
Approximating Max over Actions
$$ \max_{a’} Q_{\phi_{\text{targ}}}(s’, a’) \approx Q_{\phi_{\text{targ}}}(s’, \mu_{\theta_{\text{targ}}}(s’)) $$
• (computing $\max_{a’} \rightarrow$ computationally intractable in continuous action spaces)
↳ resolve this issue by using target policy network ($\mu_{\theta_{\text{targ}}}$)
Final DDPG Q-Learning loss
$$ = \mathbb{E}_{(s,a,r,s’,d) \sim D} \left[ \left( Q_\phi(s,a) – \left(r + \gamma(1-d) Q_{\phi_{\text{targ}}}(s’, \mu_{\theta_{\text{targ}}}(s’)) \right) \right)^2 \right] $$
↓ $\mu_{\theta_{\text{targ}}} \rightarrow$ target policy

DDPG Policy Learning

↳ Learn deterministic policy $\mu_\theta(s)$
↓ gives
Action
$Q_\phi(s,a)$ ← maximize —
Action space $\rightarrow$ continuous $\Rightarrow$ Q-Function $\rightarrow$ differentiable w.r.t. action
To solve,
$$ \max_{\theta} \mathbb{E}_{s \sim D} \left[ Q_\phi(s, \mu_\theta(s)) \right] $$
← perform gradient ascent w.r.t. policy parameters

References

‌

Views: 10
error: