Actor–critic models

Lens adapted to this topic: How the architecture works and where it is useful

The standard account treats actor–critic methods as a broad reinforcement-learning design that combines policy optimization with value estimation. The actor represents a policy and the critic estimates value, advantage, or action value, often through temporal-difference learning. This can provide lower-variance, bootstrapped feedback than basic policy-gradient methods and supports multiple action-space and learning configurations. The approach is established and useful, but its practical benefits are problem- and algorithm-dependent.

• Actor–critic methods combine policy-based and value-based reinforcement learning: the actor selects actions through a policy, and the critic evaluates them using a value-related estimate.
• A critic can act as a learned baseline or bootstrapping mechanism, helping address the high variance and sample inefficiency associated with Monte Carlo policy-gradient methods such as REINFORCE.
• The family includes on-policy and off-policy methods and can address discrete or continuous action spaces.
• Actor–critic methods have been developed for model-free and model-based settings, including continuous-time and continuous-space control.
• Actor–critic is not automatically superior to newer methods: one practical account notes that conventional AC agents do not generally have functional advantages over methods such as PPO.

Lens adapted to this topic: Where standard actor–critic assumptions can fail or fall short

The critical research account emphasizes that “actor–critic” covers algorithms whose updates need not equal the true policy gradient, especially with approximate or imperfect critics. It treats the actor–critic interaction as a source of bias, instability, or theoretical mismatch rather than only as a variance-reduction benefit. Research on corrected and experience-replay variants suggests that performance can improve, but results depend on the environment, approximation choices, and correction or stabilization techniques.

• The practical actor–critic gradient can differ from the true policy gradient of the cumulative-reward objective; the size and structure of this gap require explicit analysis.
• Because the critic is learned and approximate, the actor may be updated in a direction that is only an approximation to policy improvement, making critic representation and training dynamics important.
• The actor and critic can be viewed as interacting players in a Stackelberg-style game, offering a framework for analyzing corrections to standard actor–critic updates.
• Corrected variants such as Residual Actor–Critic and Stackelberg Actor–Critic were reported to improve sample efficiency and final performance in experiments on tabular and continuous environments.
• Experience replay and additional mechanisms such as importance-sampling corrections and trust-region optimization can make deep actor–critic agents more sample-efficient and stable, but these additions also increase algorithmic complexity.