Algorithms

Choose a workflow first: Engine algorithms emit training records while standalone solvers own their traversal. Python always owns neural-network training. In the search rows, AlwaysResample, Committed, and ExpandAll name handles from rf.chance_modes; DQN and solver rows describe their own chance treatment. The glossary also defines records, ensemble heads, and MaxN.

Composition and training output

Algorithm	Execution	Policy or solver	Learner	Training output	Example
PPO	Engine	rf.policies.Ppo	rf.learners.Ppo	PpoBatch	Connect 4 PPO training example
DQN	Engine	rf.policies.EpsilonGreedyQ	rf.learners.Dqn	DqnBatch	GridWorld DQN training example
TreeStrap + selective expectimax	Engine	rf.policies.SelectiveExpectimax	rf.learners.TreeStrap	TreeStrapBatch	TreeStrap training example
TreeStrap + UCT MCTS	Engine	rf.policies.Mcts	rf.learners.TreeStrap	TreeStrapBatch	TreeStrap MCTS training example
Minimax	Engine	rf.policies.Minimax	rf.learners.TreeStrap	TreeStrapBatch	Interactive play and evaluation examples
AlphaZero	Engine	rf.policies.AlphaZero	rf.learners.AlphaZero	AlphaZeroBatch	AlphaZero training example
CFR / CFR+	Standalone solver	rf.solvers.Cfr	—	Internal strategy tables	CFR solving example
External-sampling MCCFR	Standalone solver	rf.solvers.Cfr(variant="external_mccfr")	—	Internal strategy tables	MCCFR solving example
Deep CFR	Standalone data-generating solver	rf.solvers.DeepCfr	—	DeepCfrBatch	Deep CFR training example

Engine rows name the exact policy/learner handles passed to rf.Engine. Standalone solvers own their traversal; tabular CFR updates internal tables, while Deep CFR emits a caller-trained batch.

Supported semantics

Algorithm	Workflow	Players	Decisions	Chance	Information	Network output
PPO	Engine	Any N	Sequential or simultaneous	Sampled by the engine	Perfect or imperfect	Policy logits plus value: (rows, actions + 1)
DQN	Engine	Any N	Sequential or simultaneous	Sampled by the engine	Perfect or imperfect	Q values: (rows, heads, actions)
TreeStrap + selective expectimax	Engine	Any N	Sequential or simultaneous	Committed or ExpandAll	Perfect only	Ensemble action values
TreeStrap + UCT MCTS	Engine	Sequential <=2; simultaneous N	Sequential or simultaneous	AlwaysResample, Committed, or ExpandAll	Perfect only	One ensemble head of action values
Minimax	Engine	2	Sequential	Committed or ExpandAll	Perfect only	Q values (rows, 1, actions)
AlphaZero	Engine	Any N	Sequential or simultaneous	AlwaysResample, Committed, or ExpandAll	Perfect only	Policy logits (rows, actions) plus values (rows,)
CFR / CFR+	Standalone solver	2-10	Sequential	Exact enumeration	Imperfect-information native	No network
External-sampling MCCFR	Standalone solver	2-10	Sequential	Sampled	Imperfect-information native	No network
Deep CFR	Standalone data-generating solver	Kuhn 2-10; Leduc 2; Hold'em 2-9	Sequential	Sampled	Imperfect-information native	Per-player advantage networks; average-policy samples

Algorithm families

PPO

On-policy clipped policy gradient over GAE(lambda) advantages. Collection is windowed: collect(n) advances complete rounds under frozen weights to the record floor, bootstraps unfinished trajectories from the critic, and emits them, so every batch is internally single-version. Train a few clipped epochs per batch, discard it, and collect with the updated weights; the recorded ratio corrects action likelihood only, so strict PPO uses synchronous collect.

Sources: Proximal Policy Optimization Algorithms.

DQN

Observation-only off-policy value learning with sparse legal-action records. n_step > 1 emits multi-step windows over each agent's own decisions, with per-record gamma^k discounts; the window is uncorrected off-policy, so keep n small.

Sources: Human-level control through deep reinforcement learning, Deep Exploration via Bootstrapped DQN, Deep Reinforcement Learning with Double Q-learning, Prioritized Experience Replay, Dueling Network Architectures for Deep Reinforcement Learning, A Distributional Perspective on Reinforcement Learning, Noisy Networks for Exploration, Rainbow: Combining Improvements in Deep Reinforcement Learning.

TreeStrap + selective expectimax

Best-first search whose backed-up values become supervised learning targets.

Sources: Bootstrapping from Game Tree Search.

TreeStrap + UCT MCTS

UCT for sequential games and decoupled UCT for simultaneous multiplayer games.

Sources: UCT, MCTS variants for simultaneous move games (DUCT).

Minimax

Deterministic depth-limited minimax with expectation backup at declared chance nodes (expectiminimax) and callback-scored frontier leaves. A zero callback plays horizon-perfectly from terminal rewards alone, making this the classical fixed-strength baseline; no quiescence search, so fixed-depth play is subject to the horizon effect. Paired with TreeStrap this is the original TreeStrap(minimax) setting.

Sources: Bootstrapping from Game Tree Search.

AlphaZero

PUCT self-play with policy targets and outcome values. For sequential games, sequential_backup="auto" uses negamax at one or two players and MaxN above two; "maxn" forces MaxN at two players.

Sources: AlphaZero.

CFR / CFR+

Tabular alternating-update counterfactual regret minimization.

Sources: Counterfactual regret minimization, CFR+.

External-sampling MCCFR

Sampled CFR variant for games whose chance tree is too large to enumerate.

Sources: Monte Carlo CFR.

Deep CFR

Batched external-sampling traversals with caller-owned buffers and training.

Sources: Deep CFR.

See built-in compatibility, sampling and training, and the inference contract.