Summary

Offline RL trains a policy from a fixed transition dataset. Trading fits this setting because free exploration in live markets costs money and risk. Historical OHLC, fills, positions, and rewards become the training material.

The hard part is out-of-dataset action selection. If the dataset mostly contains flat and small long actions for a state, but the learned actor outputs max long, the critic has no direct evidence for that action. Deep Q-learning can overestimate unsupported actions. Levine et al.'s Offline RL tutorial frames this as distribution shift and out-of-distribution action error.

Classic offline RL reduces this risk through conservative value learning or behavior regularization. Diffusion-based offline RL changes the policy class. It uses a generative action model that can represent multiple behavior modes in the dataset.

For JeTech, the working position is:

Diffusion policy is not automatically superior. Trading also cares about inference latency, action smoothness, turnover, and regime transfer. We should first bring classic baselines close to CORL-quality implementations, then test whether diffusion adds measurable value.

The Offline RL Problem

Online RL collects new interaction data. Offline RL uses a fixed dataset:

D = (observation, action, reward, next_observation, done)
policy updates use only D
no new market interaction

This is attractive in trading because the agent does not need to explore dangerous actions. The constraint is that the policy should not trust values far outside dataset support.

A common failure loop looks like this:

1. The dataset covers only a narrow action range. 2. The actor moves toward unsupported actions to increase predicted reward. 3. The critic assigns high Q-values to actions it never saw. 4. The actor moves farther away. 5. Synthetic backtests improve, but the real gate fails.

Trading can fail in the opposite direction too. A conservative objective can produce a flat policy that avoids risk and earns nothing. The research question is how far a policy should improve beyond behavior while staying inside reliable support.

Classic Offline RL Families

TD3+BC

TD3+BC adds a behavior cloning term to the TD3 actor objective.

actor objective = maximize Q(s, pi(s)) + minimize distance to dataset action

It is small, fast, and useful as a sanity baseline. If TD3+BC fails on a synthetic dataset, the dataset or reward design may be broken.

The limitation is deterministic action modeling. If the same state supports long, flat, and short behavior, a deterministic actor can average the modes into a weak exposure.

ReBRAC

ReBRAC revisits TD3+BC and shows that design choices matter: normalization, actor and critic regularization, layer normalization, critic updates, and target updates can change results.

For JeTech, ReBRAC is the main actor-critic baseline because it is cheap, deployable, and easy to compare with CORL.

CQL

CQL penalizes high Q-values on unsupported actions. It is useful when a policy exploits critic overestimation, but it can become too conservative.

In trading, CQL deserves attention when synthetic Sharpe is high but real Sharpe is weak, one-sided exposure dominates, or tail losses appear in the real gate.

IQL

IQL avoids direct evaluation of unseen actions. It learns a value function with expectile regression and extracts good dataset actions through advantage-weighted behavior cloning.

This fits synthetic trading datasets because our action labelers are not expert traders. They create support. IQL then selects the actions that worked inside that support.

The limitation is dataset quality. If the dataset contains no good actions, IQL cannot invent them.

Decision Transformer

Decision Transformer treats RL as sequence modeling. It predicts actions from return-to-go, state, and action history.

Trading is path-dependent, so the sequence framing looks natural. The hard part is choosing the return target. Raw return, Sharpe-like return, drawdown-conditioned return, and turnover-adjusted return lead to different policies. JeTech should treat the current decision_transformer path as a windowed sequence policy until return conditioning is designed with the trading objective.

Diffusion-Based Offline RL

Diffusion-QL represents the policy as a conditional diffusion model. The model denoises action samples conditioned on the state, while the loss combines behavior cloning and Q maximization.

The gain is multi-modal action modeling. For a state with uncertain regime, the dataset may contain several valid action modes:

state: rising volatility, weakening trend
action modes:
  - stay flat
  - small short
  - wait for breakout then go long

A diffusion policy can keep those modes separate. A deterministic actor may collapse them into a mediocre average.

The cost is iterative sampling. Training and inference can become slow. Efficient Diffusion Policy reduces training cost, and DTQL uses diffusion as a trust region while deploying a one-step policy.

JeTech should test diffusion in stages:

Is There an Academic Ranking?

There is no universal ranking. D4RL scores depend on dataset type, support, reward density, terminal handling, normalization, and hyperparameter budget. A Clean Slate for Offline RL argues that researchers should separate implementation differences from algorithmic contributions.

JeTech's operating order should be:

This is not an academic leaderboard. It is the JeTech experiment order.

Implementation Standard

CORL is the reference implementation family. It provides single-file implementations for CQL, IQL, TD3+BC, Decision Transformer, ReBRAC, and related algorithms. JeTech code can differ from CORL, but each difference should be intentional.

Checklist:

Diffusion should come after this baseline cleanup. Otherwise, we cannot tell whether improvement came from the policy class, dataset generation, or reward leakage.