JeTech Lab Offline RL Research Roadmap
A roadmap connecting market world models, offline RL algorithms, time-series representation learning, dataset generation, and real market gates.
Summary
The next JeTech Lab research step is not another algorithm in isolation. The work should be split into four research axes:
A. World model market generator
B. Offline RL algorithm
C. Time-series pattern representation
D. Real market gateThese axes only matter when connected. A better world model still fails if the action dataset is weak. A new offline RL algorithm fails if synthetic paths are unrealistic. A larger encoder fails if the reward and gate disagree.
The research loop should stay fixed:
real OHLC
-> world model training
-> synthetic market paths
-> synthetic action/position dataset
-> offline RL algorithm
-> synthetic eval
-> real 5Y gate
-> failure taxonomy
-> next dataset or model change1. World Model Research
The world model should not be evaluated as a pure price forecasting model. Its job is to create a market distribution that can train offline RL policies.
Current definition:
input:
past 224 days OHLC
output:
future 96 days OHLC path distributionThis is generative learning, not RL. The full pipeline is world-model-assisted offline RL because generated paths become the base of the transition dataset.
First Step: Quality Report
Every world model run should emit a quality report.
| Metric group | Contents |
|---|---|
| Return distribution | 1d, 5d, 20d quantiles, skew, kurtosis |
| Volatility | realized vol, squared-return autocorrelation |
| Drawdown | max drawdown, duration, recovery profile |
| Tail | gaps, crashes, volatility spikes |
| Regime | trend, range, high-vol duration |
| OHLC validity | candle constraint violations |
| Strategy utility | simple strategy ranking preservation |
Lower loss does not prove a better training environment. The model must preserve market features that affect policy learning.
Second Step: Regime-Conditioned Generation
Add explicit conditions:
volatility_bucket = low / normal / high / crisis
trend_bucket = up / range / down
drawdown_bucket = near_high / mild / deep
asset_context = crypto_beta / equity_beta / risk_offOffline RL should see rare but important regimes. Too few crash samples teach leverage. Too many teach flat exposure. The generator needs balanced oversampling.
Third Step: Joint Transition Direction
The current pipeline separates price and action:
price path -> action labeler -> reward transitionThe medium-term target is:
price, action, reward generated together
or price generated first and actions inferred by behavior priorsJeTech should not jump straight into a large diffusion world model. A better first move is regime-aware action labeling, then action-conditioned or trajectory-level generation.
2. Offline RL Algorithm Research
Algorithm work should use CORL as the reference. JeTech implementations should be comparable to CORL for CQL, IQL, TD3+BC, Decision Transformer, and ReBRAC.
Research order:
| Priority | Algorithm | Reason |
|---|---|---|
| 1 | IQL | avoids evaluating out-of-support actions |
| 2 | ReBRAC | strong actor-critic baseline |
| 3 | CQL | controls Q overestimation and tail risk |
| 4 | TD3+BC | minimal regression baseline |
| 5 | Diffusion-QL family | useful when action support is multi-modal |
| 6 | Decision Transformer | needs return-conditioning redesign |
This is the JeTech operating order, not a universal academic ranking.
Algorithm comparisons must fix:
world_model_version
generated_path_seed
synthetic_action_labelers
reward_config
encoder_backbone
train_steps
real_gate_periodIf any of these change, the experiment compares datasets or configs, not algorithms.
Diffusion Policy Entry Criteria
Diffusion policy work should start when:
1. the same state bucket contains multiple high-reward action modes, 2. deterministic actors collapse to average exposure, 3. IQL or ReBRAC is stable but cannot use action diversity, 4. sampled action candidates improve the real gate.
The first diffusion role should be candidate generation or behavior prior, not live deployment.
3. Time-Series Pattern Research
The encoder should turn OHLC paths into usable market state representations.
JeTech keeps the price-only rule:
input = OHLC window
no handcrafted alpha leakage
encoder learns representationThe encoder still needs to capture:
| Pattern | Why it matters |
|---|---|
| Volatility clustering | position sizing and risk |
| Trend persistence | momentum behavior |
| Mean reversion zones | contrarian behavior |
| Drawdown state | risk budget |
| Breakout compression | volatility expansion |
| Gaps and shocks | tail response |
| Cross-asset beta | risk-on and risk-off context |
Candidate encoders:
| Encoder | Strength | Risk |
|---|---|---|
| causal conv or mixer | fast and stable | limited long memory |
| TCN | multi-scale temporal patterns | dilation design |
| Transformer | long context | overfit on small datasets |
| iTransformer-style | good time-series token framing | OHLC tokenization matters |
| diffusion or score features | distributional representation | expensive and harder to interpret |
Encoder research should isolate the encoder. Keep the offline RL algorithm and dataset fixed.
Representation probes should test whether the encoder separates useful states:
next 20d realized volatility bucket
next 20d drawdown bucket
trend/range regime label
tail event probability
simple strategy winner classThese probes are diagnostics, not live trading signals.
4. Dataset Generation Policy
In offline RL, the dataset is as important as the algorithm. The policy cannot reliably learn actions that are absent from the dataset.
Labeler types:
| Type | Purpose |
|---|---|
| random | action support |
| trend following | momentum archetype |
| mean reversion | contrarian archetype |
| volatility-aware | reduce exposure in high vol |
| drawdown-aware | adjust risk after losses |
| oracle-like | upper-bound studies only |
Oracle-like labelers should carry leakage metadata and stay out of default training datasets.
Every dataset should have a card:
dataset_id
world_model_version
generation_seed
symbol_set
date_source
regime_mix
labeler_mix
reward_config
cost_config
transition_count
quality_metrics
leakage_flagsWithout dataset cards, experiments cannot be reproduced.
5. Real Gate Research
The final metric is not synthetic score. It is real market survival.
Gate metrics:
| Metric | Reason |
|---|---|
| rolling 30d Sharpe | recent stability |
| 5Y Sharpe / Calmar | long-horizon risk-adjusted return |
| max drawdown | survival |
| flat ratio | policy collapse |
| one-sided ratio | directional bias |
| turnover | cost sensitivity |
| tail loss | crash response |
| seed robustness | training luck |
Failures should be classified:
| Failure type | Next action |
|---|---|
| synthetic-real gap | inspect world model and reward |
| flat collapse | adjust labeler mix, BC weight, conservative penalty |
| high turnover | adjust cost reward, smoothing, regularization |
| one-sided bet | check regime balance and action support |
| tail drawdown | strengthen tail generation and drawdown penalty |
| unstable seed | inspect capacity, normalization, dataset size |
This turns the training loop into research instead of repeated reruns.
Six-Month Sequence
| Month | Goal | Output |
|---|---|---|
| 1 | World model quality report | world_model_quality.json, path dashboard |
| 2 | Dataset cards and labeler mix | dataset_card.json, leakage flags |
| 3 | IQL/ReBRAC/CQL CORL parity | algorithm parity report |
| 4 | Regime-conditioned world model | regime-balanced synthetic set |
| 5 | Diffusion action candidates | diffusion candidate vs one-step actor report |
| 6 | Real gate failure taxonomy | gate failure report, next-action recommendation |
The order is conservative by design. Measure first, then add algorithms. Without measurement, diffusion and larger sequence models make failures harder to diagnose.
Research Position
The core question is:
Which synthetic market distribution and offline RL algorithm combination
survives the real market gate most consistently?World models, offline RL algorithms, encoders, and dataset policies are subproblems of that question. Improve each axis separately, then bind them with the same real gate.