COT Toolset Field Guide

Part 0

The map: eight packages, two stores, one direction

The toolset is a set of sibling git checkouts under one plain directory (conventionally ~/code/trading_workspace/). The workspace directory itself is not a repo. The sibling layout is load-bearing: editable installs, CI checkout steps, and requirements.txt entries all use relative ../<repo> paths, so the directory names and the side-by-side placement are part of the contract.

Dependency direction is one way, and each layer refuses the next layer's question:

cotdata  ◂  cotmetrics  ◂  cot-analyzer
crucible ◂  crucible-stack  ◂  <your strategy repo>  (example: npf)

Note that marketdata is a sibling of cotdata, not a consumer: the two agree on symbol naming by convention and import nothing from each other. Equities have no COT report, which is why they cannot share one registry.

The two stores

Everything downstream reads two parquet stores, addressed by environment variable:

• COTDATA_STORE (typically ~/code/cotdata_store): CFTC positioning, written by cotdata-update.
• MARKETDATA_STORE (typically ~/code/marketdata_store): daily bars and contract specs, written by marketdata-update.

Part 1

Clone the workspace

Create the container directory and clone the siblings into it. The public set is enough to fetch COT, fetch bars, compute metrics, run the dashboard, and validate your own strategies. The two private repos add the tuned universe and the worked-example book; skip them if you don't have access and see the fallbacks noted below.

mkdir -p ~/code/trading_workspace && cd ~/code/trading_workspace

# Public toolset
git clone https://github.com/mspinola/cotdata.git
git clone https://github.com/mspinola/marketdata.git
git clone https://github.com/mspinola/cotmetrics.git
git clone https://github.com/mspinola/cot-analyzer.git
git clone https://github.com/mspinola/crucible.git
git clone https://github.com/mspinola/crucible-stack.git

# Private (requires access): tuned universe + the example strategy repo
git clone git@github.com:mspinola/cotmetrics-config.git
git clone git@github.com:mspinola/npf.git

Directory names matter. -e ../cotdata and friends resolve relative to each repo, so renaming a checkout breaks installs, and CI reproduces exactly this layout when it checks siblings out next to each repo.

Part 2

Environments and variables

Venvs are per-repo, created with uv, but in practice one master venv does most of the work. Python 3.11 is the workspace-wide pin (cotdata/marketdata/cotmetrics/crucible accept 3.10+, but crucible-stack and the strategy layer require 3.11+, and 3.11 is what every deployed box runs).

The master environment

If you cloned npf (or once you have your own strategy repo with an equivalent requirements.txt), one venv holds everything as editable installs:

cd ~/code/trading_workspace/npf
uv venv --python 3.11
source .venv/bin/activate
uv pip install -r requirements.txt   # installs -e . plus -e ../cotdata, ../marketdata,
                                     # ../cotmetrics, ../crucible, ../crucible-stack

Working library-only (no strategy repo), give each package its own venv from its README, for example:

# marketdata, with the free Yahoo producer:
cd ~/code/trading_workspace/marketdata
uv venv --python 3.11
uv pip install -e ".[yahoo,dev]" "setuptools<81"

# cotdata:
cd ~/code/trading_workspace/cotdata
uv venv --python 3.11 && uv pip install -e ".[dev]"

# cot-analyzer (its own venv, Dash stack + editable siblings):
cd ~/code/trading_workspace/cot-analyzer
uv venv && source .venv/bin/activate && uv pip install -r requirements.txt

Editable installs are the working tree

There is no copy. import cotdata reads the checkout directly, in whatever state it is in at that moment. Four consequences: a git checkout / stash / rebase in a sibling changes your behavior with no change in your own repo; rm in a sibling removes it for every venv pointing at it; pip list versions are install-time metadata, never refreshed, so verify APIs by signature, not by pin; and stale __pycache__ can outlive deleted modules. Also verify imports resolve where you think: a sibling not installed in the active venv still imports from the workspace root as a phantom namespace package instead of failing.

python -c "import cotdata; print(cotdata.__file__)"   # a real install prints a path;
                                                      # a phantom prints None

Environment variables

Put the store variables in your shell profile:

# ~/.zshrc or ~/.bash_profile
export COTDATA_STORE=~/code/cotdata_store
export MARKETDATA_STORE=~/code/marketdata_store
export COTMETRICS_PARAMS=~/code/trading_workspace/cotmetrics-config/params.yaml
export COT_VIZ_CONFIG=~/code/trading_workspace/cotmetrics-config/params.yaml

Part 3

COT data: free, any OS, no account

cotdata downloads the CFTC's own public history files over plain HTTPS. No API key, no subscription, works on macOS, Linux, and Windows. This is the zero-cost half of the system and the first thing to populate.

export COTDATA_STORE=~/code/cotdata_store
cotdata-update --cot-all      # all four reports; first run pulls full history, cached after
cotdata-update --check        # read-only status from the manifest, no network
python -c "import cotdata; print(cotdata.get_cot('ES').tail())"

The four reports

Reading it

The public API is deliberately small. get_cot returns a date-indexed DataFrame per market; the registry maps internal symbols (ES, GC, CL, 6A, ...) to CFTC market codes, stitching historical code changes behind one symbol:

import cotdata

legacy = cotdata.get_cot("GC")                      # default report="legacy"
disagg = cotdata.get_cot("GC", report="disagg")
tff    = cotdata.get_cot("ES", report="tff")
cit    = cotdata.get_cot("ZW", report="supplemental")

cotdata.all_symbols()                               # the 51-market registry
cotdata.symbol("GC")                                # registry entry, incl. cftc_code

The store on disk is one parquet per market per report (cot_legacy/GC_088691.parquet and so on), plus per-half manifests and a status.json that schedulers can poll. There is also a vintage subsystem (cotdata-vintage fetch / ingest / diff / asof / coverage) that snapshots the CFTC's raw files over time so revisions and restatements are detectable; you don't need it on day one, but know it exists before building anything that assumes COT history never changes.

Part 4

Market bars: pick your vendor

marketdata serves two domains from one store: equities/ETFs (free, via Yahoo) and futures (Norgate or Databento). Which vendors you can run depends on your OS and budget:

Path A: zero-cost start (any OS)

export MARKETDATA_STORE=~/code/marketdata_store
marketdata-update --bars --domain equities     # every registered ETF/index via Yahoo
marketdata-update --check                      # read-only summary from the manifest

Yahoo bars are stored nearly raw plus dated dividend/split actions, and every adjustment tier (split, raw, total) is derived on read, never stored. That is deliberate: Yahoo restates its own adjusted column every time a dividend lands, while raw bars plus dated actions are immutable facts.

Path B: futures via Databento (any OS, paid)

export DATABENTO_API_KEY=db-...
marketdata-update --ingest-databento --windowed-n1-stats   # stage 1: PAID, resumable, run by hand
marketdata-update --build-databento                        # stage 2: free, local files only

Two stages on purpose: ingest appends raw vendor files under the store's _raw/ area and is billed; build is a free, offline projection into store bars, so you can rebuild forever without paying twice. Limits to know: history starts at the CME Globex floor of 2010-06-06, and eight registry markets are not on Globex at all (ICE softs SB/CT/CC/KC/OJ, ICE WBS and DX, CME lumber LBR); they are skipped, never billed.

Path C: futures via Norgate (Windows box, deep history)

The Norgate producer drives a locally installed, authenticated Norgate Data Updater; there is no macOS/Linux path at any Python version. The standard pattern is one Windows producer box on a schedule, with every other machine reading a synced copy of the store (reads never touch the network, so a synced store is fully functional everywhere). It writes both futures tiers per symbol (backadj and unadj, with propadj derived on read), plus the contract-specs table (marketdata-update --metadata: point value, tick size, margin) that cost models need.

Reading bars

import marketdata

spy = marketdata.get_bars("SPY")                    # equities default tier: split
tr  = marketdata.get_bars("TLT", "total")           # total-return tier, derived on read
es  = marketdata.get_bars("ES")                     # futures default tier: backadj
raw = marketdata.get_bars("ES", "unadj")            # as-traded, for sizing/costs

# Point-in-time futures read: the series AS IT STOOD on that date
old = marketdata.get_bars("HE", "backadj", asof="2015-06-01")

marketdata.coverage_gaps(["ES", "GC", "SPY"], stale_after_days=7)   # manifest-only, cheap
marketdata.read_metadata()                          # contract specs table

Futures bars carry more than OHLCV: Delivery Month flips exactly at each roll, and FirstContract / SecondContract name the two most active expiries each day. Note there is no per-expiry price series in the store (continuous series only, both vendors), and asof= matters whenever your logic uses ratios of back-adjusted prices, because additive back-adjustment re-anchors all history at every roll.

Part 5

The metrics layer: cotmetrics

cotmetrics turns the two stores into positioning metrics. The core object is the 0-to-100 positioning index: where today's net position sits inside its own lookback range, computed per trader group (commercials, large specs, small traders), on either raw net contracts or net divided by open interest ("OI-normalized", Larry Williams' WILLCO generalized to all three legs).

The universe file

COTMETRICS_PARAMS points at a YAML file defining the instrument universe: asset classes, per-instrument Name / Symbol / CustomLookbackWeeks / optional Role, the lookback set, and the year range. The private cotmetrics-config/params.yaml names 47 markets across 9 asset classes with per-market tuned lookbacks, and marks a handful Role: heldout (indexed but never plotted or selected, reserved as out-of-sample markets). The packaged sample has the same schema with 6 symbols, so everything runs out of the box, just on a toy universe. Writing your own file in this schema is fully supported.

Minimal working example

# stores populated (Parts 3-4), env vars set (Part 2)
from cotmetrics.indexer import get_indexer
import cotmetrics.constants as const

idx = get_indexer()          # builds once; ~90s cold on the full universe,
                             # fast afterwards off the COTMETRICS_CACHE parquet cache

name = idx.get_instrument_from_symbol("GC").name    # instruments are addressed by name: "Gold"
df = idx.get_symbols_data(name, "Custom")           # or "26" / "52"; basis=const.BASIS_OI_NORM for net/OI

print(df[[const.COMMS_IDX, const.LRG_IDX, const.SML_IDX,
          const.CLOSING_PRICE]].tail())

The returned weekly frame carries stable "wire format" alias columns downstream code reads by literal name: comms_idx / lrg_idx / sml_idx (the index per leg), z-scores, momentum, willco, oi_zscore, and setup flags like pos_idx_setup_long. Prices arrive by resampling daily bars from marketdata.get_bars to weekly Tuesday bars, matching the COT report date; two markets (MSCI EAFE and EM futures) are priced through ETF proxies (MFS → EFA, MME → EEM) because no continuous futures series exists for them.

Part 6

The dashboard: cot-analyzer

A Dash app over cotmetrics, and deliberately nothing more: cotmetrics carries no presentation config, cot-analyzer computes no metrics. Pages include a positioning heatmap, the cross-asset crowding strip, per-market graphs and analysis, Disaggregated/TFF category views, dollar exposure aggregation, and an options max-pain view.

cd ~/code/trading_workspace/cot-analyzer
./run-local.sh          # http://127.0.0.1:5001

Use the launcher, not python src/main.py: it wires the cache/log/db paths, points COTMETRICS_PARAMS and COT_VIZ_CONFIG at the private config when the sibling exists, sources the repo's .env (where COTDATA_STORE / MARKETDATA_STORE live for deployment), and refuses to start if the two stores are unset or identical. There is no hot reload; restart after every edit.

On boot the app runs a cheap manifest-only price check and refuses to start if the bar store can serve no configured instrument at all (the failure it guards against is silence: a dashboard of blank charts). A partial store warns and boots; COT_ANALYZER_ALLOW_MISSING_PRICES=1 downgrades the refusal for development.

Part 7

The validation stack: crucible and crucible-stack

This is the part of the toolset that keeps you honest. The architecture is a chain of refusals: each layer answers exactly one question and refuses the next one, so no single session can generate a hypothesis, run its own gate, and narrate the result as a win.

The pivot between layers is the TradeLog: capital-free, denominated in R (one unit of risk taken at entry). That is what lets crucible judge an edge without knowing account size; the single place R converts to currency is EquityResult.meta["r_denominator"] in the capital simulator.

The gauntlet in five minutes

crucible's headline call is run_gauntlet, four pillars: REAL (distinguishable from noise, corrected for how many variants you searched), STRONG (economically meaningful at the CI lower bound), DURABLE (holds up walk-forward over time), GENERAL (travels to markets it wasn't built on). Verdicts are a boolean passed plus an audit report; thresholds are tuned to fail a marginal result rather than pass it.

from crucible.edge import barrier_trades
from crucible.validation import run_gauntlet, walk_forward, SearchSpaceLog

log = SearchSpaceLog(scope="GC:my_grid", path="search.jsonl")   # the ledger
for params in grid:
    log.record(params, status="tried")     # BEFORE it runs, so failures still count
    entries = my_signal(px, **params)      # a boolean Series over an OHLC frame
    ...
log.mark_selected(best_params)

wf = walk_forward(px, my_signal, best_params, ...)   # leakage-free by construction

g = run_gauntlet(
    wf.stitched,        # the honest, stitched out-of-sample trade log
    prices=px,          # enables REAL's random-timing null
    wf=wf,              # adds the DURABLE gate
    n_variants=log,     # the LEDGER drives the multiple-testing correction
)
print(g.audit_report())
print(g.passed)

crucible-stack wraps this into the search-and-deploy loop: sweep() runs a grid and returns a TrialMatrix (recording both varied and fixed params, with the shared ledger attached), select() picks a winner and prices it against the whole search (PBO, deflated Sharpe, reality check, ANDed into a single trustworthy flag), simulate_equity() / equity_bands() turn the surviving log into a currency account with bootstrap bands, and orchestrate owns the DeploymentLedger, drift envelopes frozen at promotion, and re-optimization triggers. Your strategies live in your repo, registered via @register_strategy; crucible-stack ships none and a boundary test keeps it that way.

Part 8

Worked example: the NPF CS 80/20 book

Everything above converges in one config file. In npf's naming, books are written NPF <gate> <band>, so CS 80/20 means: gate CS, entry band hi: 80 / lo: 20. All books run the same engine (NpfStrategy); only parameters differ. The deploy variant lives at npf/config/npf/cmr_cs_oinorm_liquid.yaml and is the default config of every analysis mode.

Anatomy of the config

strategy:
  name: "NPF"
  mode: "fixed"
  params:
    cot_start_date: "1986-01-01"   # COT history floor for the weekly frame
    gate: "CS"                     # commercials AND small specs, opposite extremes
    oi_normalized: true            # index legs on net/OI instead of raw net
    hi: 80                         # crowded-long index threshold (loose band)
    lo: 20                         # crowded-short index threshold
    trigger: "chart"               # price confirmation before entry
    stop: "wick"                   # stop placement mode (+ always-on 2.5R hard floor)
    exit: "cot_neutral"            # exit when the GATE'S OWN legs revert to neutral
    entry_fill: "next_open"

• gate: "CS" is a conjunction, not a spread: enter long only when the commercial index is at or above hi and the small-spec index is at or below lo (mirror for shorts). The gate vocabulary is C / CL / CLS / CS plus WILLCO variants; equity markets are forced to commercials-only by rule.
• hi/lo: 80/20 is the loose band (the engine defaults to a tight 95/5). OI-normalization is what pays at the loose band.
• exit: "cot_neutral" is gate-matched: a CS entry rides until the C and S legs are both neutral again. The per-symbol index lookback is the tuned "Custom" window from the cotmetrics params file, which is why the private config matters.
• The data: block picks the universe by asset class plus explicit symbol adds; the holdout:, pardo_wfm:, and sizing: blocks feed the corresponding analysis modes below.

Commanding a run

./analyze is the one entry point, a thin dispatcher mapping friendly mode names onto main.py flag bundles. It runs with the project venv via the ./npf wrapper, expects the store variables in your environment, and defaults to the CS 80/20 liquid config:

cd ~/code/trading_workspace/npf
./analyze gauntlet                          # four-pillar verdict on the pooled book
./analyze gauntlet --n-variants 24          # declare the search size for REAL's correction
./analyze holdout                           # early/late split scorecard (2019 split, 8wk embargo)
./analyze wfa --symbol GC                   # per-asset walk-forward
./analyze mc                                # account-level Monte Carlo under the sizing block

Output lands as HTML tearsheets under results/tearsheets/, and every run first writes a durable JSON run record under results/runs/<date>/ carrying provenance: git commit, dirty-tree flag, resolved data window, and both store paths. Quoted figures always trace back to one of these. Useful passthroughs: --gross (skip the cost model), --side long|short|both, --general-by class|symbol, and pointing --config at a directory compares every config in it into one dated results folder.

Part 9

Commanding a study through Claude Code

The workspace is built to be driven by an agent, and the governance for that is written down where the agent reads it: the workspace CLAUDE.md plus npf/AGENTS.md (research governance) and crucible/AGENTS.md (using the library). Claude Code loads these automatically when you open a session in the workspace. The core rule everything else serves:

The session pattern: separate the roles

1. Generator pass. One session drafts the pre-registered spec, the config, and the code, and logs every variant it tries (including discards) in a SearchSpaceLog. It does not run the verdict.
2. Evaluator pass. A fresh session (one that did not author the book) runs the gauntlet and reads the output back, failures included, with the true variant count: ./analyze gauntlet <config> --n-variants <N>.
3. Review pass (when the spec introduces a new outcome or window). A third session reads the frozen spec and scripts, runs nothing, and appends findings plus an explicit list of what it could not check.

Example prompts that respect the seams:

# Session 1 (generator)
"Draft a pre-registered spec and a config for a <hypothesis> book on the CS 80/20
 engine. Log every variant you try in a SearchSpaceLog with a path. Freeze the spec
 under docs/. Do NOT run the gauntlet or characterize the results."

# Session 2 (evaluator, fresh)
"Run ./analyze gauntlet config/npf/<book>.yaml --n-variants <N from the ledger>.
 Report the audit output verbatim, including every failed check. Do not modify
 the config or the spec."

House rules the guide inherits

• Assume look-ahead until proven otherwise. Build trade logs with holdout / walk_forward so purge and embargo hold by construction; an imported log's leakage-freedom is unproven and must be labeled as such.
• Attack a passing book before celebrating it: perturb the frozen config (risk multiples, lookback, entry fill, embargo) and rerun the gate. Diagnosis after a pass, never a second search.
• Every quoted figure carries a reproducer (script, seed, data reference), and working notes are authored under the repo's docs/ from the first keystroke, not in a scratchpad.
• Measure, do not assume. The workspace docs are full of dated corrections where probing the actual files overturned a written assumption. When a doc and a measurement disagree, fix the doc in the same change.

Appendix

Verification checklist and known traps

Run these after setup, in order. Each one is cheap and each failure points at exactly one part of this guide.

• python -c "import cotdata, marketdata, cotmetrics, crucible; print(cotdata.__file__)" prints real paths, not None (phantom-package check, Part 2).
• cotdata-update --check and marketdata-update --check both report populated domains with recent last_dates (Parts 3-4).
• python -c "import cotdata; print(cotdata.get_cot('ES').tail(3))" returns rows.
• python -c "import marketdata; print(marketdata.get_bars('SPY').tail(3))" returns rows.
• The CotIndexer snippet (Part 5) prints index columns, and the boot log does not warn about the sample params fallback.
• cd cot-analyzer && ./run-local.sh boots without the price-store refusal and renders charts at http://127.0.0.1:5001.
• Per-repo test suites pass with a store variable exported; strategy-repo suites run against the committed fixture store, never the live one (the repo READMEs carry the exact commands).

The traps, collected