I wanted to test trading ideas on real tick data without fooling myself. This collects SPY trades and quotes back to 2019, options trades and quotes, daily snapshots of the full option chain, minute bars for 40 other assets, and public data from sources like Cboe, the CFTC, and the Fed. Everything is stored as Parquet, with a DuckDB layer on top that can be thrown away and rebuilt.
Most of the work goes into not cheating. Each feature only becomes usable on the day it would actually have been known. Standard errors are corrected for overlapping returns, and thresholds are fit on training data only. Signals that survive then get backtested on QuantConnect’s LEAN engine.
Results so far
I’ve run more than 50 experiments, and most were rejected. A handful survived, but none beats simply holding SPY on its own. A few still look useful as additions to holding it.