New BookPilot users sometimes run it in the Strategy Analyzer, get zero trades and assume something is broken. Nothing is. The Strategy Analyzer simply never sees the data a Level 2 strategy needs.
What the Strategy Analyzer replays
The Strategy Analyzer runs a strategy over historical data, bar by bar. With Tick Replay switched on, it also replays the individual trades inside each bar, including whether each one traded at the bid or the ask. That is enough to rebuild a footprint, which is why trade-based strategies can be backtested this way.
What it does not replay
Market depth. A Level 2 strategy reacts to changes in resting size: a level growing, shrinking, appearing or disappearing. Those changes are separate events in the live feed, and they are not part of historical bar or tick data. In the Strategy Analyzer, the order book is simply empty.
A depth strategy with an empty book has nothing to measure, so it never signals. Vantedge BookPilot works exactly this way, and its user guide lists "no trades in the Strategy Analyzer" as expected behaviour.
Why depth cannot be rebuilt from trades
It is tempting to think the book could be reconstructed from the trade history. It cannot. Trades only show orders that executed. Orders that were placed and then cancelled leave no trace in the trade record, and cancellations are exactly what a Pulled Wall signal looks for. Any reconstruction would be guessing at the most important part.
Where each test environment fits
| Environment | Trades with bid/ask | Market depth | FlowPilot | BookPilot |
|---|---|---|---|---|
| Strategy Analyzer, no Tick Replay | No | No | No trades | No trades |
| Strategy Analyzer with Tick Replay | Yes | No | Backtestable | No trades |
| Market Replay | Yes | Yes | Runs | Runs |
| Real-time SIM | Yes | Yes, with a depth feed | Runs | Runs |
What you lose without a backtest
Being honest about the trade-off helps you plan:
- No optimiser. You cannot sweep a range of settings over months in minutes. Each Market Replay run takes real time, even sped up.
- Smaller samples. You will test over days or weeks, not years, so results carry more uncertainty.
- Data to download. Market Replay needs replay data for each day, loaded through Tools, Historical Data in NinjaTrader 8.
What you gain is realism. In Market Replay the strategy sees the same depth updates, in the same order, as it would live. Because BookPilot's engine runs on event timestamps rather than the PC clock, replaying the same day gives the same signals, so comparisons between settings are fair.
Backtests have blind spots too
Trade-based strategies are not off the hook. In the Strategy Analyzer, FlowPilot's daily profit goal, daily max loss and cooldown are not applied; those gates count real-time trades only. Backtest fills are also idealised. A FlowPilot backtest tells you about the signal, but you still need Sim101 to see the full behaviour.
Common workarounds, and why they mislead
Some traders try to approximate a depth strategy in the Strategy Analyzer by using trade-based proxies, such as treating a burst of aggressive selling as a sign that a bid wall was pulled. That is a different strategy. It may be worth testing in its own right, but its results say nothing about how the original book signal would have behaved.
Others run a depth strategy live on SIM for a week and treat that as a backtest. It is useful evidence, but a single week covers few market conditions. A fixed set of varied Market Replay days, replayed after each change, gives you a fairer comparison than whatever the live week happened to bring.
A testing plan for each
- FlowPilot: Strategy Analyzer with Tick Replay over at least 2 to 3 months, a walk-forward check on a later period, then several Sim101 or Market Replay sessions to confirm the daily gates.
- BookPilot: a fixed set of Market Replay days with depth, replayed continuously after each single-setting change, then real-time SIM.
The FlowPilot vs BookPilot comparison lists backtesting support next to data needs, which is often the deciding factor between the two.
Neither a backtest nor a replay can tell you what tomorrow's market will do, and futures losses can be swift. Finish your testing in SIM, after Market Replay for BookPilot, before moving to an evaluation, funded or live account.