- ago
I'm trying to understand how to interpret the results when running a Monte Carlo test using Basic Run mode on a limit-order system that includes NSF positions.

In the original backtest, I have:

~1140 actual positions

~70 NSF positions

However, after running the Monte Carlo test with Basic Run mode for multiple iterations, many of the simulations end up with only a few hundred positions (200-600, no NSF), which is significantly fewer than the original backtest.

From the documentation, my understanding is that Basic Run mode repeatedly re-runs the same backtest logic, but with a randomized sampling of the original trades.

My question is whether the following interpretation is correct:
- On a given day in the original backtest, there may be enough raw entry signals to fill (for example) 10 slots.
- In a Monte Carlo iteration, because only a subset of trades is randomly sampled, it’s possible that far fewer than 10 of those trades are present.
- As a result, many slots go unfilled in that iteration, which cascades into a much lower total number of positions over the full simulation.

Is this expected behavior in Basic Run mode, and is this the primary reason why some Monte Carlo simulations end up with far fewer positions than the original backtest? Thanks.
0
1,164
12 Replies

Reply

Bookmark

Sort
Cone8
 ( 15.74% )
- ago
#1
The Basic Run just runs the Strategy with the same settings many times. You'd only see a difference if 1) you have NSF positions, and, 2) the Strategy does not assign Transaction.Weight.

I can only guess that the big difference in number of Positions must be explained by something else random in the Strategy. I guess we'd have to see the Strategy to determine that without guessing.
0
johnbens8
 ( 0.00% )
- ago
#2
I believe I encountered the same problem today. I am conducting a strategy test, and when I manually click "Run Backtest“ for many times, the number of trades is usually around 4200.

However, when I use "Basic Run" for simulation, the number of trades varies greatly, even reaching a minimum of less than 100 trades.

I am not sure if this phenomenon is related to the order type used, as I am using a Stop Order for entry.

I have not observed this phenomenon in other strategies that use Market Order for entry.
0
- ago
#3
Here is an example strategy that generates far fewer signals in MC test. Run it on WD Nasdaq100 for last ten years with 10% equity, 10 positions, 1.1 margin factor, I have ~1670 positions and 185 NFS positions. With basic run in MC the position counts are from ~40-240. From equity curve of those MC runs I can see that there's no open positions for most of the time.

CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using WealthLab.Data; using WealthLab.Indicators; using System.Collections.Generic; using System.Linq; namespace WealthScript1 {    public class MyStrategy : UserStrategyBase    {       // Buy list - symbols that pass entry conditions       private static List<BarHistory> buys = new List<BarHistory>();       // Maximum number of entry signals       private const int maxEntrySignals = 10;       public MyStrategy() : base()       {          StartIndex = 200;       }       public override void Initialize(BarHistory bars)       {          // Create indicators          rsi = new RSI(bars.Close, 5);          PlotIndicator(rsi, new WLColor(0, 0, 0));          _startIndexList.Add(rsi);          qqqHistory = GetHistory(bars, "QQQ");          qqqClose = qqqHistory.Close;          qqqSMA200 = new SMA(qqqHistory.Close, 200);          PlotIndicator(qqqSMA200, new WLColor(0, 0, 255), default, default, "QQQPane");          _startIndexList.Add(qqqSMA200);          barsLow = bars.Low;          percentMultiplier = 1.00;          percentMultiplier = (100.0 - percentMultiplier) / 100.0;          buyLimitPrice = barsLow * percentMultiplier;          PlotStopsAndLimits(3);          // Cache indicators for PreExecute access          bars.Cache["rsi"] = rsi;          bars.Cache["qqqClose"] = qqqClose;          bars.Cache["qqqSMA200"] = qqqSMA200;          bars.Cache["buyLimitPrice"] = buyLimitPrice;          bars.Cache["qqqHistory"] = qqqHistory;          foreach (IndicatorBase ib in _startIndexList)             if (ib.FirstValidIndex > StartIndex)                StartIndex = ib.FirstValidIndex;       }       public override void PreExecute(DateTime dt, List<BarHistory> participants)       {          // Clear previous buy list          buys.Clear();          // Evaluate each participant for entry conditions          foreach (BarHistory participant in participants)          {             // Check if this participant passes all entry conditions             string reason = "";             bool passed = ShouldEnterPosition(participant, out reason);             if (passed)             {                // Store RSI value for ranking                int idx = GetCurrentIndex(participant);                IndicatorBase rsiInd = (IndicatorBase)participant.Cache["rsi"];                double rsiValue = rsiInd[idx];                participant.Cache["rsiValue"] = rsiValue;                buys.Add(participant);             }          }          // Sort by RSI value (ascending - lowest RSI first) and limit to maxEntrySignals          if (buys.Count > maxEntrySignals)          {             buys = buys.OrderBy(b => (double)b.Cache["rsiValue"]).Take(maxEntrySignals).ToList();          }       }       private bool ShouldEnterPosition(BarHistory bars, out string reason)       {          int idx = GetCurrentIndex(bars);          reason = "";          // Get indicators from cache          IndicatorBase rsiInd = (IndicatorBase)bars.Cache["rsi"];          TimeSeries qqqClose = (TimeSeries)bars.Cache["qqqClose"];          IndicatorBase qqqSMA200 = (IndicatorBase)bars.Cache["qqqSMA200"];          // Condition 1: RSI < 25.00          double rsiValue = rsiInd[idx];          if (rsiValue >= 25.00)          {             reason = $"RSI={rsiValue:F2} >= 25.00";             return false;          }          // Condition 2: QQQ Close > QQQ SMA200          double qqqCloseValue = qqqClose[idx];          double qqqSMA200Value = qqqSMA200[idx];          if (qqqCloseValue <= qqqSMA200Value)          {             reason = $"QQQ Close={qqqCloseValue:F2} <= QQQ SMA200={qqqSMA200Value:F2}";             return false;          }          return true;       }       public override void Execute(BarHistory bars, int idx)       {          Position foundPosition0 = FindOpenPosition(0);          bool inBuyList = buys.Contains(bars);          if (foundPosition0 == null)          {             // Entry logic - buy if symbol is in the buy list             if (inBuyList)             {                // Get buy limit price from cache                TimeSeries buyLimitPrice = (TimeSeries)bars.Cache["buyLimitPrice"];                double limitPrice = buyLimitPrice[idx];                Backtester.CancelationCode = 1;                _transaction = PlaceTrade(bars, TransactionType.Buy, OrderType.Limit, limitPrice, 0, "Buy " + 1.00 + "% below " + barsLow.Description);             }          }          else          {             // Exit conditions             // Exit: Close above previous high             if (idx > 0 && bars.Close[idx] > bars.High[idx - 1])             {                Backtester.CancelationCode = 1;                ClosePosition(foundPosition0, OrderType.Market, 0, "Sell At Market - Price Above Previous High");             }          }       }       private IndicatorBase rsi;       private TimeSeries qqqClose;       private IndicatorBase qqqSMA200;       private BarHistory qqqHistory;       private double percentMultiplier;       private TimeSeries barsLow;       private TimeSeries buyLimitPrice;       private Transaction _transaction;       private List<IndicatorBase> _startIndexList = new List<IndicatorBase>();    } }

0
johnbens8
 ( 0.00% )
- ago
#4
I use building blocks to construct strategies. Today, I did a simple test and found that the problem may be in Symbol Ranking by Indicator.

When I add this block to the strategy, the number of results in the MC Basic Run becomes abnormal. I feel that this block itself does not produce "randomness," and this phenomenon seems like a bug.
0
- ago
#5
@cone @glitch could you check whether you can reproduce this with the shared code? Thanks.
0
Cone8
 ( 15.74% )
- ago
#6
I ran the Strategy above from 2015 to 2026. 200 Basic MC Runs gave me about 200 highly correlated simulations, with Position count varying between 1,529 (min) and 1,931. NSFs averaged around 160. Long story short, I didn't see a problem with this example.
0
- ago
#7
That's strange. Just tried a few times, it came with the above issue every time for me.

Tried on another computer and still got similar results

0
Cone8
 ( 15.74% )
- ago
#8
Look at your Monte Carlo Position Sizing control and check the options for Max Positions. I'd expect it to be the same as the main setting, but who knows?
0
johnbens8
 ( 0.00% )
- ago
#9
This is really strange. I tried rainfield's code and I also got very different position count.

At the same time, I build a block strategy myself, and the position sizing setting is fixed.


I get the exact same position count when the symbol ranking block does not take effect, as shown below.



However, when I activate the ranking block, the position count will vary dramatically.



I tested Welath Lab Data and Norgate Data, and the results are the same.
0
Cone8
 ( 15.74% )
- ago
#10
Here are my results of that block strategy with the Ranking enabled - the min Position count was 3,461.


It's troubling that you're getting those results and even more troubling that I am not!
0
Cone8
 ( 15.74% )
- ago
#11
I just noticed that I has Max Open Positions set at 0 instead of 10. Trying again...

.. and no joy. Similar results for all 50 runs.

0
Cone8
 ( 15.74% )
- ago
#12
I'm still thinking about it, and it does seem possible that reading the ranking values from the bars.Cache (which is what happens in the Strategy code) could be susceptible to a threading problem during the parallel processing done for the MC runs.

Maybe Glitch can provide a non-Parallel option for Monte Carlo, like that available for the optimizer.
0

Reply

Bookmark

Sort