BacktestComplete-inside-Meta-System

Then I'll submit a feature request to get access to the backtester class from a meta-strategy via C#.

#FeatureRequest

Maybe Glitch can plan to help you out in WL9 with that, but for now, the PerformanceVisualizer API would be the way to go - even if you're not going to use it for displaying anything.

In my testing BacktestComplete is being called for each component Strategy in the MetaStrategy.

@Glitch

First off, I'm using WL8 Build 151. Here's a test strategy that should write a file using BacktestComplete() if it's part of a meta-system, which isn't the case!?

CODE:

public class MyStrategy : UserStrategyBase
{
   //intialize the sum variable to zero, this only gets called once per backtest
   public override void BacktestBegin()
   {
      sumClose = 0;
      sumObs = 0;
   }

   //we add the closing price to the sum and increment the number of observations
   public override void Execute(BarHistory bars, int idx)
   {
      sumObs++;
      sumClose += bars.Close[idx];
   }

   //save the information that we gathered to a file
   public override void BacktestComplete()
   {
      double avg = sumClose / sumObs;
      string output = "Average Closing Price of DataSet: " + avg.ToString("N2");
      File.WriteAllText(@"C:\temp\FlagFileFor_" + Backtester.Strategy.Name + ".txt", output);
   }

   //declare private variables below
   private static double sumClose;
   private static int sumObs;

}

The issue is that Backtester.Strategy is null in the context of a MetaStrategy component. We can look into fixing that, but a workaround you can use now is:

CODE:

File.WriteAllText(@"C:\temp\FlagFileFor_" + StrategyName + ".txt", output);

@Glitch
For the above code if Backtest.Strategy is null shouldn't that trigger some exception? Could such errors be logged somewhere for debugging purpose? I have a metastrategy that contains ~20 strategies. Sometime there's error but it's not easy to figure out from which strategy and I had to make a copy of the metastrategy and remove one by one to figure out the one that caused the error. It would be quite useful if such errors from components could be logged.

QUOTE:
Sometimes there's error but it's not easy to figure out from which strategy

I'm not following. Can't your own catch{block}s handle these errors and include the StrategyName in their AddLogItem statement so which strategy that threw the error is properly logged?

For any AddLogItem statement (in your catch{block}), you want to include enough information so you can reproduce the problem later; otherwise, the error logs are useless.

@glitch
I'm using a custom helper class to export various backtester metrics to a JSON file. Should I assume that the backtester properties won't be correctly detected if this helper class is run within a meta-strategy?

CODE:

var sb = new StringBuilder();
sb.AppendLine("{");
sb.AppendLine("  \"strategyName\": \"" + JsonHelpers.EscapeJson(_strategyDisplayName) + "\",");
sb.AppendLine("  \"strategyDescription\": \"" + JsonHelpers.EscapeJson(strategyDescription) + "\",");
sb.AppendLine("  \"symbolUniverse\": \"" + JsonHelpers.EscapeJson(_symbolUniverse) + "\",");
sb.AppendLine("  \"benchmark\": \"" + JsonHelpers.EscapeJson(_backtester.Strategy.Benchmark) + "\",");
sb.AppendLine("  \"mode\": \"Long Only\",");
sb.AppendLine("  \"lastUpdate\": \"" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture) + "\",");
sb.AppendLine("  \"startingCapital\": " + JsonHelpers.ToJsonNumber(startingCapital) + ",");
sb.AppendLine("  \"backtestStartDate\": \"" + backtestStartDate.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) + "\",");
sb.AppendLine("  \"apr\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.APR)) + ",");
sb.AppendLine("  \"profit\": " + JsonHelpers.ToJsonNumber(profit) + ",");
sb.AppendLine("  \"profitPercent\": " + JsonHelpers.ToJsonNumber(profitPercent) + ",");
sb.AppendLine("  \"riskReturnMetaScore\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.RRSuperScore)) + ",");
sb.AppendLine("  \"sharpeRatio\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.SharpeRatio)) + ",");
sb.AppendLine("  \"marRatio\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MARRatio)) + ",");
sb.AppendLine("  \"recoveryFactor\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.RecoveryFactor)) + ",");
sb.AppendLine("  \"maxDrawdown\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MaxDrawdownPct)) + ",");
sb.AppendLine("  \"avgReturnYear\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MyYearlyAvgReturn)) + ",");
sb.AppendLine("  \"stdDeviationYear\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MyYearlyArithmeticStandardDeviation)) + ",");
sb.AppendLine("  \"avgBarsHeld\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.AvgBarsHeld)) + ",");
sb.AppendLine("  \"exposure\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.Exposure)) + ",");
sb.AppendLine("  \"maximumExposure\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MaxExposure)) + ",");
sb.AppendLine("  \"profitFactor\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.ProfitFactor)) + ",");
sb.AppendLine("  \"avgProfitPercent\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.AvgProfitPct)) + ",");
sb.AppendLine("  \"profitablePercent\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.WinRate)) + ",");
sb.AppendLine("  \"positionCount\": " + JsonHelpers.SafeGetInt(() => _backtester.Positions.Count).ToString(CultureInfo.InvariantCulture) + ",");
sb.AppendLine("  \"nsfPositionCount\": " + JsonHelpers.SafeGetInt(() => _backtester.Metrics.NSFPositionCount).ToString(CultureInfo.InvariantCulture) + ",");
sb.AppendLine("  \"maxMarginUsed\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.MaxMargin)) + ",");
sb.AppendLine("  \"avgEntryEfficiencyPercent\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.AvgEntryEfficiencyPct)) + ",");
sb.AppendLine("  \"avgExitEfficiencyPercent\": " + JsonHelpers.ToJsonNumber(JsonHelpers.SafeGetDouble(() => _backtester.Metrics.AvgExitEfficiencyPct)));
sb.AppendLine("}");
return sb.ToString();

QUOTE:
Can't your own catch{block}s handle these errors and include the StrategyName in their AddLogItem statement so which strategy that threw the error is properly logged?

Well, if I manually wrapped everything in try/catch and wrote custom logging to the status line, I might be able to track it down.

I usually build strategies with Blocks first, then convert them to C# and enhance them over time. Most of the code generated from Blocks doesn’t come with try/catch wrappers around every method, so in many cases there’s no built-in exception handling to surface issues like this.

What made this especially hard to diagnose is that the exact same code works perfectly when the strategy is run standalone. It only fails inside the MetaStrategy context, that’s not something a user would reasonably suspect.

WealthLab appears to swallow the exception silently. There’s no error message, no indication which strategy failed, and no stack trace etc. From the user side, it just looks like the strategy or some code that should run but “didn’t run,” not that a null-reference exception occurred because a property changed behavior between standalone and MetaStrategy execution.

It would be much easier for user if WL reports the exception somewhere with the strategy name / sub-strategy that failed.

I’ll work on improving that behavior, thanks for reporting this!

QUOTE:
What made this especially hard to diagnose is that the exact same code works perfectly when the strategy is run standalone. It only fails inside the MetaStrategy context,...

This sounds more like a WL bug to me. So what was the fix?

QUOTE:
WealthLab appears to swallow the exception silently. There’s no error message,...

I think that happens only in the main strategy code say for Initialize or Execute. It doesn't happen within custom Visual Studio libraries, which is where the bulk of my code (which rarely changes) resides. I would agree, error handling in main strategy code could be improved. But my main strategy code is high-level and short, so there's not that much logic to debug there.