I have noticed that in certain situations, both the SMAC and Exhaustive optimization stops too early without executing the number of runs it was configured for. I originally thought it was being clever and only executing runs that were necessary to reach the optimal setting, but I see now that's not the case.
Example, I am trying to optimize from .880 to .920 by steps of .001:
But on SMAC, it only does 5 runs:
There is a much better value around .907 that it never finds.
If I try to do Exhaustive, it says there will be a total of 41 permutations, but also only does 5 runs, and never tries .907:
Am I doing something wrong here?
Example, I am trying to optimize from .880 to .920 by steps of .001:
But on SMAC, it only does 5 runs:
There is a much better value around .907 that it never finds.
If I try to do Exhaustive, it says there will be a total of 41 permutations, but also only does 5 runs, and never tries .907:
Am I doing something wrong here?
Rename
The SMAC optimizer ignores Step size altogether. This is normal. Step size is irrelevant to SMAC.
Perhaps you need to adjust the number of iterations on SMAC, but I'm not sure if that's going to improve your multidimensional model fit much more. For those cases where I don't like the multidimensional fit by SMAC, I typically try to refit them with the Particle Swarm optimizer.
You need to appreciate that all these optimizers apply a low-pass filter to the multidimensional vector space they fit so they don't get caught in a local minimum or maximum that's not optimal to the overall general solution. Yes, that reduces their precision some, but if it gets caught in a local min/max, then you'll have a less than optimal overall answer.
What you can do is reduce the number of parameters you're optimizing against, which will better smooth out the solution vector space.
Perhaps you need to adjust the number of iterations on SMAC, but I'm not sure if that's going to improve your multidimensional model fit much more. For those cases where I don't like the multidimensional fit by SMAC, I typically try to refit them with the Particle Swarm optimizer.
You need to appreciate that all these optimizers apply a low-pass filter to the multidimensional vector space they fit so they don't get caught in a local minimum or maximum that's not optimal to the overall general solution. Yes, that reduces their precision some, but if it gets caught in a local min/max, then you'll have a less than optimal overall answer.
What you can do is reduce the number of parameters you're optimizing against, which will better smooth out the solution vector space.
Your Exhaustive run obviously uses a step size of 0.005.
Yes, but why? You can see in the screenshot that it's set to .001. I just did a brand new test on a new strategy, and got the same behavior:
There is some sort of bug in WL8 that is causing it to use the incorrect step size. Is there a hard-coded limit of .005 as being the smallest allowable step size?
There is some sort of bug in WL8 that is causing it to use the incorrect step size. Is there a hard-coded limit of .005 as being the smallest allowable step size?
@mdosey - I just tried a test with a buy block that has your settings in Post #3. The strategy sells after 5 bars. I used the Exhaustive optimizer, and it showed 41 total permutations. All permutations ran in the optimizer - no problem. I tested with both a single symbol and a small dataset.
My WL8 build is 129 with the extensions I use all up-to-date.
Below is a portion of the optimizer tabular output:
My WL8 build is 129 with the extensions I use all up-to-date.
Below is a portion of the optimizer tabular output:
Paul, thank you for taking the time to test it. I just realized that I was using "Exhaustive (non-Parallel)" when the bug occurs. I switched to regular "Exhaustive" and it worked correctly. Can you confirm whether your test works correctly on "Exhaustive (non-Parallel)"?
Using Exhaustive (non-Parallel) I am seeing the same issue that you reported. It is stepping by 0.005 and not by 0.001. I get 5 runs like you reported.
Also: The SMAC optimizer is designed to work in high-dimensional parameter spaces i.e. strategies with more than one optimizable parameter.
If I am interested to see the influence of a single parameter I use the "Exhaustive" optimizer for my own "Learning and Understanding Mode".
Such an endeavor works best if combined with the IS/OS Scorecard that shows results for two different data ranges at the same time. A very good measure against over-optimization.
The IS/OS ScoreCard is available with the finantic.ScoreCards extension (https://wealth-lab.com/extension/detail/finantic.ScoreCard)
If I am interested to see the influence of a single parameter I use the "Exhaustive" optimizer for my own "Learning and Understanding Mode".
Such an endeavor works best if combined with the IS/OS Scorecard that shows results for two different data ranges at the same time. A very good measure against over-optimization.
The IS/OS ScoreCard is available with the finantic.ScoreCards extension (https://wealth-lab.com/extension/detail/finantic.ScoreCard)
I have this resolved for Build 130.
Your Response
Post
Edit Post
Login is required