Review - Backtesting Automation Sequence Manager (BTSeqMgr)

Formal reviews of trading platforms and trading platform add-on software. We want YOUR review!
User avatar
Posts: 948
Joined: Thu Jul 26, 2007 7:20 pm

Review - Backtesting Automation Sequence Manager (BTSeqMgr)

Postby progster » Wed Jun 04, 2008 11:23 pm

Note: In order to ensure visibility of all media, be sure to join the Forum!

Backtesting Automation Sequence Manager (BTSeqMgr)
by Professional Software Solutions

Review by Progster, 6/4/2008

The TradeStation 8.3 optimizer allows a user to easily analyze hundreds or thousands of permutations of trading strategy settings applied to a stock, futures, or currency symbol. After the optimization process is complete, TradeStation (TS) provides a very convenient point-and-click interface to which allows the user to directly load any iteration back into the chart for visualization of that iteration’s trades and performance.

Choosing an optimization iteration to view in TradeStation

Unfortunately, this convenient optimize/visualize workflow can be applied to only a single symbol on a single timeframe, because the TS optimizer operates at the chart level and does not provide any means for changing the chart’s symbol or timeframe on the fly during the course of an optimization.

This inability to optimize on more than one symbol/timeframe is a limitation that TS user’s have either lived with or tried to work-around for years.

I am pleased to be able to report that an advanced, well-working solution is now available, Backtesting Automation Sequence Manager (BTSeqMgr) by Professional Software Solutions.

What It Does

BTSeqMgr allows the user to define optimization sequences, each of which encompasses not only strategy parameters, but also a list of symbols and a list of timeframes over which to optimize.

Then, overnight or over a course of days, BTSeqMgr will cause TradeStation to perform all the optimizations specified in the sequence definition.

BTSeqMgr will create a new workspace, and a new set of performance report outputs for every symbol/timeframe combination defined in the sequence, and will do so unattended.

As an example, here is a display of the output files produced by BTSeqMgr for the symbol AA, with the timeframe of each test being advanced from 1 minute bars up through 60 minute bars:

BTSeqMgr multi-timeframe minute output files

And since we defined the sequence for this test to use the list of Dow 30 stocks, this set of output files was repeated for each stock in the list – and all were generated overnight without human supervision.

How many symbols did you test last night? :D

The sections below will provide a tour of the product, and further discussion based on the experience of using BTSeqMgr for several weeks.

The Top Level Pane

BTSeqMgr top-level pane

The top level pane is a compact area which allows you to define new backtesting sequences, edit exiting sequences, view the set of iterations generated by a sequence definition, access the internet .pdf help file, start a sequence, watch the progress of a running sequence, and carry out various general aspects of program administration.

Above is a picture of the top level pane being used to select a sequence for editing.
Below is a picture of the pane reporting status while a sequence is in progress.

Backtesting Sequence in progress

The Edit and New buttons operate on sequence definitions as their names imply.

The Steps button invokes Excel (which BTSeqMgr assumes is present) to display an itemized list of the iterations that will be performed during your backtest sequence:

Backtest sequence step display

This is a good way to verify that you have created your sequence definition to produce the desired results.

The Skip button allows you to edit a symbol list which “subtracts” symbols from a sequence run. The skip list can be used to filter out problematic symbols without the need to alter already-defined symbol lists. It can also be used to restart an incomplete run partway thru a symbol set (by putting all the already-done symbols in the skip list).

The Summary button allows you to manually invoke the otherwise automatic option of creating an all-symbol results summary (such as seen in the first post here: Renko BrickAge BreakOut Strategy Research )

This turns out to be very convenient when the user has a reason to move or copy results files from their original locations into a single directory for organizational, archival, or summary purposes.

The middle two boxes display the status of an executing sequence and provide running commentary on the steps as they happen.

At the bottom of the pane are the remaining controls.

The Start button initiates a sequence, and while the sequence is running, this becomes a Stop button. Under some circumstances it also becomes a Wait button.

The Exit button quits the BTSeqMgr application, and the remaining buttons all do what their names imply.

The Configuration button opens a new pane with an extensive set of editable configuration items:

BTSeqMgr configuration edit pane

These config items all begin with sensible defaults, but can be changed to fine tune the application for peak performance and desired behaviors in the user’s environment. It is worth noting that the Configuration and Preference Items are a single set, global to the application (as opposed to being stored in the individual sequences).

The log is extremely detailed, and offers a step-by-step description of everything the program does. Though the level of detail may exceed your interest as a user, it has tutorial value, and also provides the ability for you to render useful feedback to the developer if ever an issue should arise.

BTSeqMgr log

User avatar
Posts: 948
Joined: Thu Jul 26, 2007 7:20 pm

Progster's Review of BTSeqMgr (#2 of 3)

Postby progster » Wed Jun 04, 2008 11:30 pm

The Backtesting Sequence

The backtesting sequence (or simply sequence for short) is where you choose all the symbols, all the timeframes, and all the strategy parameter variations that you want to run in a single test.

The sequence is thus a “master optimization set” which can do in one unattended run what would otherwise require dozens, hundreds, or even thousands of individual TS optimizations, each of which would normally have to be set up by a human after the completion of the previous one.

Sequences are defined in the sequence editor pane:

Backtesting sequence editor

The essential genius of the program is that it takes sets of Symbol and Timeframe specifications (in TradeStation these are normally attributes of a single chart definition) and makes them part of a compound backtest definition (the sequence).

BTSeqMgr provides you an interface to name, define, and save (in that order) these sequence definitions, and then provides the smarts to very cleverly “drive” TS itself to execute your definitions unattended.

View the Video: BTSeqMgr sequence editor in action

Special assistance is provided for each of the 3 major elements of a sequence definition.

Sets of symbols can be saved as symbol lists.
Sets of timeframes can be saved as (Time)Frame lists.
Add Parameter wizard provides very simple import of TS strategy parameters.

View the Video: BTSeqMgr Add Parameter wizard in action

BTSeqMgr Output

BTSeqMgr output files

For each symbol in your sequence, BTSeqMgr will auto-generate six output files, making it very easy to browse your optimization results.

Let’s consider the files generated for ADBE:

First off, BTSeqMgr saves the individual workspace used for optimizing each and every symbol. Simply open a symbol’s workspace to view the TS optimization report and reload any individual iteration into the chart to view the trades visually and view that iteration’s performance report.

Secondly, BTSeqMgr saves the “winning” performance report for each symbol as a .mht file. This file is ready for immediate opening in your browser to view the equity curve, MFE/MAE, etc.

Next, BTSeqMgr saves a .csv file containing a one-page summary of the performance stats for the winning iteration.

After that, BTSeqMgr saves an Excel spreadsheet for each symbol containing all the info for the “winning” performance report for that symbol. This spreadsheet information is as you would generate yourself by manually saving a performance report from TS in Excel format.

But what of the entire set of iterations? BTSeqMgr has got this covered too, with the entire optimization report for the symbol exported in .csv format.
For its final per-symbol act, BTSeqMgr saves a .posa file, which is a special file used as input for PSS’s even more advanced product, Diamond Backtesting with Walk Forward Manager (not covered in this review).

All of this info, is automatically generated for every symbol in your sequence, without any extra action required on your part. (Optionally, some of this info can be suppressed in the program configuration, if you wish.) This is truly automation in service of the market analyst (you!).

Finally, at the end of a sequence (after optimizing all symbols), BTSeqMgr
create seeks through the results and creates the _Summary.csv file. This file contains the basic performance stats for the winning optimization for each symbol all in a single file:

Multi-symbol summary file

This is a recent (and very welcome) addition to the program, indicating that BTSeqMgr is a living, evolving product, well-supported by its author, Burkhard Eichberger.

A description of the program’s output cannot be complete without mentioning two housekeeping items that are also created:

  • A copy of the original workspace loaded at the start of the sequence
  • The sequence definition file itself.
All of the above information is found in a single directory, unique to each sequence run, automatically placed and named by the program. Everything you need to document and reproduce a run in the future is present. I have to give two big smiles to this level of thoughtfulness :D :D . The real needs of the trader/analyst, today and in the future have been elegantly met.

User avatar
Posts: 948
Joined: Thu Jul 26, 2007 7:20 pm

Progster's Review of BTSeqMgr (#3 of 3)

Postby progster » Wed Jun 04, 2008 11:59 pm

My Experience

Without a doubt, BTSeqMgr has taken my whole-interval TradeStation backtesting to new, higher level. Using it, I have been able to generate directories full of test results overnight, or in a day or two, that would have taken me many days, or even weeks, to generate had I been required to specify and kick-off every symbol’s optimization personally.

My own previous way of solving the problem of walking TS through the optimization of many symbols was a custom-written Macro Express macro. I am happy to report that BTSeqMgr blows away my old technique in every respect.

Realistically, you need to expect and deal with the occasional hiccup. BTSeqMgr is very clever in its implementation, and I have had many “perfect” runs, but any program of this kind will sometimes be caught off guard when the application being “driven” (TradeStation, in this case), or another application active on the user’s computer, throws it a curveball.

There is retry logic built into the program, but it cannot catch every possible situation which may arise on your computer during the lifetime of the sequence run.

The bottom line is that there will occasionally be some missing output, or a run will be disrupted for some reason prior to completion. When you discover this, it’s usually a matter of a couple minutes to edit the symbols list, deleting all the symbols for which you have completed output, and then kicking off another run to complete the job.

After two or more partial runs, simply copy all the files into a common directory and use the Summary button to generate a new, complete summary file.

One thing to keep in mind is that you cannot minimize TradeStation during a backtesting sequence run. BTSeqMgr needs unimpeded access to the TradeStation application, and so you should do your best to arrange that ahead of time. While it is possible to mouse around and do other things during portions of the run as TS is in the throes of optimizing a symbol, basically you need to dedicate your computer to the test at hand.

Here are my tips for the best experience:

  • Download all your symbol data to Cache prior to running a sequence, and then run the sequence with TradeStation offline.

  • Start with some short sequences to see the program in action. Graduate to longer sequences after you are familiar with the program’s operation.

  • Take advantage of the multiple-license options to get an even better deal on this very attractively priced program.

Most importantly, get your TradeStation optimizations on the bus by using BTSeqMgr. :D

To Purchase BTSeqMgr:

Visit CFT's BTSeqMgr Product page

User avatar
Posts: 948
Joined: Thu Jul 26, 2007 7:20 pm

Re: Review - Backtesting Automation Sequence Manager (BTSeqMgr)

Postby progster » Tue Jul 15, 2008 11:51 am

Additional Videos: (click to view)

How to Complete a Backtest After a Few Missed Symbols


(Note: this method might still be applicable under certain circumstances, but most cases are now more easily handled by the newly built-in ability to start a defined sequence at a desired step - Progster, 20090602 )

User avatar
Posts: 948
Joined: Thu Jul 26, 2007 7:20 pm

Review Update - BTSeqMgr 1.4e

Postby progster » Tue Jun 02, 2009 3:39 pm

BTSeqMgr Review Update – June 2009

It’s been about a year since CFT’s original review of BTSeqMgr, and both TradeStation and BTSeqMgr have made important strides during this time.

In TS 8.5 (released Feb 2009), TradeStation added a Genetic Optimizer (GO). The GO allows huge parameter spaces (i.e. with millions of permutations) to be intelligently searched in an effective, non-exhaustive manner.
Because only a fraction of all the possible iterations are actually tested in a genetic optimization, the optimization completes much faster than an exhaustive optimization. Because of the genetic algorithm, the genetic optimization tends to converge on (find) good answers.

Even though the very “best” single result (the most “perfect curve fit”) will likely remain unfound, this is of little consequence compared to the advantages of completion speed and breadth of coverage that the GO provides.

Here is the TS Strategy Format dialog with the new GO Setup section:

TradeStation Genetic Optimization Setup

With the release of BTSeqMgr 1.4d in May of 2009, it is now possible to automatically drive a TradeStation Genetic Optimization across multiple symbols and timeframes. This is a powerful and enabling new capability for TradeStation users.

Here is the BTSeqMgr Sequence Editor dialog, showing the new Genetic Optimization setup area:

BTSeqMgr Genetic Setup

As can be seen, the BTSeqMgr user is now given a choice between genetic and exhaustive optimization. All of the TradeStation-defined parameters of a genetic optimization are available for definition and saving as part of a BTSeqMgr test sequence.

By setting the Generations and Population parameters to bound the number of test runs included in the strategy parameter optimization for each symbol/timeframe combination, the user may control (approximately) the overall time devoted to the optimization of each such combination. This has profound implications for how a trader/analyst goes about conducting a test campaign to validate or invalidate a candidate trading strategy.

The bottom line is that BTSeqMgr 1.4d and TS 8.5 together (and later versions of each) are a game-changing combination for TradeStation users.

With the TS GO, a very quick answer as to whether any given strategy has any possible potential on a given market and timeframe can usually be obtained in a matter of minutes.

With BTSeqMgr, a more detailed study of promising candidate strategies across multiple symbols and timeframes takes only a short time to define, and can then be run unattended on a dedicated computer. Overnight, we can now get sets of answers that might have taken weeks to get before.

Other BTSeqMgr Enhancements

Support for Genetic Optimization is not the only improvement which has come to BTSeqMgr.

Some of the other improvements since our June 2008 review are:

  • Added Show Command button to display the command sequence
  • Added Volume Chart Option to Interval definition
  • Added new Volume (share) based chart loading
  • Added saving of the BTSeqMgr.ini Configuration file to new SEQ_XXX folders
  • Added symbol skip list and edit button
  • Added new automatic summary when sequence has completed
  • Added new summary button to create summary for any folder
  • Added new automatic saving of the TradeStation Optimization Spreadsheet
  • Extended the sequence name display
  • Added explicit checking if other charts are opened before starting sequence
  • Added ability to easily establish a starting point mid-way through a defined sequence
Here is BTSeqMgr’s main display (as of Version 1.4e4), showing a sequence in progress:

BTSeqMgr 1.4e4 Sequence in Progress

Notice how there is now plenty of room for fully descriptive sequence names.

Here is a picture of the Excel display of sequence steps that is created after the ‘Show Steps’ button is pushed:

BTSeqMgr Sequence Steps display in Excel

You can see how multiple timeframe tests have been defined for multiple symbols. All these tests, for example, could be run overnight while you sleep.

If you define a set of tests that runs for a long time and you find that you need your computer back partway through, you can interrupt the process and resume it later on the first uncompleted step. (The at-a-location resumption process is an option when you choose ‘Show Steps’.)

BTSeqMgr Sequence Restart Choice Dialog

BTSeqMgr Sequence Restart Location Dialog
BTWFMgr_ResumeDialog_02.png (4.19 KiB) Viewed 11617 times

In Summary

BTSeqMgr is more capable and easier to use than ever. Combined with the TS Genetic Optimizer, BTSeqMgr delivers a new level of test automation and results generation that has never before been seen from the TradeStation platform.

Unless your market investigation and strategy testing needs are truly trivial (e.g. just a symbol or two, on just a timeframe or two) BTSeqMgr is a must-have “platform multiplier” that no TradeStation-based analyst or trader can afford to be without.

June 2009

Return to “Trading Software Reviews”

Who is online

Users browsing this forum: No registered users and 1 guest