The wxWidgets / wxPython library is very slick. It was not difficult whatsoever to create my GUI toolbar (pictured at right), and I do not anticipate any headaches with further UI features going forward.

Visual metaphors

A picture is worth a 1,000 words, so I’ll leave any explanation of the UI to my album below.

It was fun searching Google Images for the right icons to use on my toolbar. Unfortunately, I didn’t bother to track where I got any of them or what licensing they had, so I’ll need to remember to source new icons before any actual release of this program.

Tertiary data

I did succeed in implementing my secondary data binary tree, as articulated in my last post. I still need to write a few scripts that will generate smaller monthly- and weekly-granulated data sets as well as singular company data sets. These data sets I am terming tertiary data to reflect another generational step in refinement.

If I ever have to re-build my secondary data… I probably won’t. The process took 30 hours from start to finish, to iterate over each of the ~15 million keys in the primary data and to write these to the secondary data with certain quote information deleted and everything keyed by standard names… I can probably fold the process into the script I will write for creating tertiary data and thereby avoid this middle step altogether.

Database mutability

Thus far in the project, I have been aiming for a division between Midas Data Miner, which should import and write data, and Midas Backtester, which would only access database read-only, hopefully from faster-reading data structures that the former program creates (which become no longer suitable for write access as a consequence of optimization).

This remains a good plan, but one disadvantage of database immutability for the Backtester program is that it will not be able to cache any computed quote values… For example, if in the course of building a portfolio, I decide I want to screen out all companies with P/E ratios exceeding 15, this information will need to be calculated millions of times in the simulation. It may be beneficial (for future backtests) to cache the P/E ratio as a unique attribute (rather than calculate newly each backtest) or, better yet, to cache the True/False outcome of the screen (PE>15 predicate) so that the conditional does not need to be evaluated again.

So there are a few remedies: 1) Backtester can modify tertiary data sets directly (which should be VERY SLOW as these data sets will be written as slow-write / fast-read containers), 2) Backtester can keep a list of unique calculations it performs so that Data Miner can perform and save these calculations the next time it generates tertiary data, or..

3) Backtester maintains its own cache data set of stocks that pass a given screen on a given day. I like this solution the most… Basically, each time a screen is run for the first time (as part of a portfolio backtesting simulation), the basket of stocks that pass the screen are cached as a set keyed by (screener_id,date) in a new binary tree. This keeps Backtester and Data Miner independent of each other - except that Backtester should be able to detect when secondary and/or tertiary data has changed so that it can flush its cache if necessary.

Next steps

Just because my UI is looking so slick, I probably will fold all the scripts presently written in Data Miner into Backtester (under the aptly-named Data Import tab). I’ll probably do this first.

The first new feature I plan to tackle is the Database Viewer. I will first implement it as a quasi-spreadsheet that lists dates as rows, tickers as columns, and then colored cells. Each cell will be colored depending on the quality of the data available - i.e. green for whole, yellow for missing fundamental / price data, red for missing entirely.

Speed

I think I may be more productive writing for a GUI project, because accomplishments are very visible and thereby that much more gratifying… Hopefully I can make a huge dent on everything (everything?) before fall classes start.

/s/ Patrick