So while sequential access and un-pickling/re-pickling market days of information between my hard drive and my RAM to conserve memory was a solution, there was a better one to be found.

I am now using Zope Object Database (ZODB) and its OOBTree class. ZODB is much more slick than my proprietary pickling scheme… and binary trees are much quicker data structures for what I am trying to do (I was using mere dictionaries before).

I could try to explain the advantage of ZODB and binary trees in plain English—but I would fail… I will tell this much: Any hard drive/RAM space concerns are resolved with this implementation.

The database implementation is working great, and now all I need to do is focus on finding new sources of market data to import. I’m very pleased with everything.

I have a lot of time right now, so hopefully I can refine at least 10 years of contiguous fundamental & daily market data as a start to begin coding my historical portfolio/screening tools. I also hope to write a user interface for these features (and let the data mining tool remain an ugly command-line tool as it should not need to be used again once a complete primary PZTMDB file is refined).