There are some tough realities and compromises that have to be made in running all this code. First and foremost I was not able to get this application to run reliably in San Diego or Los Angeles (Where I am from). There is a reason that huge Data Centers are located in New York and Chicago; it's just more reliable to take the data feed from the source. When you send data across the country it can be affected by latency and interruptions. The problem with firehose data is that if you loose any of it in transmission you can't go back and get it again. If you have data loss, your data will have gaps that will not match the real time market. The side effect of this is that all your subsequent math will be different than the market. This is one of the reasons why big financial groups focus on the concept of "clean data". It's important to understand this concept because your going to be doing so many calculations down the line that a small error will become a large error.
Another concern that drove me crazy was trying to have the trade ideas as close as possible to the bar close of the period. For example, let’s say your trading on a 15 minute time period. Before you send out trade ideas you need to make sure the current 15 minute bar is closed. Then you start making mathematical calculation. When I started out it was taking 10 minutes to run all the application to generate the trade ideas, this was totally unacceptable. Two changes ended up working to fix this for me. The first was having the python applications send signals to tell the next application that the previous step was completed. The second change was application threading to utilize the massive processor core count on Amazon cloud servers. Each core on an amazon EC2 instance can run 2 threads so a 64 core server can process 128 stock symbols in parallel. The production code that is running today takes about 1 to 3 minutes (depending on trade volume) to fully complete and send out trade alerts. That's amazing for the 2000 or so symbols on the compatible symbol list.