Our Infrastructure in 2014-q4 - Hardware
Afer the first part if this update has described our own framework and software infrastructure at this moment, it is time to talk about our hardware. We have more than 50 strategy instances ready to be deployed (or in trading) and that only works when you have the hardware to actually run all the testing.
While the software is one side our hardware is the other. Investments in this area pay off – often quite directly by higher development productivity. Our systems are split into 2 different areas. Development happens in house – and we run a small data centre in the basement, with a 10kw power load (so: not so small). Trading happens from rented / owned machines in remote data canters close to the exchange. Close, not collocated at the exchange as we do not do high frequency trading and rather save on those costs. Chicago for example ha a machine from SteadFast with a very low latency – good enough for us – to the CME. We will ignore the trading hardware for the moment – this could possibly be elaborated on another post.
Our development hardware consists at the moment of 2 significant virtual machines running on our development backend and a significant set of machines for backtesting. These are:
A SQL Server Developer Edition with 48 GB memory and 8 cores attached. Running on Hyper-V it is not using the usual “shared storage” approach but has a significant number of directly attached dedicated discs. What is significant? With the current planned updates we are talking of:
6 Samsung 843T 960GB SSD in a RAID 50 for a 3000GB storage space to hold back test and optimization data.
2 Samsung 843T 480GB SSD that are reconfigured to 64gb each in a Raid 0 as temporary scratch space (TempDb). We may use more space later, but for now that is plenty. This data is very operation intensive, which is why we isolate it on separate SSD – we may well use a Raid 10 of more SSD here in the future.
2 Seagate 450 gb 10k RPM SAS discs as a mirror for the log file, which has a write cache on some of those Samsung 843T SSD, done transparently by our Adaptec 71605Q Raid controller.
Together this hardware makes sure we have ample capacity for data storage and processing. Before the last upgrades it was all based on Seagate SAS Discs (the same we still use for the log) and we had severe issues on the disc side, even though we only had the ability to store 450gb of data, around 2 weeks’ worth of optimizations (so we did optimize, then back test the relevant data pairs for archival, then delete the optimization data to reclaim the space). How much data do we deal with? That depends on strategies tested, but on our top days we are adding about 0.5 billion trades when doing optimizations. Each successful trade (we also record trades that do never get a position) had a minimum of 2 orders and this a minimum of 2 executions. That is a lot of data.
This serves a nearly 1tb archive of pre-calculated tick data for our CME development. These are served from an array of 300 GB Velociraptors (yes, a little older) in a Raid 10 consisting of 8 discs. During back tests – which are quite IO intensive – we observed serving around 600mb of data quite continuously – that translates into 6 gigabit, possibly only because this server is connected to the network with 10gigabit. Due to the way our network is structured we are – and we know that – limited in network bandwidth. Something we will have to work with soon (by upgrading some switches and bundling some 1gigabit links going to our blade centre or by adding more compression to our data, which we are working on).
Our Comuting Grid
A cluster of currently 20 machines that are doing back testing. We call that our “Grid”, from grid-computing. These range from 8 core AMD based machines to a 12 core high end Intel Extreme machine – but the bulk is consisted of our ”Monster”, a Dell M1000E blade center that we got for a good price. It has 16 blades, each adding 8 cores to our testing capabilities. Yes, it is outdated – but when the time comes we will likely start replacing it with Broadwell based processors. Then a blade will likely add 40 threads instead of 8 to our capabilities. The 3 AMD based machines form a specific sub-cluster in that they have additional software installed to deal with our Nanex data tapes. Any extraction work for data is scheduled exclusively to them.
It is likely that we will start to slowly upgrade these in 2015 - once the new Broadwell Xeons become available we can likely replace this whole setup in a year for the cost of saved electricity. All our machines are very power hungry and that results in a quite serious electricity invoice.
On a pure infrastructure layer we can add another nice not too expensive used addition – a 16kw USV giving us about 1 hour operational time under our full load, which is around 10kw. Being based in Poland we sometimes do experience multiple small power outages during the week, and this has simply saved us many times from system crashes.
Is that a perfect setup?
No. Our strategy developers manage to regularly schedule enough for the grid to take a week to process. That said, they would likely do the same on 10 times the hardware. More modern processors would likely reduce our significant costs for electricity, though. Still – this is a case for “not yet” as we are awaiting the availability of Broadwell Xeon processors to replace our current infrastructure. On top, we are experimenting with using a SQL Server Analysis Service on a second virtual machine (and a separate physical machine) to provide more capable / ad hoc / interactive data analysis possibilities (like integrating back test results into MS Excel). But it is what we have right now.
How does that compare to your setups? Any ideas where we made a mistake? Any comments are welcome.