5 possible problems when trading from a virtual server
Virtual Servers are a hot topic – especially for trading. Colocation at an exchange location is expensive – very expensive - and often done under quite complicated commercial contracts. A virtual server is easy for trading. A real server at Aurora (the CME location) is complex – you have to take a rack, and internet is not included. You can arrange internet, but it is a separate contract. A virtual server for trading is a lot easier – you just sign a contract with a provider who deal with the complexities. Classical – non financial – colocation is also possible, and here offers are a lot easier to navigate. For trading, for example, at the CME one can avoid Aurora and go with a normal colocation provider (the term colocation is not related to finance but actually means co-locating a server with your provider, today to put it at a datacentre provider). If one can live with a small delay – 3ms or less if chosen intelligently – one can get a better offer at a normal data center. Chicago for example – one can Aurora and go with Steadfast, for example. Still, a lot of options and a virtual server is cheaper. Companies like Speedy Trading servers cater specifically for the need of a trader. Choose right, and a virtual server for trading is easy to set up.
But: There are negative sides, too. There are disadvantages. Whether you can live with them or not – is a business choice for trader. This article list the disadvantages of using a virtual server for trading. We assume you are close to the exchange – maybe even collocated at the same facility. A virtual server for trading is easy, just pay. Let us see at the negative sides.
First one is security. A virtual server is hardly ever secure. The provider has access to the “hardware” because it is simulated, and if running on current virtualization software he can take a backup without shutting down the server. He likely will if he is good (to provide a jus seconds behind copy in case of a server crash, as we do for our trading servers…. Replicating them to a reserve system every 30 seconds). But it means the provider has access to the discs, can steal the strategies. Virtual servers for trading are easy – and if one uses a specialized provider even more so, but that also means the provider knows what happens on the server. A normal provider hosts 100.000 servers maybe – and does not know which do trading unless he does data analysis. A trading VPS provider knows his customers are traders. This is a theoretical security problem. Virtual servers for trading are easy – but that is one of the negatives.
Naturally one can mitigate them. A reputable provider is not too likely to risk his business for a small traders not proven algorithms. One can use all kinds of complex “enter password before booting” disc encryption schemes. Or go down to real hardware, not virtual, possibly running virtualization under hardware under your own control. But it is always a risk.
Storage and Disc Performance
A virtual server for trading is easy – and cheap. Disc performance costs a lot of money. At the end one needs fast discs (10.000 RPM or higher – there are (small) 15.000 RPM discs. Or go directly and full to SSD. This is what we do right now with the large servers in our company that host our infrastructure – and databases. A Raid 5 of 750GB SSD gives a performance still 100 times faster than discs. Because this is what we talk of. Discs are fast (in MB/S) only when one stream is delivered and it is continuous. Once the head moves, performance goes down brutally – on average we can target 300 to 400 IOPS (IO Operations per Second). The enterprise SSD we use (Samsung 843T, 960GB model reconfigured to 750GB for more endurance) guarantees 35000 write IOPS (at 4kb fully random). That IS 100 times faster. Read is even more differential.
This is hardly an issue when trading normally as one does not start the machine often, and trading software is not IO bound most of the time… and that can be optimized. It is mostly an issue when patching. Updating your trading virtual server can turn into a very long job. As can software updates.
There is only one way around that and that is making sure that the disc performance of the host – not the virtual machine – is suitable. For a high price trading virtual server I can expect at least a SSD backed solution (using an SSD as read and write cache), or even a full SSD solution. Both do not cost that much – and make a great experience. If not – then one should really make sure to not cause a lot of IO and do things like patching on weekends when the market is closed. One should also make sure that the virtual server has enough memory to not excessively swap out memory to disc – a cheap virtual server will be memory limited and IO limited and that may be a deadly combination.
Virtual servers are cheap. For trading they can turn expensive fast, when the performance needed is not there. After all, like the IO budget of the disc, the processing power of the cores is distributed among the virtual machines. Trading is not normally a high performance issue – yes, you need speed, but not a lot of processing power. Algorithms are easy, processing power only comes into play with backtesting, and even more with optimization. A virtual server for trading may be cheap, but how cheap it really is depends on whether it runs good enough. A very cheap provider may sell you a virtual server for trading and – it may then be so oversubscribed and people try to run backtests that you get into problems.
How can this be avoided? Instead of going for a cheap virtual server for trading, one should choose one that has plenty of CPU reserves with a provider that has clean policies for oversubscription. If one splits a 4 core Xeon (with Hyper-Threading) among a maximum of 8 virtual machines that get both two cores, then this still means I can get a “fair share” – which is 2 half cores. That costs maybe more money than a super cheap virtual server, but it is something that really can pay off. Generally – never go for a virtual server with only one core, as windows can use the second core to offload for example IO loads to that core. This makes things a lot faster even at startup.
Internal Clock Reliability
You run a virtual server for trading? Forget your clock accuracy. There is nothing that can be done here. When using a virtual server the clock will fall behind – and be corrected forward by the integrated drivers from the host. A virtual server for trading has a good time source possibly in the data feed, but a log with the local time may be off. By seconds. Fast. The clock in a modern OS is loaded from the hardware once at start, then running in an interrupt. Bad news is – a virtual server has skew because once the interrupt happens the virtualization layer will schedule the virtual machine to get a slice on a core…. Once it is free. The more the CPU is used, the larger the skew. But even without significant load, it is there.
One can easily handle this by knowing it. Prefer using the timestamp from your signal provider. A good provider sends a good timestamp – Rithmic for example is very accurate. And use that as basis for logging. Awareness of the problem is most of the solution here.
Network Bandwidth and Latency
Similar to clock skew, you may have latency and bandwidth variations. When a network packet comes, the VM is scheduled – IF it is scheduled immediately. The server may wait a small time so he can handle 2 or more packets for a virtual machine at the same time. Hardware helps – SVR-IO is an Intel technology splitting the physical network card for direct use by virtual machines. A good virtual server provider will use that to reduce latency for trading servers (and CPU load for non-trading machines). But that is not all – most host servers will be connected with 1 gigabit ports that are shared between the virtual machines. Load on the network means latency. At the end, even with Quality of Service control one has to accept variations.
How can that be handled? Not at all – or by not putting anything on a virtual server that cannot handle a small sub-millisecond delay. If you get a virtual server specifically for trading it should have a “no public hosting” policy – and a “no download” policy to make sure the line is not clogged. If you cannot deal with small variations in latency – get a physical server. One that you control.
At the end…
A virtual server for trading is significantly cheaper than a physical server. This can be good, but one has to be aware of the limitations a virtual server has. Trading is not different from any other application in this regard. The good news is that there are some really nice providers for virtual servers for trading that can locate at the exchange.
At NetTecture (the company behind trade-robots) we do not use virtual servers that we rent – we rent complete hardware (or own it). We do use virtual servers for trading - on our own servers. This way we can handle most of the possible negative aspects of running a trading system on a virtual server.
And you? How do you host your automated trading systems? Which provider does work for you?