Trading Engine Struggles

Around a year and a half ago, I decided to build my own trading engine from scratch. Maybe it was the lure of hands-free investing while I work my 9-to-5 or the thrill of tackling an immensely challenging problem that drove me to that decision. I wanted to document my journey so far and hopefully save someone else time and money before they attempt to build their own automated system or use a 3rd party platform, regardless of whether it's related to trading or not. I went through a gamut of platforms out there from using fully supported platforms like Interactive Brokers (IBKR) to open-source, semi-implemented platforms like QuantConnect. Each platform trades away the control or performance you actually need for convenience you don't, and that tradeoff is ultimately what pushed me to build from scratch.

My first attempt at getting something working was using a fully supported platform, TWS, through IBKR. This is their system that is intended for automated trading, but involves setting up their software locally then establishing connections to their feeds and running your code to integrate with their platform. The first struggle was just navigating the nuances of their application setup.[1] This is not an easy to read API, but rather layers of documentation obfuscating the core functionality. I was eventually able to connect the feeds and query historical data, but it just wasn't practical especially if I wanted to host the setup so it can run separately from my personal computer. I also ran into data issues where I wasn't able to capture trades data. Some brokers only provide quotes, so you're only able to process what people were willing to buy or sell, not what they actually ended up buying or selling through real transactions. The TWS python client was also not well abstracted, so their methods had many parameters and many options per parameter, none of them typed with Pydantic or some other type helper in python. This made it extremely cumbersome to develop as I had to constantly reference the documentation for what each option meant and if I needed to use it at all. They have since updated their API and it looks slightly better, but overall, it is still difficult to navigate and make any real progress let alone reveal a clear path to a self-hosted solution.

From there, I found my way to QuantConnect.[2] This platform tries to abstract the hosting of your algorithm so you can focus on just strategy development. In many ways, it's a good product, especially at a $60 per month subscription fee for an individual researcher plan. I was able to backtest strategies with daily data just fine, but as soon as I tried incorporating lower timeframes at the intra-day level, I ran into a problem. Speed. It would take at least 10 minutes just to backtest a year's worth of intra-day data if not longer and that's with optimizing on only pulling the intraday data on the days I was interested in that level of granularity. I probably burned through months of development time just waiting for my backtests to complete. The platform also didn't have much support for external packages, meaning you weren't allowed to simply conda install or pip install third-party libraries. You have to either create them yourself internally or put in a request to have it made available by the maintainers. This was another severe limitation. I ended up creating a whole work-around by setting up an API Gateway with lambda functions that would obscure forbidden POST calls to third party platforms as GET calls just so I wouldn't have to copy client code directly in my project as well as all the other required dependencies that simply were not available. In the end, QuantConnect did save time on the deployment side, but cost even more time on the development side especially if you wanted to do anything remotely complex or data intensive. The platform does have much more robust support that solves some of these issues but only for enterprise customers, at a $5000 yearly fee. The further I explored the platform, the more I felt it was more like a toy sandbox that pushed you towards higher fees if you actually wanted to create a comprehensive, robust trading algorithm, despite it being sold as a democratizing platform.

After some searching, I found a broker API with a clean enough surface worth using, TradeStation. The one drawback with their platform is they have no officially supported client which would have been helpful given their recent v3 API upgrade. I decided to build a client in python here, but after considering the need for performance necessary to process trades data, I realized I would be blocked in the future. That led me to start exploring Golang as it gave me the performance I needed. It also fit well for building a CLI which I can easily host and run on my own ECS instances with minimal dependencies. The last thing I'd want is to accidentally upgrade a python library that causes unexpected behavior. I'm still in the process of creating a Go client here, but have the general shape and am able to handle basic calls.

The trickiest hurdle so far has been the engine design. A few of the challenges I've had to work through:

• Startup and shutdown websocket connection handling when updating the universe of stocks to trade
• Recovering from a service outage that can leave the algorithm out of sync with the external broker, impacting order tracking
• Time nuances between backtesting and live trading, like keeping track of relative time and time zones

Navigating all these hurdles, while creating a modular design with clear interfaces that enable me to choose different data sources and stream them while minimizing the number of changes I have to make to the codebase has been a challenge. Not to mention accumulating the backtesting data and transforming into parquet and running into OOM errors when trying to run AWS Batch jobs as cheaply as possible with Fargate Spot. Thankfully, I set up a Slack bot that pings me whenever the job fails, but I need to find a way to incrementally transform the raw compressed data to store in S3 and query with Athena, not an inherently straightforward task.

Despite all these challenges, at the very least, I've found a clear path forward where I have full control over how I want the trading engine to run. While it might require custom libraries and deployments, I'm no longer limited in the ways mentioned regarding 3rd party platforms. I've already seen significant performance benefits from using parquet and Athena along with Go to run test strategies. Simple daily data that used to run in 20 seconds now runs in 2 seconds or less. Perhaps it was my hubris speaking when I first started on this journey. I thought it was going to be simple, just connect to the data source and then have all the flexibility of writing an algorithm in code. After experimenting with different designs for the engine, I'm close to having the initial components set up and ready to receive live data. I'll post again soon with updates.

References

1. TWS API Documentation — Interactive Brokers
2. QuantConnect — Algorithmic Trading Platform