Making the Case for Automated FIX Certification

On October 5, 2012, a broker trading on India’s National Stock Exchange placed 59 erroneous orders, triggering a flash crash that wiped nearly 60 billion dollars from the value of some of that country’s largest public companies. It happened just 2 short months after another trading glitch cost Knight Capital to lose $440 million in just 45 minutes. These are just two in a series of trading incidents that have damaged the markets over the past two years.

Candyce Edelen, PropelGrowth

Many blame human error combined with high frequency trading (HFT) for these issues. While regulators and politicians can debate the intrinsic value of HFT, they can’t debate the importance and necessity of adequate testing and quality assurance. In many cases, the regression testing done by exchanges and brokers before releasing new functionality is woefully inadequate. The release windows for new functionality or bug fixes is often so short that electronic trading teams don’t have enough time to fully test. For example the Knight glitch happened immediately after Knight modified their systems in response to new functionality being released by the New York Stock Exchange. They had only 6 weeks to determine requirements, make the necessary code changes, do internal regression testing, test with customers, and test with the exchange before NYSE put the new features into production.

Much regression testing and client connectivity testing is handled manually. Manual tests generally run through normal trading workflow – sending orders and getting executions to make sure both systems process the flow properly, testing certain fat finger errors to ensure that risk controls work properly, and testing the system’s ability to handle a disruption in network connectivity. But they don’t usually accommodate “corner cases” – events that don’t happen frequently, but that can be disastrous if they occur – those black swans that are hard to predict, difficult to mimic, and nearly impossible to replicate manually.

Recently some large broker-dealers and exchanges have been automating more of their regression testing and client certification as a way to reduce systemic risk. Automation also tends to decrease the time required for client on-boarding, making it useful not only for risk management but also for shortening time to revenue with new customers.

The client on-boarding process takes several steps. First, developers from the bank or exchange and the client have a conversation about the specification – a document that describes the computer language that their systems will use to process trading workflow messages. The developers then configure their trading system and order routing gateways to conform to the specifications. Once that’s done, the process is handed over to a quality assurance (QA) tester at the bank to start testing connectivity and the ability to transmit orders. This QA person or on-boarder holds the client’s hands, walking them through each of a long series of tests. Once the testing is successfully completed, the on-boarder verifies certification and transfers the client to a FIX support team.

We wanted to know more about this process, so we talked with a senior FIX support engineer from a tier-one broker-dealer who wished to remain anonymous. His bank automated the client testing and certification process in June, 2012. The support engineer explained, “We automated the entire testing process, allowing it to be done independently and hands-off for my team.” Ordinarily, this team needed as many as six man-days to certify one client, mostly because every test potentially identifies an issue that needs to be fixed by one side or the other, resulting in the teams needing to stop testing, go make the fix, and then start testing again. According to this engineer, automating the process “reduces days of hand-holding and lets the client go at their own pace. It’s very helpful in global environments because we don’t need to respect specific trading hours. It saves the clients a lot of headaches as well in terms of not needing to schedule testing during market hours.” This bank chose a vendor product for automation. The support engineer explained, “We use a vendor-supplied product called FIX Conductor. It gives you an entire FIX department in a box.”

FIX certification is intended to make certain that counterparties can communicate without error. While one of the greatest strengths of the FIX Protocol is its flexibility, it does allow for different interpretations. This means the trading partners will inevitably need to make changes to their engines and/or applications during the testing process, and sometimes these changes can be substantial. Users test their ability to correctly process trading workflow messages, verifying that the system correctly handles order submissions, cancels, replacements, partial fills, allocations, etc. The trading process from order initiation straight through to matching, settlement and reporting is tested to confirm that the systems are configured and communicating correctly. They also test the ability of the market data systems to receive data without corruption and process and trade on market data correctly. Jacob Northey, Co-Founder of The LaSalle Technology Group explains, “The fundamental reason for certification is to mitigate the risk of trading errors in the live trading environment.”

Several industry sources have reported that exchanges often do the bare minimum of testing because the manual process is unwieldy and causes delays in time to revenue. Jacob Northey notes that the exchange which does robust testing is the exception, not the rule. For example, many exchanges perform 15-20 tests that focus on the most common trading situations such as order entry, cancel and replace. BM&F BOVESPA is one of those outliers. They are currently automating their certification process. According to Northey, “they will run 80-100 tests for each counterparty with 4-5 behaviors for each test.”

Direct Edge, the fourth largest stock exchange in the United States, is an exchange platform that has automated regression testing using Greenline Financial Technologies. According to Saro Jahani, Direct Edge CIO, “we created a fast and robust automated regression testing environment to handle most scenarios. It allows us to run multiple regression cycles 24 hours a day.” This automation has decreased the timeline for full regression testing from several weeks to a matter of hours. Jahani continued, “Our time to deployment reduced dramatically when executing a full suite of regression testing across twelve different protocols.”The difficulty for exchanges and broker-dealers testing manually is that every test must be configured and set up before the on-boarding process begins, and configuring these tests takes a lot of work. Northey said, “A quick testing professional would take at least 5 seconds to type an order onto the screen. To test completely, you need lots of orders, so it can take anywhere from a few minutes to fifteen minutes to set up each test case. If you’re automated, this same test can be done at order-routing speeds of less than a second without manual intervention.”

In addition to simplifying the process of setting up test orders, automation can also mitigate the human factor. Testing complex order types or workflows takes quite a lot of time to set up. As Jacob points out, “People don’t want to design tests that are hard to set up.“ Northey used the example of testing allocations. “A manual test might include only five or six orders and executions allocated to a single account. It might pass the test just fine, but it is not representative of normal trading conditions. Normal allocations can have thousands of orders and executions that are allocated to many different accounts. Testing this manually requires a great deal of labor and a diligent tester. Automating the test makes it easy to scale the load on the system.”

The complex tests are not done because they’re hard to set up and complex to execute – or even impossible to execute in a manual environment. But most trading issues will only crop up when the systems are stressed with volume and volatility. These high speed multi-dimensional issues can only be tested in an automated environment. Jacob Northey further explains, “Developers are under pressure to implement changes traders require quickly. Testers and on-boarders are under pressure to get clients in to production. The impact is that in manual environments, less testing happens, not more. If testing were easier to do, people would do more of it.”

1 of 2