Check the Time: Majority of Firms have Time Synchronization Errors

Automated trading and historical analysis depends on knowing the time of events and the order in which they occur. With inaccurate time stamp many enterprises are at risk of unsuccessful trading strategies, even falling out of compliance. Unfortunately, the problem is all too common, and many firms would be surprised at the extent of the damage.

Victor Yodaiken, is CEO of FSMLabs, an operational software firm that offers the TimeKeeper, a hardware or software tool to track multiple time sources, detect and attack bad data sources, and generate automated failover. He says with the standard tools out there are generally a couple milliseconds off. In financial services firms are on a 10gig or 40gig network, but most devices still being sold are 100 megabits per second, causing data to slow, and results in a lot of errors, he explains.

"Traditional providers have not kept up with the technology, link management, or support for the network speed," he said in an interview. "Even people who are not on the cutting edge want to trade at a higher rate, and when you try trading at a higher range using machines all over the world, time becomes critical."

Essentially, when computers ask the operating systems for the time it needs to be close to synchronized with the government run official time which comes from a relatively weak, albeit precise, GPS satellite signal that isn't always accessible. The next challenge is getting that signal to the application code across hundreds and thousands of computers around the world.

"It's a classic computer science problem, it seems trivial, but getting it to work is really difficult," says Yodaiken, "Currently, the state of the art out there is very limited. People have inherited timing infrastructure that was built up without much planning or thought, and the large majority have inadequate infrastructure. The situation is usually terrible and much worse than they understand it to be."

Even with that connection accomplished, there are common scenarios that impact the accuracy and distribution of time. Deliberate sabotage (spoofing), an unplugged cable, or even overcast weather that disrupts the satellite's low power signal. Equipment can break, routers can be disconfigured, unnoticed for extended periods. Imagine a network device, somewhere in basement, with hundreds of feet of cable to an antenna on the roof. A lot of physical things can go wrong.

It's difficult to detect these errors. According to Yodaiken people set systems up and believe whatever it tells them. "They say to us things like, 'it says it's within 12 microseconds,' but it's hard to test that and know what it really means." In one case, after setting up their TimeKeeper application, a firm came back and said 'your software doesn't work' because it seems to be going haywire, "But we looked and saw the time feed they relied on was wrong. Every once in a while it would jump forward a couple hours [HOURS!]. Nobody had noticed."

In another example, "We were in a big company and they were complaining about time problems. It turns out IT was telling them they had a high quality time source, but our software showed an antenna wasn't working, also that it was sending time from NY to Holland every now and then… A lot of things can go wrong."

Another common problem, Yodaiken says, is that people lock down networks and say we only take this kind of information packets for security reasons, and they don't know about time particle packets. When they try to get feeds from data centers, they'll find routers have been misconfigured to not carry protocol properly.

False Correlations

Algorithms are largely built based on trends and correlations of observed events. Developers are looking for relationships that suggest A leads to B leads to C. But if the software is misaligned, and time the things are happening is not right, you'll miss real correlations and get fake ones, throwing off the entire algorithm.

Having accurate time stamps is also handy with regulators. "Forensic records can show customers and regulators that they did the right thing. If timing is off, it may look ilk you're front running but you're not," explains Yodaiken. "People on the up and up are very concerned with being able to preserve a high quality audit trail of time. We can give them multiple time sources, which nobody else can do, and we can show GPS time that agreed with this, and we show we did due diligence and kept the time as accurately as possible. Whereas, with other technology you don't really know. 'My software did this, but where did the time come from? Was it a hack? I don't know. Did anything go wrong? I don't know?'"

In Search of Solutions

For an industry that lives and breathes the mantra 'time is money,' the failover is increasingly weighing on the industry.

"They know there's an issue and want to understand it." They see their systems are inadequate, extremely delicate, with no records, and no trace of what's going on. "With all this technology, you can get silent failures very easily - we've had customers discover using this technology that they had a place offline and nobody had noticed. Systems were just drifting. We see people who think they are getting high precision when they're not. We see some patchy distribution times, - not distributing to everyone who needs it. I will say that there are a lot of companies who don't know how badly their data is."

Yodaiken reports a large increase in inquiries over the last two years, and perhaps a 50% increase in sales the last year for the time stamping technology. "There's been a phenomenal pickup, and it's much wider. We used to only get high frequency traders and other technologically advanced people - and now we're getting inquiries on the business level… Due to regulation people are much more worried," he adds. "They want to make sure components are solid." Becca Lipman is Senior Editor for Wall Street & Technology. She writes in-depth news articles with a focus on big data and compliance in the capital markets. She regularly meets with information technology leaders and innovators and writes about cloud computing, datacenters, ... View Full Bio