пятница, 3 марта 2017 г.
четверг, 2 марта 2017 г.
среда, 24 февраля 2016 г.
понедельник, 4 января 2016 г.
суббота, 10 октября 2015 г.
Based on Trading Show New York 2015 presentation.
This post outlines some ideas on how particular changes in quality assurance might affect the trading technology. There are lots of discussions on Internet-of-Things, Cloud and Web. However, there are other things in software verification and they are a little bit closer to our area - exchange and brokerage platforms.
These are some interesting QA trends:
- FrAgile process
- Crowd-sourced testing
- Formal methodologies
- Cognitive technology
Agile methods are very popular in start-up and web-industry. Agile methods do not necessarily result in better products, but they can make software developers much happier. Please refer to the great opening keynote address during Google test automation conference GTAC 2011 – Test is dead, to learn about enhanced version of agile – FrAgile development process.
As Reid Hoffman said: “If you are not embarrassed by the first version of your product, you've launched too late”. Test ideas first to find the right product, before building the product right. Have many users by the time your competitors will have a well tested product.
Large user pool can help you to use Crowd-testing. Give your product to users and collect immediate feedback on your software. To do it efficiently, you would need to rely on sophisticated code instrumentation and data collection. What is the point of applying crowdsourcing if you are not capable of efficiently processing its output?
But can we really use FrAgile process and Crowd-testing in finance? Can we allow an algo going berserk to burn money and face prosecution? Maybe what we need is the very opposite to agile?
There are areas like transport, medicine, nuclear power that have more rigorous controls. Quality assurance methods evolve there. Advances in model checking allowed PROVING as opposed to ASSUMING that something will work. There was a post in the TABB Forum a few months ago questioning why are we not using mathematics and computer science similar to what NASA uses to design safe autopilot? Is not it embarrassing that we have outages in the financial markets at the same time when spacecrafts are gliding over the outskirts of the solar system?
We can invest heavily into formal methods to have more stable financial software. But, the thing to remember is that despite of all the formal methods the New Horizons probe had an outage that knocked it out of contact with Earth for three days. To expect that your systems will break is the only way to improve the quality of your software.
So what's next? Can we learn something from other industries? Be both faster and safer. To understand how to do it lets look into the Doomsday scenario. Lets talk about people who had severe problems with technology - Battlestar Galactica and Sarah Connor.
The only reason the characters were able to make it until the end of the second season is that they were relying on the very same technology that was trying to kill them. They were protected by systems having the same level of sophistication as those they were struggling with. This is the mentality that we should adapt. When we use a complex platform, we need to build software to test our software. And you do not want your risk control and test instruments to be inferior to what will hit you. Having a good robot on your side is the only way to survive the robot apocalypse.
So, what's the impact on the trading technology? Developing testing instruments and monitoring tools should has the same priority as the trading platform itself. It is possible to use agile methods and continuous integration if your system is designed with testability and risk control in mind. At the very least you can deploy, start, restart and configure it automatically. Also, it is good to be able to shutdown the system if necessary. Do not expect formal artefacts from the agile process. Have a parallel stream instead to develop the necessary test harness. Software engineer in test approach proposed by Google is more or less building software to test software.
We can also use formal verification, theorem proving, static analysis and other things. But, the software will break anyway. The absence of adequate monitoring and kill switches is what turns a problem into a disaster.
What about audit and regulatory requirements? Develop passive testing tools. They do not generate any messages themselves. They capture the traffic and store it for analysis. Passive testing tools can gather all the evidence you need.
Ensure that your trading system can co-exist with the tools. Passive testing tools can also serve as both surveillance and client on-boarding systems. You can use them for the crowd-sourcing as well. If your test environment is opened to others and you have a strong passive test capability you'll be able to get a lot of valuable feedback without even asking for it.
LSEG's Technology services division incorporates MillenniumIT, GATElab and Exactpro. As a result, we have access to a unique set of software from all three companies and can use it to build test infrastructure for any system. Lets consider testing trading algos. We can deploy one or several scalable matching engines to serve as market simulators. We can deploy GATElab’s algopath or Exactpro tool called Minirobots to act as market participants. They can both replay some data and at the same time mimic a realistic market impact. In addition, we can have latency and front-running proxies for better tests diversity.
It is possible to take the next version of the algo and put it to the test. However, what is the point of running it once? Markets are not deterministic. Good simulated markets are not deterministic either. We will get a slightly different result every time we run the tests. You need to repeat your tests multiple times to assess the quality and efficiency of your software.
This is where everything comes into place:
- Efficient restart and testability to enable unattended test execution
- Machine-readable specifications to synchronize all the tools and formally iterate the possible tests
- Passive testing capabilities and potentially market surveillance to capture and store all the relevant metrics (both P&L and system related ones)
You can run as many tests as you like and pass your system through a wide range of scenarios. But, we all have limited time and also limited hardware. It is worth mentioning now two more quality assurance trends.
To select efficient limited subset of tests from an infinite number of possibilities, cognitive technology and mutation testing are there to help. It is possible to use them to find the optimal test scenario subset for your trading platform. They can also help you in enhancing the test set along the changes in your system and in surrounding markets.
Exactpro Research Page - http://www.exactprosystems.com/research
EXTENT Trading Technology Trends and Quality Assurance Conference - http://extentconf.com
Test automation tools allow simulating connected automated systems, running tasks in an unattended mode and generating load as in real-life market environment. The process of selecting a tool to test a trading or post trade system usually starts with looking at generic testing solutions widely used in other industries. No one has been fired for using standard commercial solutions, but the problem with most of them is that they are targeted at desktop and web interfaces. However, integration with trading APIs usually requires developing connectivity and session libraries as well as a framework that would allow to uniformly define message contents and expected results. Essentially, one will need to build one testing tool on top of another.
Exchange testing tools evolution
The are a few specialized testing tools available that already support FIX, FAST, ITCH, binary exchange protocols and have expandable architectures. The user specifies what to send into the market and what the expected result from every gateway should be. This process is called active testing. If the tool fits the purpose and the QA team is capable enough, thousands of test cases to validate most of the exchange platform’s functionality under test can be prepared and maintained.
The execution of huge regression libraries produces massive amounts of data. Additional insights can be obtained from analyzing these large data sets. The next evolutionary step in test tools is passive testing where one uses non-intrusive test tools that capture all messages generated in the system, normalize them, and store them for processing. Passive testing tools can help obtain valuable results about the system and the test library coverage. The Exactpro team has also found that passive test tools are invaluable in supporting client on-boarding and certification. Clients connect to the exchange platform and perform the necessary steps, while the tool captures these activities and produces pertinent client self-certification reports.
One can create a good functional testing tool and a scalable load testing tool in order to generate hundreds of thousands of messages per second and stress one’s platform to its limits. However, there is another area in between - intermittent defects that do not manifest themselves every time the tests are performed, potential black swans. Such defects are at the confluence of the functional and non-functional testing. To trigger them, one needs to repeat the tests many times and under load, and they can be easily missed in the enormously large load testing logs. These defects might not have severe technical consequences, though.
Ideal passive testing tool
What characteristics should a passive tool have to capture defects at the confluence of functional and non-functional testing? It should be scalable, stable, have minimal impact on the system under test; it should collect all events from all components in the system; it should efficiently capture all of the data for analysis; it should notify the user when something goes wrong; it should support the ability of the user to configure scenarios that trigger such a notification; finally, when something indeed goes wrong, the user should be able to restore the transactions around the event.
An ideal passive testing tool for exchanges functions exactly like a production monitoring / market surveillance solution.
Production Monitoring System
A good active testing tool should be scalable, stable, connected to many test environments, and easily configurable. It should be able to validate data consistency across various feeds, store the audit trail, and be capable of executing complex scenarios.
Ideal active testing tool
The above is clearly an algo-trading system. Instead of running Excel in and getting Excel out, connect something that behaves like a market participant with an algo engine.
Production algo-trading engine
Is that it? Can the test tools of the next generation be just algo-trading and market-surveillance platforms? Not really. There are two distinct reasons for that. First, imagine trying to fit several algo engines and an additional monitoring solution into your budget. It will substantially increase the cost of testing. The other limitation in using ordinary real-life systems is that their intention is to do what is right. However, the purpose of testing is to find what is wrong. Testing tools of the next generation will be very much alike production systems, but deformed and mutated to decrease the hardware footprint and to enable all possible permutations.
In a more distant future, there might be test tools based on machine-learning techniques. However, they can be enabled only if the test tools of the next generation are scalable and mutable. As singularity is not here yet, human intelligence is still required to create and operate test tools and large regression libraries. The best part of any testing solution is the human mind behind it. Exactpro builds software to test our clients’ software and has developed tools for trading, clearing and surveillance systems testing. We constantly work on evolving our test tools, so if one ever needs a testing tool of the next generation and/or a comprehensive quality assurance process, we can help.