I recently had an (email)conversation with a prestigious research institute that asked me a series of questions about Complex Event Processing (CEP) technology as a platform for trading systems. They were interested in how it compared to traditional trading systems. Their questions were very insightful and I suspect many technology buyers have similar thoughts as they look to spend their IT dollars (pounds, yens, euros). CEP has been around for a number of years now, most notably in Capital Markets and it has been used to develop trading systems across many asset classes but questions still abound. Ask 10 people where they think it is on the hype cycle, and you’ll probably get little in the way of consensus.
Anyway with their permission, they have agreed to let me anonymously publish the question and answer session.
|Question 1||I think algorithmic trading platforms now face a “paradigm shift” as a change from traditional architecture (data is permanent while query is temporal) to CEP architecture (data is temporal while query is permanent).|
CEP does manage ‘time’ better than traditional systems, in that it understands that data is streaming as time is advancing. So the analytic functions it provides must handle the in-motion and time attributes of data in addition to the data itself (i.e. an average calculation is against a time window of the data set). CEP does this much better that traditional systems which are designed to analyze static data. I think the classic notion of CEP, or an emerging one given that CEP is still relatively nascent as compared to other technology platforms (i.e. appServers, messaging buses) is that it can only manage temporal data, specifically where the data is transient, enduring only for the short lifespan as it flows through a CEP-based application. For some CEP vendors this is true and frankly a limitation. CEP systems that define some measure of permanence to data such that the only difference between today’s market data and yesterday’s is the date are better suited to cope with the varying complexities of algo strategies and their analytical needs. As a result, CEP-based strategies can be designed, coded and tested in a more logical/easier manner. The strategy builder can focus on semantic logic (i.e. what’s today’s price volatility vs. yesterdays) and optimizations and not waste time coding data integration.
|Question 2||Traditional [trading] engines are those not using CEP. I wonder if there are some pros/cons about CEP engines? Can CEP engines run all the types of existing algorithms? I think there are some kinds of restrictions/limitations in CEP engines because of their architecture. Is it true? For example, CEP engines can’t run (or don’t run well) complex algos using some types of operations such as matrix calculations or so. (This is just an example. I don’t know if this is true or not.|
CEP is typically considered clear-box technology (as compared to traditional engines being black-box). The overarching definition implies a number of factors:
a) CEP as ‘clear box’ means you have to write/implement strategies.
b) It is a tool that can transform the IP (intellectual property) of a firm into actionable strategies.
c) No shrink-wrapped algos, but generally many examples as ‘best practice’
Of course no one wants to start from scratch doing this, so having the componentry necessary in the form of analytical functions, aggregations, etc. that have an inherent understanding of market data (trades, quotes, orderbooks, corporate actions) can improve efficiency and speed to get the end result.
CEP is not a panacea, it is not the ideal platform for performing Monte Carlo simulations for example. As I mentioned earlier CEP is attuned to processing data in-motion. Matrix calculations are typically performed on a static vector of data. Such functions are callable from CEP, just as most any mathematical functions can be leveraged from (for example) MATLAB or R.
|Question 3||Do CEP engines always run any type of algo faster than traditional ones? I think some algos suit well with CEP engines while others not Is it true?|
Agreed, CEP an handle many types of algos, some are not all that well suited. A goal of a CEP engine is to be a fast processing platform but allow algos to be built using more approachable tools that just coding in C++. A CEP-based graphical modeling tool provides a means to ‘open-up’ the construction of algos to traders, quants and other non-programmer types.
|Question 4||Will CEP engines replace traditional ones or coexist with them? Thinking of input/output relationship, I think CEP engines suit well with 1:1 relationship while traditional ones suit well with 1:n. Thinking of stop orders, traditional engines do as follows:1. Store hundreds of thousands of orders
2. Monitor prices changes
3. When a specific price is reached or passed (“input”), then run tens of thousands of specific orders in parallel (“output”)Do CEP engines suit well with this type of processing?
In all likelihood CEP will coexist with traditional systems in a cooperative manner. CEP is good at fast processing and managing small amounts of state. Trade signal detection is at the heart of alpha-seeking algo’s. CEP is well suited for that usage. It is equally good as a pricing engine, servicing a variety of (tiered) clients with prices/quotes in a streaming or RFQ model. CEP scales well to match today’s multi-core hardware, taking advantage of parallelism and threading.
The inherent requirement in the above example is managing the state for tens of thousands of orders. Those in-flight orders require a lot of processing to handle cancels, rejects, amends, fills, partial fills, just plain failures. However, this is not a use-case that is ill-suited to CEP. Logically, one models how “one” order is processed handling all the conditions and errors known. This can then easily be tested to ensure correctness and robustness using historical data in a backtesting environment.Scaling that ‘one’ order process to now process thousands concurrently simply means multiple parameterized instances. The CEP engine is responsible for ‘scaling’ to meet the demand (number of concurrent threads, etc.). The use-case is inherently about data in-motion (orders stream out/executions stream in) with support for command/control for cancel/amend.
CEP is no longer a niche technology it has hit the mainstream. But it is not a panacea or quick-fix solution for building trading systems or any other Cap Markets vertical solutions. For those savvy enough to recognize the merits of the technology, it is apparent where it fits in a greater technology stack combined with tick database, EMS/OMS functionality and visualization such that the whole is greater than sum of the parts.
Well there you have it, once again thanks for reading,
For an occasional opinion or commentary on technology in Capital Markets you can follow me on twitter, here.