Federated Market Transaction Audit Trails

Scenario Name: Federated Market Transaction Audit Trails

Scenario Authors: Harvard University

Brief Summary: More and more of the financial transactions on Wall Street (and indeed most other world exchanges) occur automatically and electronically. Trading firms employ a set of programs that monitor a variety of inputs, whether they be stock prices, earnings announcement, world news, etc., and react to this data according to logic particular to the firm. Trading rules might be as simple as, "sell all stock of company X if its price falls below Y," or they may be arbitrarily complex. Thus, these programs can be conceived of as maintaining internal state that changes in response to external events. In turn, a given state change may trigger one or more actions. Alternatively, an external entity may query this internal state and then take action contingent upon the results of the query.

The ability to audit such a system quickly becomes exceedingly important. Firms want to know which events triggered which actions and why. Perhaps more importantly, regulators may want to review such records for forensic purposes. Complicating either of these tasks is the fact that other firms may be employing separate but equivalent software, and each piece of software may well be reacting to the actions of another piece of independent software, thus creating a complex federated trading infrastructure. Thus, determining the causal history of one firm's actions may require synthesizing the trade history of an arbitrary number of other firms. This, in turn, implies that each firm must maintain data that is both sufficient in detail and compatible in format such that this forensic synthesis is possible.

Background: On May 6, 2010, the Dow Jones Industrial Average of the US Stock Market dove over 1000 points in the span of a few minutes, a loss of nearly 10% of its value. Just as quickly, it gained back nearly 600 points. Investors, and regulators, were quick to demand an explanation for this erratic behavior. Remarkably, no one was able to offer such an explanation, and to date no official cause has been determined. Absent a complete and reliable provenance-tracking system, forensic analysis of these events by regulators has been difficult if not impossible, and consequently no one is sure if such unstable behavior could happen again, and if so, under what conditions. This uncertainty constitutes potentially billions of dollars.

Users: Trading firms, regulators, trading firm clients

Requirement for provenance: Without a complete audit trail, reconstructing the causes of a given action or set of actions (such as stock purchases and sales) is impossible. Given the likelihood of competing implementations of trading software, each with its own provenance tracking mechanism, an interoperable provenance model becomes crucial in particular for regulators who must reconstruct the behavior of the entire trading ecosystem.

Provenance Questions:

• What prompted a given trade?
• Why did every firm engage in massive selling all at the same time?
• What precipitated a given market behavior?
• Did a firm change its behavior between time X and time Y?

Technologies used: Low-latency event stream processing engine

-- PeterMacko (on behalf of Nick Murphy) - 15 May 2010
to top