Big Data, The Solutions Define Its Future

I’ve often been asked to define the term Big Data. The reason the question even arises is due to the incredible explosion of hype. In the past 12-months alone the use of the term has increased over 1,200%. A Google search will net countless articles from pundits spouting their perspectives sprinkled with marketing-spin. Broadly speaking Big Data is associated with many aspects of commercial business, specific technologies and in some circles the science behind behavioral targeting.

However, in an attempt to uncover the true value behind Big Data it can be measured by two salient points. First there is the functional hardware supporting Big Data.  Bigger, faster, parallel hardware architectures have not only enabled more compute power but also massive growth in storage capacities. This classic Moore’s Law model has created maximal efficiencies in storage per dollar. It has also engendered a feedback loop, increased storage capacity begets demand for ever more capacity – as users store deeper and broader data over longer time periods.

Yet hardware and network infrastructure have long been subject to commoditization. Interoperability creates the practical necessity but this entropic trajectory makes hardware’s relevance in the Big Data equation equal to that of electricity.  The same applies to file systems, common relational storage architectures and cloud-based delivery vehicles – essential to the Big Data equation no doubt but simply commodity. Choices are equivalent to condiments, no matter how you dress it up you’re still eating a hamburger.

The great advancements in this foundational compute power have paved the way for Big Data’s true advantage and my second point, deriving business benefit through focused Big Data solutions. The ability to tell a story with the data is what elevates a Big Data solution over the storage and computational power. The story is germane to an industry such as finance and creates relevance and monetizes the data for a business.

Raj De Datta of BloomReach talks of Big Data application solutions – “… they harness the world’s data to
deliver you a better outcome – like more revenue“.  It’s the focused Big Data solutions that enable businesses to have access to better information, operate more efficiently, and be more profitable.  You will never hear a CEO crying for more technology to generate revenue.

Andy Palmer, co-founder of Vertica once wrote: “Big Data is useless unless you architect your systems to support the questions that end users are going to ask”.  Big Data solutions are about providing the right data in the right format to the right people at the right time.

Business is not aiming for do-it-yourself (DIY) big data solutions. Firms don’t want to be pioneers with vendors either, a significant reason why hardware and storage technologies are not and will not be the driving force behind Big Data. In Capital Markets the competitive pressures demand fit-for-purpose solutions for both buy-side and sell-side firms alike. Firms are looking for vendors that can deliver solutions and a data platform that can meet their time-to-market requirements. Front office solutions target price discovery and analyzing market trends, they are also geared toward controlling trade costs and managing risk. There is a very high price tag associated with the army of programmers needed to develop and support Big Data solutions, especially for tick data.

Market data comes in many shapes, sizes and encodings. It continually changes and requires corrections and an occasional tweak. Discovering new alpha and optimizing existing strategies demands a confidence in the resulting derived analytics. Financial Big Data solutions must manage the vagaries of data sources and time for trades and order books.  They must map ticker symbols across a global universe of exchanges and geographies and accurately reflect pricing thru cancellations, corrections, corporation actions and symbol changes. These are challenging financial-data management obstacles beyond the scope of ordinary storage architectures or file systems. Content-aware Big Data solutions leveraging the best of high-performance, scalable compute power are uniquely tuned to fulfill the demanding needs of quantitative analysts and algo traders. It is the Big Data solutions that ultimately define an end game, that Holy Grail for profitability. Firms should not lose sight of that.

Once again thanks for reading.
Louis Lovas

For an occasional opinion or commentary on technology in Capital Markets you can follow me on  twitter, here.