Big Data, The Solutions Define Its Future

I’ve often been asked to define the term Big Data. The reason the question even arises is due to the incredible explosion of hype. In the past 12-months alone the use of the term has increased over 1,200%. A Google search will net countless articles from pundits spouting their perspectives sprinkled with marketing-spin. Broadly speaking Big Data is associated with many aspects of commercial business, specific technologies and in some circles the science behind behavioral targeting.

However, in an attempt to uncover the true value behind Big Data it can be measured by two salient points. First there is the functional hardware supporting Big Data.  Bigger, faster, parallel hardware architectures have not only enabled more compute power but also massive growth in storage capacities. This classic Moore’s Law model has created maximal efficiencies in storage per dollar. It has also engendered a feedback loop, increased storage capacity begets demand for ever more capacity – as users store deeper and broader data over longer time periods.

Yet hardware and network infrastructure have long been subject to commoditization. Interoperability creates the practical necessity but this entropic trajectory makes hardware’s relevance in the Big Data equation equal to that of electricity.  The same applies to file systems, common relational storage architectures and cloud-based delivery vehicles – essential to the Big Data equation no doubt but simply commodity. Choices are equivalent to condiments, no matter how you dress it up you’re still eating a hamburger.

The great advancements in this foundational compute power have paved the way for Big Data’s true advantage and my second point, deriving business benefit through focused Big Data solutions. The ability to tell a story with the data is what elevates a Big Data solution over the storage and computational power. The story is germane to an industry such as finance and creates relevance and monetizes the data for a business.

Raj De Datta of BloomReach talks of Big Data application solutions – “… they harness the world’s data to
deliver you a better outcome – like more revenue“.  It’s the focused Big Data solutions that enable businesses to have access to better information, operate more efficiently, and be more profitable. 
You will never hear a CEO crying for more technology to generate revenue.

Andy Palmer, co-founder of Vertica once wrote: “Big Data is useless unless you architect your systems to support the questions that end users are going to ask”.  Big Data solutions are about providing the right data in the right format to the right people at the right time.

Business is not aiming for do-it-yourself (DIY) big data solutions. Firms don’t want to be pioneers with vendors either, a significant reason why hardware and storage technologies are not and will not be the driving force behind Big Data. In Capital Markets the competitive pressures demand fit-for-purpose solutions for both buy-side and sell-side firms alike. Firms are looking for vendors that can deliver solutions and a data platform that can meet their time-to-market requirements. Front office solutions target price discovery and analyzing market trends, they are also geared toward controlling trade costs and managing risk. There is a very high price tag associated with the army of programmers needed to develop and support Big Data solutions, especially for tick data.

Market data comes in many shapes, sizes and encodings. It continually changes and requires corrections and an occasional tweak. Discovering new alpha and optimizing existing strategies demands a confidence in the resulting derived analytics. Financial Big Data solutions must manage the vagaries of data sources and time for trades and order books.  They must map ticker symbols across a global universe of exchanges and geographies and accurately reflect pricing thru cancellations, corrections, corporation actions and symbol changes. These are challenging financial-data management obstacles beyond the scope of ordinary storage architectures or file systems. Content-aware Big Data solutions leveraging the best of high-performance, scalable compute power are uniquely tuned to fulfill the demanding needs of quantitative analysts and algo traders. It is the Big Data solutions that ultimately define an end game, that Holy Grail for profitability. Firms should not lose sight of that.

Once again thanks for reading.
Louis Lovas

For an occasional opinion or commentary on technology in Capital Markets you can follow me on  twitter, here.

About Louis Lovas

Director of Solutions, OneMarketData, with over 20 years of experience in developing cutting edge solutions and a leading voice in technology and trends in the Capital Markets industry.
This entry was posted in Algorithmic Trading, Analytics, Big Data, Complex Event Processing, Equities, Foreign Exchange, Futures and Options, OneMarketData, OneQuantData, OneTick, Tick database. Bookmark the permalink.

4 Responses to Big Data, The Solutions Define Its Future

  1. Uta says:

    Hi it’s me, I am also visiting this web site on a regular basis, this website is actually nice and the people are truly sharing fastidious thoughts.

  2. Everything is very open with a very clear explanation of the challenges.
    It was truly informative. Your site is extremely helpful.

    Thanks for sharing!

  3. Its like you learn my thoughts! You appear to grasp so much approximately this, such as you
    wrote the e book in it or something. I think that you
    can do with a few % to force the message house a bit, but instead of that, this is wonderful blog.
    A great read. I’ll definitely be back.

  4. endia.info says:

    Excellent way of explaining, and nice post to obtain
    facts about my presentation subject, which i am going to deliver
    in institution of higher education.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s