White Paper
Making More of Data Using AI
Sponsored by
Commodity Technology Advisory LLC Houston TX and Prague CZ
www.comtechadvisory.com
Data Management
A ComTech Advisory Whitepaper
Table of Contents Introduction ................................................................................................................................ 3 Increasing Complexity ................................................................................................................. 4 Social Media................................................................................................................................ 5 Artificial Intelligence ................................................................................................................... 6 Genic DATAiQ.............................................................................................................................. 6 About DataGenic ......................................................................................................................... 7 About Commodity Technology Advisory LLC ................................................................................ 8
Š Commodity Technology Advisory LLC, 2014 2
Data Management
A ComTech Advisory Whitepaper
“DataGenic is bringing to market a new product – Genic DATAiQ
– a cross-platform, cross-device web application for monitoring and analyzing front office trading data in real time. The product is designed to be intelligent offering users improved market awareness with AI and CEP enhanced data processing techniques such as semantic interpretation of news and reports that help to identify key events and patterns, enabling a quick response. “
Introduction Commodity traders now have access to a wide and increasing number of data sources and significantly larger volumes of data of all types. New regulations such as REMIT, for example, that are designed to increase market transparency and reduce possible market manipulation, ensure that many more new types and sources of important data are now generally available. A recent proprietary survey conducted by ComTech Advisory on behalf of DataGenic1 concluded that for around half of those surveyed, the volume of data they have to deal with has almost doubled over the last two-years. Commodity traders will need to cope with and manage, an ever-increasing amount of data in the future. Social media has also become a very popular medium over the last several years and arguably, it is increasingly a relatively untapped source of potentially useful intelligence for commodity and other traders. Social media outlets such as Twitter2 and the blogosphere are increasingly being utilized in other asset classes to provide traders with trading indicators and sentiment analysis. However, can social media really have any value for commodity traders and how are useful signals to be extracted from among all of the daily noise from such sources? 1
Data, Social Media and Artificial Intelligence, Proprietary survey for DataGenic, Commodity Technology Advisory, January 2014
2
See for example, ttps://nysetechnologies.nyx.com/en/sfti-social-media-tool
© Commodity Technology Advisory LLC, 2014 3
Data Management
A ComTech Advisory Whitepaper
Although the DataGenic survey found that the initial interest in the use of social media to support commodity-trading activities was very low (Figure 1), when respondents were challenged to think about social media as a source of trade intelligence, they became significantly more interested. In particular, the respondents struggled to understand how any valuable trend data could be extracted from the vast volume of noise. Artificial Intelligence is already used in some industries to help identify potentially useful signals from very noisy and voluminous data and surely this could be applied also to social media data. Figure 1: Relative Importance of Different Types of Data (5 is very important, 3 is average and 1 is unimportant) Social Media Algorithmic Data Technical Indicators News Historical Data Weather Data Fundamental Data Realtime Market Prices 0
1
2
3
4
5
At the same time as the quantity of data sources and volume of data grows, the issue of managing that data becomes more significant, increasing both direct and indirect costs accordingly at a time when trading margins are thinner. Trading firms are trapped in the quandary of needing to stay competitive in terms of access to trading intelligence while attempting to keep their costs down.
Increasing Complexity Commodity trading is increasingly complex due to the progressive introduction of new regulation, new trade instruments and exchanges, and generally changing market dynamics; such as the entry of Commercial and Industrial hedgers and exit of the US banks, for example. The DataGenic survey adequately demonstrates this growing complexity showing that almost half of its respondents routinely trade more than ten different markets and more than ten different products. Furthermore, around a third of the survey’s respondents utilized more than 25 different data sources (Figure 2). This increasing complexity can mean that traders need to stay abreast of much more information in order to form trading strategies and to make effective trading decisions. Furthermore, they need access to that information quicker and they need to be assured that the information is accurate.
Š Commodity Technology Advisory LLC, 2014 4
Data Management
A ComTech Advisory Whitepaper
Figures 2 and 3: Number of Data Sources Used and Increase Over the Last Two Years 0%
1 to 10
14% 32%
9%
11 to 25 25 to 50
9%
51 to 100 36%
100+
Less
4% 23%
18%
Same
55%
Less than doubled Doubled
The quality and accuracy of data varies significantly depending upon its source and much work often needs to be performed on the raw data to validate, check, potentially convert, derive calculated values and much more, before it may be deemed to be useful. Multiple versions of data sets also need to be stored and maintained across date ranges and to include any and all changes made to the data to ensure repeatability. The types of data that traders work with are also increasingly diverse. The survey showed what types of data traders used and the relative value of that type of data to the commodity trader (Figure 1). While tools and services exist to help users with data and data management, the survey also showed that internal IT still plays an important role in data management. The majority of respondents said that their internal IT normalizes or structures data for the traders. However, a common complaint across the respondents was the amount of time spent cleaning and preparing data as opposed to actually analyzing it. They also complained of the lack of good data management techniques and tools available to them generally. As the quantity of data increases, these issues will surely only get worse.
Social Media Social media includes tools like Facebook and Twitter generating vast amounts of data every minute. It also includes the blogosphere where experts and amateurs alike post their views. Only a small amount of this ‘data’ might be of use for traders however. An example of such use could include local weather and emergency conditions tweeted by residents such as perhaps, transportation disruption in a producing area that may have an impact on price formation due to reduced supply availability. Social media could equally well include broker’s blogs or the writings of trading commentators. The issues with social media though include its reliability (much of it is opinion based on little of no factual analysis) and its sheer volume. Despite that, there are a number of companies already offering social media derived data products and a number of generic news platforms are now incorporating it.
© Commodity Technology Advisory LLC, 2014 5
Data Management
A ComTech Advisory Whitepaper
A number of software products already exist to help traders make use of social media 3, including Reuters and Bloomberg. These tools and services are mainly targeted at equities traders and even include tools to automate trading based on sentiment analysis derived from social media and the blogosphere. University research in 2010 showed that social media mentions of a brand tended to correlate with its share price. Other types of social media can also be used in this way including the number of ‘likes’ on YouTube and Facebook, for example. However, can commodity traders utilize these same social media resources as well? Social media, like Twitter, along with conventional news sources and unconventional news sources like blogs, can be used to provide sentiment analysis and buy/sell signals that might prove useful especially when used in combination with traditional data. Part of the issue is the vast amount of ‘noise’ that needs to be filtered out as well as an ability to know what keywords to track and where to track them. Without appropriate tools, this task would be somewhat impossible or at least not very cost effective.
Artificial Intelligence One way to solve this problem is via Artificial Intelligence (AI) using Natural Language Processing (NLP) and Complex Event Processing (CEP). Both NLP and CEP have not been used extensively to date as a data management tool in the commodity trading arena and DataGenics’ survey showed that just under a third of those interviewed had never used AI tools but almost half could see it having value. However, as the volumes of data increases having computers detect patterns and categorize data according to rules Figure 4: The Value of AI set by the operators is one way to High Value make better use of the data and 6% highlight information that might ordinarily have been missed. If you 25% Moderate involved the vast volumes Value associated with social media of any 0% 38% Low Value kind, then the need to utilize Artificial Intelligence is essential. A.I. is already used on Wall St. to 31% None develop trading signals, trading strategies and even to perform automated trading so why not in commodity trading too?
Genic DATAiQ Seeing these trends, DataGenic is bringing to market a new product – Genic DATAiQ – a crossplatform, cross-device web application for monitoring and analyzing front office trading data in real time. The product is designed to be intelligent offering users improved market awareness with AI and CEP enhanced data processing techniques such as semantic interpretation of news and reports that help to identify key events and patterns, enabling a quick response. This may be a buy/sell signal. It has also been designed to be intuitive with a low learning curve, and responsive 3
The Power of Social media influencing trading and the markets, Investopedia, February, 2012 http://www.investopedia.com/financial-edge/0212/the-power-of-social-media-influencing-trading-and-the-markets.aspx
© Commodity Technology Advisory LLC, 2014 6
Data Management
A ComTech Advisory Whitepaper
interface; encompassing drag and drop and gestures and gamification to make using the tool fun and rewarding from the desktop or mobile device. Genic DATAiQ is also extensible so that users can add their own data, reports, widgets and indeed custom apps. More importantly perhaps, Genic DATAiQ is free to use with a valid email address.
About DataGenic Since its creation in 2002, DataGenic has been recognised as an international specialist in the delivery of data management solutions. DataGenic has developed an unrivalled knowledge of interval data, market pricing and reference data required by the industry, the complex usage of this data in analysis, trading, decision support, risk management, portfolio optimisation, operations and sales, and the technology needed to deliver and integrate information across the enterprise. DataGenic Solutions meet the complex data management requirements for companies that provide services in Finance, Utilities, Oil & Gas and the Public Sector.
Š Commodity Technology Advisory LLC, 2014 7
Data Management
A ComTech Advisory Whitepaper
About Commodity Technology Advisory LLC Commodity Technology Advisory is the leading analyst organization covering the ETRM and CTRM markets. We provide the invaluable insights into the issues and trends affecting the users and providers of the technologies that are crucial for success in the constantly evolving global commodities markets. Patrick Reames and Gary Vasey head our team, who’s combined 60-plus years in the energy and commodities markets, provides depth of understanding of the market and its issues that is unmatched and unrivaled by any analyst group. For more information, please visit http://www.comtechadvisory.com. ComTech Advisory also hosts the CTRMCenter, your online portal with news and views about commodity markets and technology as well as a comprehensive online directory of software and services providers. Please visit the CTRMCenter at http://www.ctrmcenter.com. ______________________________________________________________________________________ 19901 Southwest Freeway Sugar Land TX 77479 +1 281 207 5412 Prague, Czech Republic +420 775 718 112 ComTechAdvisory.com Email: info@comtechadvisory.com
Š Commodity Technology Advisory LLC, 2014 8