Text Analytics Past, Present & Text Analytics Past Present & Future: An Industry View Seth Grimes Alta Plana Corporation @ @sethgrimes g June 5 2014 June 5, 2014
Text Analytics: An Industry View
2
JADT – June 5, 2014
Text Analytics: An Industry View
3
Analytics is the systematic application of algorithmic methods that derive and deliver information, typically expressed quantitatively, whether in the form of tit ti l h th i th f f indicators, tables, visualizations, or models. • Systematic means formal & repeatable. • Algorithmic contrasts with heuristic.
JADT – June 5, 2014
Text Analytics: An Industry View
4
Text analytics past: Pioneers…
JADT – June 5, 2014
Document input and processing
Knowledge handling is g key
Hans Peter Luhn A Business Intelligence System “A Business Intelligence System” IBM Journal, October 1958
Desk Set (1957): Computer engineer Richard Sumner (Spencer Tracy) and television network librarian Bunny Watson (Katherine Hepburn) and the "electronic brain" EMERAC.
Text Analytics: An Industry View
6
“Statistical information derived from word frequency and distribution is p a relative measure off significance, g f , first f for f used byy the machine to compute individual words and then for sentences. Sentences scoring highest in significance are extracted and printed out to become the auto-abstract.” H.P. Luhn, h The Th Automatic A C Creation off Literature Abstracts, Ab IBM B Journal,l 1958. 195 JADT – June 5, 2014
Text Analytics: An Industry View
10
Pipelines and patterns IBM’s MedTAKMI, 997 1997‐
http://www.research.ibm.com/trl/projects/textmining/index_e.htm JADT – June 5, 2014
Text Analytics: An Industry View
11
Exhaustive extraction An (old) Attensity example – NLP to identify roles and p , pp relationships, for a law‐enforcement application .
JADT – June 5, 2014
Text Analytics: An Industry View
12
Language engineering GATE: General Architecture for Text Engineering.
http://gate.ac.uk/ JADT – June 5, 2014
Text Analytics: An Industry View
13
Text analytics present: Business, technology, applications, and , gy, pp , solutions…
JADT – June 5, 2014
Text Analytics: An Industry View
14
“Organizations embracing text analytics all report having an epiphany moment when they suddenly knew more than before.” ‐‐ Philip Russom, the Data Warehousing Institute, 2007 http://tdwi.org/articles/2007/05/09‐what‐works/bi‐search‐and‐text‐analytics.aspx
JADT – June 5, 2014
Text Analytics: An Industry View
15
Linguistics statistics and semantics Linguistics, statistics, and semantics Text analytics (typically) involves linguistic modelling, statistical characterization, learned patterns, and semantic understanding of text‐derived features – Named entities: people, companies, places, etc. P tt Pattern‐based features: e‐mail addresses, phone numbers, b d f t il dd h b etc. p Concepts: abstractions of entities. Facts and relationships. Events. Concrete and abstract attributes (e.g., “expensive” & “comfortable”) including measure‐value pairs. Subjectivity in the forms of opinions sentiments and Subjectivity in the forms of opinions, sentiments, and emotions: attitudinal data.
– applied to business ends. pp JADT – June 5, 2014
Text Analytics: An Industry View
16
Sources It’s a truism that 80% of enterprise‐relevant information g originates in “unstructured” form: E‐mail and messages. Web pages, online news & blogs, forum postings, and other social media. i l di Contact‐center notes and transcripts. Surveys feedback forms warranty claims Surveys, feedback forms, warranty claims. Scientific literature, books, legal documents. ...
Non‐text “unstructured” content? Images Audio including speech Video
Value derives from patterns. JADT – June 5, 2014
Text Analytics: An Industry View
17
Value What do we do with text, whether online, on‐social, or in p the enterprise? 1. Post/Publish, Manage, and Archive. 2. Index and Search. 3. Categorize and Classify according to metadata & contents. 4. Extract information and Analyze.
JADT – June 5, 2014
Text Analytics: An Industry View
18
Semantics analytics and IR Semantics, analytics, and IR Text analytics generates semantics to bridge search, BI, and pp , g g applications, enabling next‐generation information systems. Semantic search (search + text) Search based applications (search + text + apps) Text analytics (inner circle)
Information access (search + analytics)
Search
BI/Big BI/Bi Data
Applica‐ tions
Synthesis (text + BI)/(big data) NextGen CRM, EFM, MR, marketing, apps apps… JADT – June 5, 2014
Text Analytics: An Industry View
19
Content, composites, connections 1
JADT – June 5, 2014
Text Analytics: An Industry View
Content, Composites, Connections, 2
20
Content, composites, connections 2
JADT – June 5, 2014
Text Analytics: An Industry View
21
Applications Text analytics has applications in: Intelligence & law enforcement. Life sciences & clinical medicine. Media & publishing including social‐media analysis and p g g y contextual advertizing. Competitive intelligence. Voice of the Customer: CRM, product management & marketing. Public administration & policy. Public administration & policy Legal, tax & regulatory (LTR) including compliance. Recruiting. g
JADT – June 5, 2014
Text Analytics: An Industry View
22
Opinion, sentiment & emotion
JADT – June 5, 2014
Text Analytics: An Industry View
23
Sentiment analysis A specialization, of relevance to: Brand/reputation management. Customer experience management (CEM). Competitive intelligence. p g Survey analysis (EFM = Enterprise Feedback Management). Market research. Product design/quality. Trend spotting.
JADT – June 5, 2014
Text Analytics: An Industry View
24
Data exploration via dashboards and workbenches.
JADT – June 5, 2014
Text Analytics: An Industry View
25
Text analytics present: The market…
JADT – June 5, 2014
Text Analytics: An Industry View
26
http://altaplana.com/TA2014 JADT – June 5, 2014
Text Analytics: An Industry View
27
What are your primary applications where text comes into play? Voice of the Customer / Customer Experience Management
39%
Research (not listed)
38%
Brand/product/reputation management
38%
Competitive intelligence
33%
Search, information access, or Question Answering
29%
Customer /CRM
27%
Content management or publishing
25%
Online commerce including shopping, price intelligence,…
16%
Life sciences or clinical medicine
15%
E-discovery
14%
Insurance, risk management, or fraud
13%
Other
11%
Product/service design, quality assurance, or warranty claims
10%
Financial services/capital markets
9%
Intellectual property/patent analysis
8%
Law enforcement
6%
Military/national security/intelligence
5% 0%
5%
10%
15%
20%
25%
30%
35%
40%
JADT – June 5, 2014
45%
Text Analytics: An Industry View
28
Voice of the Customer Text analytics is applied to improve customer service and y y boost satisfaction and loyalty. Analyze customer interactions and opinions – • E‐mail, contact‐center notes, survey responses. • Forum & blog posting and other social media.
– to – • Address customer product & service issues. Address customer product & service issues • Improve quality. g p • Manage brand & reputation.
Assessment of qualitative information from text helps users – • • • •
Gain feedback on interactions. Assess customer value. Understand root causes. Mine data for measures such as churn likelihood Mine data for measures such as churn likelihood. JADT – June 5, 2014
Text Analytics: An Industry View
29
The commercial scene
JADT – June 5, 2014
Text Analytics: An Industry View
30
Online commerce Text analytics is applied for marketing, search optimization, p g competitive intelligence. Analyze social media and enterprise feedback to understand the Voice of the Market: • Opportunities • Threats • Trends
Categorize product and service offerings for on‐site search and faceted navigation and to enrich content delivery. Annotate pages to enhance Web‐search findability, ranking. Scrape competitor sites for offers and pricing. Analyze social and news media for competitive information.
JADT – June 5, 2014
Text Analytics: An Industry View
31
E Discovery and compliance E‐Discovery and compliance Text analytics is applied for compliance, fraud and risk, and y e‐discovery. Regulatory mandates and corporate practices dictate – • Monitoring corporate communications • Managing electronic stored information for production in event of litigation
Sources include e mail (!!), news, social media Sources include e‐mail (!!) news social media Risk avoidance and fraud detection are key to effective decision making • Text analytics mines critical data from unstructured sources • Integrated text‐transactional analytics provides rich insights
JADT – June 5, 2014
Text Analytics: An Industry View
32
What textual information are you analyzing or do you plan to analyze? 61%
blogs (long form+micro) 42%
news articles 38%
comments on blogs and articles bl d i l
37%
customer/market surveys
36%
on‐line forums on line forums 32%
Facebook postings scientific or technical literature
31%
online reviews
31%
2014
26%
e‐mail and correspondence
2011
22%
contact‐center notes or transcripts employee surveys
20%
chat
20%
2009
19%
social media not listed above
16%
Web‐site feedback 0%
20%
40%
60%
80%
JADT – June 5, 2014
Text Analytics: An Industry View
33
What textual information are you analyzing or do you plan to analyze? Twitter, Sina Weibo, or other microblogs blogs (long form) including Tumblr news articles comments on blogs and articles customer/market surveys on‐line forums Facebook postings scientific or technical literature online reviews e‐mail and correspondence contact‐center notes or transcripts employee surveys chat social media not listed above i l di t li t d b Web‐site feedback medical records text messages/instant messages/SMS other patent/IP filings speech or other audio field/intelligence reports crime, legal, or judicial reports or evidentiary materials h h h h l photographs or other graphical images warranty claims/documentation video or animated images point‐of‐service notes or transcripts insurance claims or underwriting notes
46% 43% 42% 38% 37% 36% 32% 31% 331% 26% 22% 20% 20% 19% 16% 13% 12% 12% 12% 11% 11% 9% 7% 5% 5% 5% 5%
0%
5%
10% 15% 20% 25% 30% 35% 40% 45% 50% JADT – June 5, 2014
Text Analytics: An Industry View
34
Do you currently need (or expect to need) to extract or analyze... Topics and themes
Current; 66%
Sentiment, opinions, attitudes, emotions, …
Expect; 22%
Current; 54%
Relationships and/or facts p /
Expect; 28% Expect; p 33%
Current; 47%
Named entities – people, companies, …
Current; 56%
Concepts that is abstract groups of entities Concepts, that is, abstract groups of entities
Expect; 25%
C Current; t 51%
Metadata such as document author, … Other entities – phone numbers, part/product … Semantic annotations
Current; 47% Current; 34%
Expect; 24%
Current; 33% 0%
10%
20%
Expect; 23% Expect; 23%
Current; 31%
Events
E Expect; t 28%
Expect; 21% 30%
40%
50%
60%
70%
80%
90% 100%
JADT – June 5, 2014
Text Analytics: An Industry View
35
“The share rise in users “Th h i i who selected Arabic coincided with Arabic…coincided with much of the civil unrest… in Middle Eastern countries.” http://bits.blogs.nytimes.com/2014/03/09/the ‐languages‐of‐twitter‐users/ g g f /
JADT – June 5, 2014
Text Analytics: An Industry View
36
Non-English language support? Other
9%
Other European or Slavic/Cyrillic
5%
Other Ot e East ast Asian sa
2%1%
0% 2%
C Current t
Other Arabic script (including Urdu,… 3% 1% Other African Turkish or Turkic
Within 2 years
2% 0% 3%
4%
Spanish
38%
Scandinavian or Baltic
7%
Russian Polish
3%
8%
Portuguese
21% 13%
3%
Korean
8%
7%
15%
Italian
18%
Hindi, Urdu, Bengali, Punjabi, or…2% Greek
17%
4%
4%
Japanese
11%
10%
2% 2%
German
34%
French
24%
36%
Dutch
9%
Chinese Bahasa Indonesia or Malay
20%
17%
7% 16%
28%
1% 3%
Arabic
10%
0%
17%
10%
20%
30%
40%
50%
60%
JADT – June 5, 2014
Text Analytics: An Industry View
37
Software & platform options Text‐analytics options may be grouped in general classes. • Installed text Installed text‐analysis application, whether desktop or analysis application, whether desktop or server or deployed in‐database. • Data mining workbench. • Hosted. • Programming tool. • As‐a‐service, via an application programming interface (API). • Code library or component of a business/vertical application, for instance for CRM, e‐discovery, search.
Text analytics is frequently embedded in search or other y q y end‐user applications. The slides that follow next will present leading options in each category except Hosted… JADT – June 5, 2014
Text Analytics: An Industry View
38
What is important in a solution? 64%
ability to generate categories or taxonomies 54%
ability to use specialized dictionaries, taxonomies, ontologies, or …
% 53%
broad information extraction capability
53%
document classification 45%
deep sentiment/emotion/opinion/intent extraction
44% %
low cost
43%
"real time" capabilities
41%
sentiment scoring
40%
support for multiple languages 37%
open source
2014 (n=139) 2011 (n=136)
36%
predictive‐analytics integration
2009 (n=78) 9( 7 )
33%
big data capabilities, e.g., via Hadoop/MapReduce
33%
ability to create custom workflows or to create or change …
32%
BI (business intelligence) integration
30%
sector adaptation (e.g., hospitality, insurance, retail, health care, …
28%
supports data fusion / unified analytics
25%
hosted or Web service (on‐demand "API") option
22%
media monitoring/analysis interface 0%
10%
20%
30%
40%
50%
60%
JADT – June 5, 2014
70%
Text Analytics: An Industry View
39
User decision criteria Primary considerations include – Adaptation or specialization: To a business or cultural domain, Adaptation or specialization: To a business or cultural domain language, information type (e.g., text, speech, images) & source (e.g., Twitter, e‐mail, online news). By‐user customization possibilities: For instance, via custom taxonomies, rules, lexicons. S ti Sentiment resolution: Aggregate, message, or feature level. t l ti Agg g t g f t l l (What features? Topics, coreferenced entities?) What sentiment? Valence & what else? Emotion? Intent? Outputs: E.g., annotated text, models, indicators, dashboards, exploratory data interfaces. Usage mode: As‐a‐service (API), installed, or hosted/cloud. Capacity: Volume, performance, throughput, latency. Cost. JADT – June 5, 2014
Text Analytics: An Industry View
40
A few French companies
JADT – June 5, 2014
Text Analytics: An Industry View
41
Academic spin offs Academic spin‐offs
People Pattern JADT – June 5, 2014
Text Analytics: An Industry View
42
Text analytics future: Synthesis and sensemaking. y g
JADT – June 5, 2014
New York Times, September 8, 1957
Text Analytics: An Industry View
44
Emotion in text
JADT – June 5, 2014
Text Analytics: An Industry View
45
Emotion and outcomes
JADT – June 5, 2014
Text Analytics: An Industry View
46
Beyond Text Audio including speech. Images Images. Video.
http://www.geekosystem.com/ f facebook‐face‐recognition/ f g
http://flylib.com/books/en/2.495.1.54/1/
http://www.sciencedirect.com/science /article/pii/S0167639312000118
JADT – June 5, 2014
Text Analytics: An Industry View
47
The world of big data Machine data (e.g., logs, sensor outputs, clickstreams). Actions interactions and transactions: geolocation and Actions, interactions, and transactions: geolocation time. Profiles: individual, demographic & behavioral. , g p Text, audio, images, and video. Facts and feelings.
JADT – June 5, 2014
Text Analytics: An Industry View
48
(Accessible) data everywhere
JADT – June 5, 2014
Text Analytics: An Industry View
49
A big data analytics architecture (example)
http://www.geeklawblog.com/2011/12/lexis‐advance‐platform‐launch‐two.html JADT – June 5, 2014
Text Analytics: An Industry View
50
Sensemaking “It is convenient to divide the entire information access process into two main components: information retrieval through searching and browsing, and analysis and synthesis of results. This broader process is often referred to in the literature as sensemaking. Sensemaking refers to an iterative process of formulating a conceptual p f f g representation from of a large volume of information.” – Marti Hearst, 2009
http://searchuserinterfaces.com/
JADT – June 5, 2014
Text Analytics: An Industry View
51
En route
http://www.businessweek.com/magazine/content/04_19/b3882029_mz072.htm JADT – June 5, 2014
Text Analytics Past, Present & Text Analytics Past Present & Future: An Industry View Seth Grimes Alta Plana Corporation @ @sethgrimes g June 5 2014 June 5, 2014