Building specialized applica/ons using Solr; Migra/on from FAST ESP
Rahul Agarwalla Head of Interna/onal Business Uchida Spectrum Inc.
©2011 Uchida Spectrum, Inc. All rights reserved.
Uchida Spectrum Overview
SoDware License Business 1995 ~ • So)ware License Sales • License Management Repor:ng • License Procurement System • License Adjustment Consul:ng
Network Technology Services 1997 ~ • Network System Consul:ng Services ― Ac:ve Directory Network ― Exchange Messaging Network
• License Management System Consul:ng ― So)ware Management Server
• Portal System Consul:ng
― Share Point Portal Server ― Websphere Portal Server
©2011 Uchida Spectrum, Inc. All rights reserved.
Enterprise Search Business 2002 ~ • Enterprise Intelligence Applica:on ― SMART InSight G2 Enterprise ― SMART InSight G2 Professional
• Search PlaRorm Consul:ng & Support ― FAST ESP ― Lucene/Solr ― Lucid Works Enterprise
Page-2
Page-2
Some of Uchida Spectrum’s customers
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-3
Page-3
SMART/InSight History
Customers in Japan, China & India: • • • •
2 of top 3 Japanese car manufacturers Top consumer electronics company Large financial ins8tu8ons China’s biggest eCommerce firm
2005: SMART InSight 1.1 2004: PlaRorm for custom solu:ons 2003: FAST Alliance
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-4
Page-4
What is today’s buzz word?
Smart Phone
• Extreme scalability • Flexibility & Extensibility • Feature rich search ©2011 Uchida Spectrum, Inc. All rights reserved.
Page-5
Page-5
What I learnt from the Japan catastrophe
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-6
Page-6
The power of community
Japanese Government [Closed/big brother] • • •
Slow, behind the curve Legacy/CYA Confusion
Japanese People [Open community] • • •
Quick response Disclose / Share Prac:cal Impact
Power shift
Driver of innova/on
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-7
Page-7
Lessons from FAST ESP Migra/on: advantage LWE/Solr
•
Key Issues: 1. 2. 3.
• • • • •
Smaller record and index size enable faster index maintenance # of records per node: rule of thumb 10m vs. 2m Licensing & Maintenance Cost: less than ½
Scalability: 5x Cost Performance: 10x High Flexibility Lower Opera/ons Cost Faster Innova/on
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-8
Enterprise Search expecta/ons • Big data scale • Security is important • Disparate data: geography, systems, languages, format, structures • KM is good to have, databases are cri:cal • Support different users & usage: department, role, tasks • High recall
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-9
Page-9
Lessons from FAST ESP Migra/on: Filling the gaps
•
Security • • •
•
ACL security: complex requirements File System: file & folder level control CRM/ERP… : Keeping ACLs up-‐to-‐date
Content aggrega/on • • •
Connectors Normaliza:on Open source op:ons for ESP pipeline • •
©2011 Uchida Spectrum, Inc. All rights reserved.
Openpipeline Pypes
Page-10
Building specialized applica/ons: Content fusion
•
Content fusion from disparate data: • •
Single index ≠ integra:on Modeling of content rela:onships is essen:al
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-11
Virtual integra/on based on search
Applica/on layer
Content sets and inter-‐rela/onships
Content store Big table, flat index
Search Index Search Index Search Index
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-12
LWE Adapter
Search Index LWE
©2011 Uchida Spectrum, Inc. All rights reserved.
SolrAdapter
Solr
Content Security . . . . . .
Boos:ng
Transform
……
Result Pipeline
Query Pipeline
• Data transforma:on: - key:key, key:value, field names • Query & Result transforma:on • Boos:ng / Relevancy algorithm • Security • Mul:-‐Language support • Federa:on & mashups
Tagging Pipeline
Search Service
Append Pipeline
Virtual integra/on based on search…2
Other
……
Page-13
Building specialized applica/ons: Personaliza/on
• •
Applica/on flow depends on the task Data Personaliza/on increases produc/vity
• SMART InSight approach: Task based UI •
• •
©2011 Uchida Spectrum, Inc. All rights reserved.
Schema independent widgets for analy:cs & visualiza:on Portalized Personalized: widgets, func:ons, content, fields
Page-14
Knowledge Center: made possible by Solr Scalability and low TCO gives us ability to build new features • •
Knowledge Centre has logs of all user ac:vity in SMART InSight This would be too costly with a commercial Search Engine and would not be feasible in a database
Using this rich data we can: • • • •
Profile users, groups and networks Personalize Recommenda:ons Create social ranking algorithms Usage analy:cs
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-15
Overview of SMART InSight for Automo/ve Task based UIs NHTSA Internet Page
Widgets
Ajax Portal
Personaliza/on
Benchmarking
EDR Dealers
Contents Set
SA
Design
PLM
SA
Engineering
Specs
SA
Claims
Knowledge Centre
Data Chain
SA
Claims
Convergent Knowledge Framework
Virtual Integra/on Framework
Repair
Recommend Profiling
Metadata Analysis
Parts Catalog
Knowledge Log
CAD Internal
Management & Security
Early Defect Warning
Content Model
Claim Analysis
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-16
: Interac/ve Click Log Analysis System
• > $50 Billion sales / year • > 800 Million Items • > 370 Million Users • Billions of clicks per day Access Log
Solr Hadoop
Solr, Hadoop + SMART/InSight G2
xxxxxxxx
Xxxx Xxxx Xxxx Xxxx xxxx
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-17
: Global Research Community • Top Academic Ins/tutes: • Faculty, Research Fellows & Post graduate students • Govt. Departments & Corporate R&D • Scien:sts and researchers
Research Discovery & Collec/ve Intelligence (Knowledge Centre)
Broadcast Search
Dynamic Result Merging Solr ©2011 Uchida Spectrum, Inc. All rights reserved.
• > 270 content sources: Socie/es, Associa/ons, Publishers & Open • IEEE, ACM… • Elsevier, Wiley, Springer…
Real /me indexing Page-18
Demonstra/on Â
Š2011 Uchida Spectrum, Inc. All rights reserved.
Page-19
Contact Details
Rahul Agarwalla Head – Interna/onal Business rahul@spectrum.co.jp www.spectrum.co.jp
©2011 Uchida Spectrum, Inc. All rights reserved.
Page-20