VideoSegmentationofLife-LoggingVideos
MarcBola˜nos1 ,MaiteGarolera2 ,andPetiaRadeva1,3
1 UniversityofBarcelona,Barcelona,Spain
2 HospitaldeTerrassa-ConsorciSanitarideTerrassa,Terrassa,Spain
3 ComputerVisionCenterofBarcelona,Bellaterra(Barcelona),Spain mark.bs.1991@gmail.com,MGarolera@cst.cat,petia.ivanova@ub.edu
Abstract. Life-loggingdevicesarecharacterizedbyeasilycollecting hugeamountofimages.Oneofthechallengesoflifeloggingishowto organizethebigamountofimagedataacquiredinsemanticallymeaningfulsegments.Inthispaper,weproposeanenergy-basedapproachfor motion-basedeventsegmentationoflife-loggingsequencesoflowtemporalresolution.Thesegmentationisreachedintegratingdifferentkindof imagefeaturesandclassifiersintoagraph-cutframeworktoassureconsistentsequencetreatment.Theresultsshowthattheproposedmethod ispromisingtocreatesummariesofeverydayperson’slife.
Keywords: Life-logging,videosegmentation.
1Introduction
Recently,withtheappearanceofdifferentLifeLogging(LL)devices(SenseCam [1],Looxcie[2],Narrative(previouslycalledMemoto)[3],Autobiographer),peoplewearingthemaregettingeagerforcapturingdetailsabouttheirdailylife. Capturingimagesalongthewholedayleadstoahugeamountofdatathat shouldbeorganizedandsummarizedinordertobeabletostorethemandreviewlater,beingabletofocusjustonthemostimportantaspects.Ontheother hand,LLdataappearverypromisingtodesignnewtherapiesfortreatingdifferentdiseases.LLdatahavebeenusedtoretainandrecovermemoryabilities forpatientswithAlzheimer’sdisease[1]aswellastocaptureanddisplaythe healthyhabitslikenutrition,physicalactivities,emotionsorsocialinteraction. In[4]theyareusedasanaidforrecordingtheeverydaylifeinordertobeable todetectandrecognizeelementsthatcanmeasurepersons’qualityoflifeand, thus,toimproveit[5,6].
LLdevicesbeingwornbyapersonthewholeday,havethepropertytocapture imagesforlongperiodsoftime.Dependingonwherethedeviceispositioned (head-mounted,onglasses,camerawithapin,hungcamera,ear-mounted,etc.) determinesthefieldofviewandthecameramotion(usually,glasscamerawould bemorestableandwouldgiveinformation,wherethepersonislookingat, meanwhilecamerahungontheperson’sneckmovesmoreandlacksinformation onwherethepersonislookingat).Ontheotherhand,ahungupcamerahas theadvantagethatisconsideredmoreunobtrusiveandthus,causeslessrepeal
F.J.PeralesandJ.Santos-Victor(Eds.):AMDO2014,LNCS8563,pp.1–9,2014. c SpringerInternationalPublishingSwitzerland2014
Fig.1. Illustrationofthethreepersonmovement-relatedeventstobedetected
fromthepersonsaroundrecordedbythecamera[7].Anotherimportantcharacteristicisthetemporalresolutionofthedevice.MeanwhileLooxciehashigh temporalresolutionand,thus,providessmoothcontinuousvideos,manyother LLdeviceslikeSenseCamhaslowtemporalresolution(2-4framesperminute) makingdifficulttoconsiderconsecutiveframesasvideos.Moreover,objectsin consecutiveimagescanappearinverydifferentpositions.Ontheotherhand, lowtemporalresolutioncamerashavetheadvantagestoacquireareasonable amountofimagesinordertocapturethewholedayofthepersonandallow toprocessimagescoveringlongperio dsoftime(weeks,months).Duetothis reason,inthisarticle,wefocusonsequencesegmentationwithaSenseCamthat isabletoacquireandstoreimagesduringthewholedayactivitiesoftheperson wearingthecamera.Moreover,beinghungontheneck,SenseCamislessobtrusivethathead-mounteddevices,buthaslowtemporalresolutionandsignificant freecameramotion.Usually,adaycapturedbyaSenseCamusedtocontain around4000imageswithnosmoothtransitionbetweenconsecutiveframes;ina monthmorethan100.000imagesaregenerated.
Developingtoolstoreduceredundancy,organizedataineventsandeaseLL reviewisofhighinterest.In[8],theauthorsproposedamethodforsegmentingandsummarizingLLvideosbasedonthedetectionof”important”objects [9].Dohertyet.al.proposeddifferentmethodslikeselectingspecifickeyframes [10],combiningimagedescriptors[11]andusingadissimilarityscorebasedon CombMIN[12]tosegmentandsummarizealsolow-resolutionLLdata.The workin[13]reviewsdifferenttechniquesforextractinginformationfromegocentricvideos,likeobjectrecognition,activitydetection,sportsactivitiesorvideo summarization.
EventsegmentationinLLdataischaracterizedbytheaction(movement)of thepersonwearingthedevice.Therelationbetweensceneandeventdepends
ontheperson’sactionthatisnotalwaysvisibleintheimages;thus,standard eventdetectiontechniquesinvideoarenotuseful.Wegroupconsecutiveframes inthreegeneraleventclassesaccordingtothehumanmovement(seeFigure 1):”Static”(personandcameraaremaintainingstatic),”InTransit”(person ismovingorrunning)and”MovingCamera”(personisnotchanginghis/her surroundings,butthecameraismoving-e.g.personisinteractingwithanother person,manipulatinganobject,etc.).Similareventclassificationhasbeenproposedandaddressedbyvideotechniquesin[8],wherehigh-temporalresolution LLdataareprocessed.Takingintoaccountthatinthecaseoflow-temporal resolutiondata,videoanalysistechniquesarenotusable,westudyanovelsetof imagefeaturesandintegratetheminanenergy-minimizationapproachforvideo segmentation1 .Incontrastto[8],weshowthatanoptimalapproachisachieved bycombiningasetoffeatures(bluriness[14],colour,SIFTflow[15],HoG[16]) andclassifiersintegratedinaGraphCut(GC)formulationforspatiallycoherenttreatmentofLLconsecutiveframes.
Thepaperisorganizedasfollows:insection2,weexplainthedesignand implementationofoureventextractionmethod.InSection3,wediscussthe resultsobtainedandfinishthepaperwithConclusions.
2Methodology
Toaddresstheeventsegmentationproblem,ourapproachisbasedontwomain steps:first,weextractmotion,colorandblurrinessinformationfromtheimages andapplyaclassifiertoobtainaroughapproximationoftheclasslabelsinsingle frames(Figure2).Second,weapplyanenergy-minimizationtechniquebasedon GCtoachievespatialcoherenceoflabelsassignedbytheclassifierandseparate thesequencesofconsecutiveimagesinevents.
2.1FeatureExtractionofLife-LoggingData
Giventhatthethreeclassesarebasicallydistinguishedbythemotionofthe cameraortheperson,aswellasthebigdifferencebetweenframes,robustevent segmentationneedsmotionfeaturesthatdonotassumesmoothimagetransition. Hence,weproposetoextractthefolowingfeaturetypes:
SIFTflowdata[15,17]: calculatedas8components,whichdescribethe motiononeachcardinaldirectionscaledbyitsmagnitude.
Blurriness[14]: calculatedas9componentsrepresentingtheblurrinesin eachcelldividingtheimagein3x3equalrectangles.
Colordifference: colorhistogramdifferencebetweenthecurrentimageand thefivepreviousones.WiththeuseoftheSIFTflowfeaturesbetweeneach pairofconsecutiveimages,weexpecttofinddifferencesbetweensequencesof imageswithlabel”Static”,whichshouldhavealowmagnitudeandlittleresilient directionofthedescriptorsinasignificantpartoftheimages.Labels”Moving 1 Althoughthelowtemporalresolution,westillspeakaboutvideosofdata,refering totheconsecutiveimagecollectionacquiredduringaday.
Fig.2. Diagramofthemainstepsfollowedbyourmethod
Camera”and”InTransit”shouldhaveamoreclearmovement.Atthesame time,thelasttwoclassesshouldbedifferentiatedhavingvectorsofflowwith undefinedandconstantlychangingdirection(inthe”MovingCamera”class) vs.thosepointingfromthecentertotheexternalpartoftheimageduetothe movement,whenwalkingforthe”InTransit”class.TheadvantageofSIFTflow isthatitisabletofindthecorrespondenceofpointsalmostindependentlyof theirdifferenceintheimageposition.Abouttheseconddescriptor,blurriness, wealsoexpectdifferentbehaviourfordistinguishingthe”Static”fromtheother labels,whichshouldhaveamoremarkedblureffect.Colordifferencesisexpected tobeinformativespeciallyforthe”MovingCamera”and”InTransit”classes.
2.2GC-BasedEventSegmentationofLLData
Eventsaresupposedtobesequencesofframeswiththesameclasslabel.Inorder toobtainsuchsequences,weapplyaGC-based[18,19]energy-minimizationtechniquetogetareliableeventsegmentation.GCsarebasedontheminimization oftheenergyresultingfromthesumoftwodifferentterms:
where fi arethesetoffeaturesusedfortheenergyminimization, Ui istheunary term, Pi,n isthepairwiseterm,whichrelatesanyimage i inthesequencewith eachofitsneighbours n ∈ Ni ,and W istheweightingtermforbalancingthe trade-offbetweentheunaryandthepairwiseterm.The unaryterm, Ui in ourcase,issetto1 LH ,being LH theresultfromaclassifieroutputthat representsthelikelihoodforeachimagetobelongtooneofthethreedefined classes.The pairwiseterm Pi,n isasimilaritymeasureforeachsampleon eachcliqu´e(allneighboursofagivensample)withrespecttothechosenimage featuresthatdeterminesthelikelihoodforeachneighbouringpairofimages (withaneighbourhoodlengthof11inourcase)tohavethesamelabel.TheGC
Fig.3. FractionofthetotalsummaryofeventsresultingfromadatasetusingKNNbasedGCsegmentation.Eachrowrepresentsoneofthe3classes,withthetotalnumber ofimagesandlabelbelongingtoeachofthemattheright.
algorithm[18,19]usingagraphstructurefindstheoptimalcutthatminimizes thesumofenergies E (f )assigningaclasslabeltoeachsampleasaresultofthe energyminimization.
Takingintoaccountthatthepairwisetermshould”catch”thefeaturesrelationbetweenconsecutiveframes,itusesdifferentfeaturesfromtheclassifier ones,namely:
- Color: RGBcolorhistogramswith9bins(3percolor).
- HoG[16]: Histogramoforientedgradientswith81componentsperimage tocapturechangesintheimagestructures.TheGCalgorithmassignsallthe consecutiveimageswiththesamelabeltothesameclass,andthusdetermines thefinaleventdivision.Figure3illustratesdifferentsamplesoftheextracted eventsfromthethreeclasses.Thelengthofeacheventisgivenontheright.For visualizationpurpose,eacheventisuniformlysubsampled.Notethatthe”T” eventsrepresentimagesequenceswith significantchangeofthescene(rows4 and7).”S”eventsarerepresentingastaticpersonalthoughtheimagescandiffer duetohandmanipulation(rows2,6and9),and”M”eventssuggestmoving person’sbody(rows1,3,5,8,and10).
3Results
Inthissection,wereviewthedatasetsusedinourexperimentsandthemost relevantperformedvalidationtests.
3.1DatasetsDescription
GiventhatthereisnopublicSenseCamdatasetwitheventlabels,forthevalidationweusedthedatasetfrom[6]thatcontains31749labeledimagesfrom10 differentdaystakenwithaSenseCam.Forthepurposeofthearticle,553events weremanuallyannotatedwith57.41imagesperevent,onaverage.
3.2ParameterOptimization
RegardingtheGCunaryterm,weperformeddifferenttestsusingtheoutputof twoofthemostpopularclassifiersinthebibliography:SupportVectorMachines (SVM)[20]andK-NearestNeighbour(KNN).Nevertheless,themethodallows touseanyclassifierthatprovidesascoreorlikelihoodtobeusedinthegraphcutscheme.Inpursuanceofobtainingthe mostgeneralizedresultpossible,when applyingtheRadialBasisFunctionSVMandtheKNN,wedesignedanested foldcross-validationforobtainingthebestregularizing(λ)anddeviation(σ ) parametersforthefirst,andthebest K valueforthesecondclassifier.Weused a10-foldcrossvalidationselectingrandomlythebalancedtrainingsamples.The optimalparametersobtainedwere: λ =3and σ =3, K =11onKNNwith Euclideandistancemetricand K =21onKNNwithcosinedistancemetric.
Withthesetests,ourpurposewastotesttheweightingGCparameterandto provetheimportanceofusingtheGCschemecomparedtotheframeclassificationobtainedbytheSVM/KNNclassifiers.Regardingtheweightvalue W ,we usedarangefrom0to3.75inintervalsof0.15points.WecanseeinFigure4 thedifferenceinaccuracybetween theKNNandtheGCfordifferent W values. Notethatfor W =1.75,theclassificationofframesimprovedfrom0.72to0.86. Itresultedthatinthiscase,weobtained108eventscomparedtothe56eventsin thegroundtruthoftestset10.Notethatinthiscasetheaccuracyis0.86representing15%ofimprovementregardingthebaselineclassificationresult,although theautomaticapproachtendstooversegmenttheevents.
3.3GCPerformanceforEventSegmentation
Asummaryoftheaverageimprovementofusingframeclassifier(theSVM/KNN) versusintegratingitintheGCschemecanbeseeninFigure4.Here,KNNe standsforKNNusingEuclideanmetricsandKNNcstandsforKNNwithCosinemetrics.Analysingtheresults,wecanobservethattheKNNobtainshigher accuracythantheSVM,andthataddingtheGC”labelsmoothing”afterit, theresultsarewidelyimproved.Theonlyaspecttotakeintoaccount,specially, whenusingtheKNNwithEuclideandistanceisthattheperformanceonallthe classesisfarfromthebalancedone(theaccuracyofclass”S”ismuchhigher thanthatoftheotherclasses).Inthiscase,aKNNwithcosinemetricsisagood compromiseofoverallaccuracyaswellas accuracyofeachclass,separately.RegardingtheresultusingSVM,GChasnotbeenabletoimprovetheresultsofthe SVM(onaverage).However,itstillhastwoadvantages:1)obtaininganaverage numberofeventsmoresuitableandrealisticwithrespect toeachdatasetand
Fig.4. ImprovementinaccuracyusingdifferentweightsfortheGCwithrespectto theKNNwithcosinemetrics;testsonthe10thdataset(left).Accuracyforeachclass (T,S,M)andaverageaccuracyfortheclassifiers(SVM,KNN)andtheGC(right).
2)havingmoresimilaraverageofaccuracyforeachclass(withoutanynegative peackofperformancelikeclass”M”incaseofSVM).
Inordertoseekredundantimagefeatures,weappliedaFeatureSelection(FS) basedontheStudent’st-test.WetestedthegainobtainedbytheFSmethodand thebestp-valueforitnotusingthelessrelevantfeaturesneitherfortheclassifier (SVM/KNN)norfortheGC.Comparingtheaccuracyresults,weobtainedno statisticaldifferenceinperformanceofthemethodwithandwithoutfeature selection.VerysimilarresultswereobtainedbytheSequentialForwardFloating Searchmethod[21].
OncewehaveappliedtheGCvideosegmentation,wehavethefinalsequence dividedintoeventsandclassifiedwiththerespectivelabels.Eventswithavery lownumberofimages,wouldcorrespondtotooshorteventsi.e.withlessthan 8images(lessthan2minutesinrealtime).Sincesuchsequenceswillnotbe enoughtoextractinformationinthefuture,neitherforobtainingasummary norfordetectingactionsofinterestoftheuser,theyaredeleted.
Thelimitationsofthemethodarerelatedtotheambiguitybetweenthe”T” and”M”labels,duetotheirmotionsimilarity,thatmakedifficulttoclassify. Moreover,the”free”motionofthecameraisdifficulttodifferentiate(foranyof theclassifiersused),andthis,addedtothefactthatweusetheHOGswithout anypreviousimageorientation(thatmightbeaproblemwhenthecamerais rotated),aresomeaspectsthatmightbeimprovedinfuturework.
4Conclusions
Inthiswork,weproposedanewmethodformotion-basedsegmentationofsequencesproducedbyLLdeviceswithlowtemporalresolution.Themostremarkableresultsarerepresentedbyintegratingawidesetofimagefeaturesand
aKNNclassifierwithcosinemetricsintotheGCenergy-minimization.Theproposedalgorithmachievedthemostbalancedaccuracyforthe3differentclasses. Ourmethodproposestoolstodetectmo tion-relatedeventsthatcanbeused forhigher-levelsemanticanalysisofLLdata.Themethodcouldeasetherecognitionofperson’sactionandtheelementsinvolved(objectsaround,manipulated objects,persons).Theeventscanbeusedasabasetocreate information”capsules”formemoryenhancementofAlzheimerpatients.Moreover,themethod canrelatethe”InTransit”labeltoexercisingactionoftheperson,ortheabundanceandlengthof”Static”eventsevidencingsedentaryhabits[22,23].Followingworksonhigh-temporalresolutionLLdata[9],importantpeopleandobjects canbedetectedandrelatedtothemostusefulandsummarizedstoriesfoundin theLLevents[24].OurnextstepsaredirectedtowardsLLsummarizationand detectionofinterestingevents,peopleandobjectsinlow-resolutiontemporalLL foreitherimprovingthememoryoftheuserorvisualizingsummarizedlifestyle datatoeasethemanagementoftheuser’shealthyhabits(sedentarylifestyles [22],nutritionalactivityofobesepeople,etc.).
Acknowledgments. ThisworkwaspartiallyfoundedbytheprojectsTIN201238187-C03-01,Fundaci´o”JaumeCasademont”-GironaandSGR1219.
References
1.Hodges,S.,Williams,L.,Berry,E.,Izadi,S.,Srinivasan,J.,Butler,A.,Smyth,G., Kapur,N.,Wood,K.:Sensecam:Aretrospectivememoryaid.In:Dourish,P.,Friday, A.(eds.)UbiComp2006.LNCS,vol.4206,pp.177–193.Springer,Heidelberg(2006)
2.Eisenberg,A.:WhenaCamcorderbecomesalifepartner,vol.6.NewYorkTimes (2010)
3.Bowers,D.:Lifelogging:Bothadvancingandhinderingpersonalinformationmanagement(2013)
4.Sellen,A.J.,Whittaker,S.:Beyondtotalcapture:aconstructivecritiqueoflifelogging.CommunicationsoftheACM53(5),70–77(2010)
5.Hoashi,H.,Joutou,T.,Yanai,K.:Imagerecognitionof85foodcategoriesby featurefusion.In:2010IEEEInternationalSymposiumonMultimedia(ISM), pp.296–301.IEEE(2010)
6.Bola˜nos,M.,Garolera,M.,Radeva,P.:Activelabelingapplicationappliedtofoodrelatedobjectrecognition.In:Proceedingsofthe5thInternationalWorkshopon MultimediaforCooking&EatingActivities,pp.45–50.ACM(2013)
7.Vondrick,C.,Hayden,D.S.,Landa,Y.,Jia,S.X.,Torralba,A.,Miller,R.C.,Teller, S.:Theaccuracy-obtrusivenesstradeoffforwearablevisionplatforms.In:Second IEEEWorkshoponEgocentricVision,CVPR(2012)
8.Lu,Z.,Grauman,K.:Story-drivensummarizationforegocentricvideo.In: 2013IEEEConferenceonComputerVisionandPatternRecognition(CVPR), pp.2714–2721.IEEE(2013)
9.Lee,Y.J.,Ghosh,J.,Grauman,K.:Discoveringimportantpeopleandobjectsfor egocentricvideosummarization.In:IEEEConferenceonComputerVisionand PatternRecognition(CVPR2012),pp.1346–1353.IEEE(2012)
10.Doherty,A.R.,Byrne,D.,Smeaton,A.F.,Jones,G.J.F.,Hughes,M.:Investigating keyframeselectionmethodsinthenoveldomainofpassivelycapturedvisuallifelogs.In:Proceedingsofthe2008InternationalConferenceonContent-BasedImage andVideoRetrieval,pp.259–268.ACM(2008)
11.Doherty,A.R., ´ OConaire,C.,Blighe,M.,Smeaton,A.F.,O’Connor,N.E.:Combiningimagedescriptorstoeffectivelyretrieveeventsfromvisuallifelogs.In:Proceedingsofthe1stACMInternationalConferenceonMultimediaInformationRetrieval, pp.10–17.ACM(2008)
12.Doherty,A.R.,Smeaton,A.F.:Automaticallysegmentinglifelogdataintoevents. In:ImageAnalysisforMultimediaInteractiveServices,WIAMIS2008,pp.20–23. IEEE(2008)
13.Bambach,S.:Asurveyonrecentadvancesofcomputervisionalgorithmsforegocentricvideo(2013)
14.Crete,F.,Dolmiere,T.,Ladret,P.,Nicolas,M.:Theblureffect:Perceptionand estimationwithanewno-referenceperceptualblurmetric.HumanVisionand ElectronicImagingXII6492,64920(2007)
15.Liu,C.,Yuen,J.,Torralba,A.,Sivic,J.,Freeman,W.T.:SIFTflow:Densecorrespondenceacrossdifferentscenes.In:Forsyth,D.,Torr,P.,Zisserman,A.(eds.) ECCV2008,PartIII.LNCS,vol.5304,pp.28–42.Springer,Heidelberg(2008)
16.Dalal,N.,Triggs,B.:Histogramsoforientedgradientsforhumandetection.In: IEEEComputerSocietyConferenceonComputerVisionandPatternRecognition (CVPR2005),vol.1,pp.886–893(2005)
17.Liu,C.:Beyondpixels:exploringnewrepresentationsandapplicationsformotion analysis,Ph.D.thesis,MassachusettsInstituteofTechnology(2009)
18.Boykov,Y.,Veksler,O.,Zabih,R.:Fastapproximateenergyminimization viagraphcuts.IEEETransactionsonPatternAnalysisandMachineIntelligence23(11),1222–1239(2001)
19.Delong,A.,Osokin,A.,Isack,H.N.,Boykov,Y.:Fastapproximateenergyminimizationwithlabelcosts.InternationalJournalofComputerVision96(1),1–27 (2012)
20.Cortes,C.,Vapnik,V.:Support-vectornetworks.MachineLearning20(3),273–297 (1995)
21.Pudil,P.,Novoviˇcov´a,J.,Kittler,J.:Floatingsearchmethodsinfeatureselection. PatternRecognitionLetters15(11),1119–1125(1994)
22.Kelly,P.,Doherty,A.,Berry,E.,Hodges,S.,Batterham,A.M.,Foster,C.:Canwe usedigitallife-logimagestoinvestigateactiveandsedentarytravelbehaviour?resultsfromapilotstudy.InternationalJournalonBehavioralNutritionandPhysical Activities8(44),44(2011)
23.Kerr,J.,Marshall,S.J.,Godbole,S.,Chen,J.,Legge,A.,Doherty,A.R.,Kelly,P., Oliver,M.,Badland,H.M.,Foster,C.:Usingthesensecamtoimproveclassificationsofsedentarybehaviorinfree-livingsettings.AmericanJournalofPreventive Medicine44(3),290–296(2013)
24.Shahaf,D.,Guestrin,C.:Connectingthedotsbetweennewsarticles.In:Proceedingsofthe16thACMSIGKDDInternationalConferenceonKnowledgeDiscovery andDataMining,pp.623–632.ACM(2010)
HumanPoseEstimationinStereoImages
JoeLallemand1,2 ,MagdalenaSzczot1 ,andSlobodanIlic2
1 BMWGroup,Munich,Germany {joe.lallemand,magdalena.szczot}@bmw.de http://www.bmw.de
2 ComputerAidedMedicalProcedures,TechnischeUniversit¨atM¨unchen,Germany slobodan.ilic@in.tum.de http://campar.in.tum.de
Abstract. Inthispaper,weaddresstheproblemof3Dhumanbody poseestimationfromdepthimagesacquiredbyastereocamera.ComparedtotheKinectsensor,stereocamerasworkoutdoorshavingamuch higheroperationalrange,butproducenoisierdata.Inordertodealwith suchdata,weproposeaframeworkfor3Dhumanposeestimationthat reliesonrandomforests.Thefirstcontributionisanovelgrid-based shapedescriptorrobusttonoisystereodatathatcanbeusedbyany classifier.Thesecondcontributionisatwostepclassificationprocedure, firstclassifyingthebodyorientation,thenproceedingwithdetermining thefull3Dposewithinthisorientationcluster.Tovalidateourmethod, weintroduceadatasetrecordedwithastereocamerasynchronizedwith anopticalmotioncapturesystemthatprovidesgroundtruthhuman bodyposes.
Keywords: HumanPoseEstimation,MachineLearning,DepthData.
1Introduction
Humanbodyposeestimationindepthimageshasseentremendousprogress inthelastfewyears.TheintroductionofKinectandothersimilardeviceshas resultedinanumberofnewalgorithmsaddressingtheproblemof3Dhuman bodyposeestimation[1,2,3,4].
AlthoughtheseKinect-likesensorsworkinreal-timeandusuallyprovide depthimagesofagoodqualitywithasmallamountofnoiseanddeptherrors asdepictedinFig.1,theyalsohavethedisadvantagesofonlyworkingindoors andataverylimiteddepthrange.Forthesereasons,humanposeestimation usingKinecthasextensivelybeenusedforgenericindoorscenarios.Manyother applicationshowever,especiallyautomotivedriverassistance,implytheuseof outdoor-suitablesensorsase.g.stereocameras.Sincestereo camerasystemsare becomingstandardinmoderncars,thereisaneedfor3Dhumanposeestimation fromstereodata.Forthatreason,weproposeanewalgorithmusingastereo camerawhichprovidesrealtimedepthimagesatarangeofupto50meters, whichisabout5timeshigherthanindoorsensors.AscanbeseeninFig.1,realtimestereoalgorithmsintegratedinvehiclesgenerallyproducenoisyimages,
F.J.PeralesandJ.Santos-Victor(Eds.):AMDO2014,LNCS8563,pp.10–19,2014. c SpringerInternationalPublishingSwitzerland2014
wheresomeregionsareerroneouslyfusedtogether(redcircles)andtheboundariesoftheobjectscanbeeffectedbyahighnumberofartifacts(greenandblue circles).Thesereconstructionartifactsintroducedbystereoreconstructionaffect theresultsofstate-of-the-artmethods,liketheoneofGrishicketal.[2],which wereimplementedandappliedtothesedata.Thisisbecauseofthetwomain reasons.Firstly,itisverydifficulttoperform360degreeposeestimationusing asingleforestasthereisahighconfusionbetweenfrontandback.Secondly thefeaturevectorproposedin[1]seemstoperformpoorlyonthestereodata. Thereforewepresentanewmethodforhumanposeestimation,adaptingrandomforestclassificationandregressionmethodologyintoatwosteppipelineto reliablyestimate3Dhumanbodypose.Thefirststepconsistsinclassifyingthe shapeofapersonintoaclusterwhichrepresentsitsorientationwithrespectto thecamera.Inthesecondstep,theskeletonposeofthepersonisestimatedusing aregressionrandomforesttrainedonlyontheposesofthedetectedorientation. Inordertomakethispipelineoperationalweintroduceanovelgrid-basedfeature.Thisfeatureovercomesseveraldisadvantagesthatappearwhenusingthe depthcomparisonfeatureintroducedbyShottonetal.[1]onthestereodataas shownintheresultsection.
Toverifyandvalidateourmethod,weintroduceadatasetwhichisrecorded withastereocamerasynchronizedwiththeARTmarkerbasedsystemforhuman motioncapture 1 .Theorientationclassificationisalsoevaluatedonthepublicly availablepedestrianstereodatasetintroducedin[5].
2RelatedWork
Manyalgorithmsforhumanposeestimationfromdepthimageshaveemergedin thelastyears.Shottonetal.[1]proposetouseaclassificationrandomforestto classifyeachpixelofaforegroundmasktoagivenbodypart,theninferthejoint locationsfromthepredictedbodyparts.Girshicketal.[2]extendedthisworkby learningaregressionmodeltodirectlypredictthejointlocations.Thisapproach considerablyimprovedtheperformanceof thepreviousalgorithmespeciallyfor occludedjoints.Bothworksrelyonalargesynthetictrainingdatasetinorder toachievegoodresultsandtargetgoodqualitydepthimages.
In[3],Tayloretal.trainaregressionrandomforesttocreateamappingfrom eachpixelofasegmentedforegrounddepthimagetoahumanbodymodel. Takingintoaccounttheforestpredictions,physicalconstraintsandvisibility constraints,theyuseanenergyminimizationfunctiontopredicttheposeofthe modelandtheattachedskeleton.Thisapproachimprovespredictionaccuracy comparedtopreviousworksandisabletopredictposesinthe360degreerange, butstillreliesonthetreestructurestrainedusingtheclassificationapproachof [1].
Sunetal.[6]introduceaconditionalregressionforestlearningagloballatent variableduringtheforesttrainingstepthatincorporatesdependencyrelationshipsoftheoutputvariables,e.g.torsoorientationorheight.
1 www.ar-tracking.com
Asimpledepthcomparisonfeatureiscommontoallthesemethods.Each dimensionofitconsistsofthedifferenceindepthcomputedattworandomoffsets fromthereferencepixelatwhichthefeatureiscomputed.Astheforeground masksinstereodatacontainmanyerroneousboundaries,thefeaturecannot beconsistentlyextractedforthesamepose.Theproposedgrid-basedfeatureis robusttotheseerrorsbecauseitconsistsofcellswheredepthandoccupancy distributionareaveragedoverthewholecell.
Pl¨ankersandFua[7]useanarticulatedsoftobjectmodeltodescribethehumanbodyandtrackitinasystemofcalibratedvideocameras,makinguseof stereoandsilhouettedata.UrtasunandFua[8]additionallyintroduceatemporalmotionmodelsbasedonPrincipalComponentAnalysis.Bernieretal.[9] proposea3Dbodytrackingalgorithmonstereodata.Thehumanbodyisrepresentedusingagraphicalmodelandtrackingisperformedusingnon-parametric beliefpropagationtogetaframebyframepose.Unlikethethreepreviously mentionedworks,whichrequireinitializationandtrackthehumanpose,our proposedmethodworksonsingleframesandperformsdiscriminativeposeestimation.Uptothebestofourknowledge,thisproblemhasnotyetbeenaddressed forthekindofnoisyinputdataasproducedbystereocamerasorsimilardevices.
Keskinetal.[10]useatwo-layerrandomforestforhandposeestimation. First,thehandisclassifiedbasedontheshape,thentheskeletonisdetermined forthegivenshapeclusterusingaregressionforest.Thoughsimilarto[10],we introduceanovelgrid-basedfeatureandatwostageclassificationmethodfor humanposeestimationinnoisystereodata.
In[11],EnzweilerandGavrilaproposeanalgorithmforsingle-framepedestriandetectionandorientationestimationbasedonamonocularcamera,where orientationisdividedinto4directions.Incontrasttothis,theproposedmethod isbasedonthedepthinformationfromthestereocameraandtheorientation clustersareencodingdirectionaswellasdifferentposeswithinthisdirection.
3Method
Thissectionintroducesthegrid-basedfeaturevectorwhichisusedboth,forthe classificationofhumanbodyorientationsandthehumanposeestimationper determinedorientationanddescribesthetwostepclassificationpipeline.The firststepinvolvesdeterminingthehumanbodyorientation.Whilethesecond computesthe3Dposeofaskeletonchoosingfromposesoftheestimatedorientationcluster.Finally,wedescribehowtheclassificationandposeprediction steparecombined.
3.1Grid-BasedFeature
Theproposedgrid-basedfeaturedividestheshapeofapersonintoarbitrary cells,thenaveragesoverdepthvaluesandoccupancydistributions.
Let Ω ⊂ R2 beasegmentedforegroundmaskinagivenimage.Theconstructionofthefeaturevectorconsistof4consecutivesteps.Thefirststepdetermines
theboundingboxaroundtheforegroundmask.Inthesecondstep,thebounding boxisdividedintoan n × m gridofcells ci,j .Notethatthisdivisionisscaleinvariant,astheboundingbox,regardlessofitsactualsize,isdividedintothesame numberofcells.Inthethirdstep,weattributeeachpixeloftheforegroundtoits correspondingcellanddeterminethemedianposition, xci,j ∈ R and yci,j ∈ R andmediandepth zci,j ∈ R ineachcell.Thiscellstructurenowrepresentsa verysimpleencodingoftheshapeofaperson.Ifacellisleftunoccupied,itis assignedaveryhighvalue.Finally,thepixel-wisegrid-basedfeatureisgivenby:
forapixel pk = {xk ,y
,z
}.Figure1showsthedifferentstepsofgenerating thefeaturevector.Inthisway,thefeaturevectorisabletoignoresmallerrors ofthestereoalgorithmespeciallyaroundbordersandsystematicerrorsofthe algorithmaretakenintoconsiderationasshowninFig.1(b).Theresultsection providesanalysisoftheinfluenceofthefeaturedimensionontheperformance oftheclassifier.
Fig.1. (a,b):ComparisonbetweenthedataqualityacquiredwithKinect(a)andwith thestereocamera(b).(c)Differentstagesofcreatingthefeaturevectorherefor5 × 7 cellsfromlefttoright:theboundingboxtightlylaidaroundtheforegroundmask,the subdivisionoftheboundingboxintoagridofcells,thecomputedmedianineachcell inred,thefeaturevectorforarandomlychosenpixelingreenandtheconnectionto eachcellmedianinyellow.
3.2GeneralTheoryonTrainingRandomForests
Arandomforestisanensembleofdecorrelatedbinarydecisiontrees.Itistrained onadataset Δ,consistingofpairs ψi = {fi ,li } offeaturevectors f andthelabels l ,learningthemappingfromthefeaturestothelabels.Eachtreeistrainedona subsetofthetrainingdataensuringthatthetreesarerandomized.Ateachnode ofthetree,adecisionfunction gν,τ (f ) ≡ ν ∗ f<τ istrainedsendingsamplesto theleftchildnodeifthisconditionisverifiedelsetotherightchildnode,where ν choosesexactlyonefeaturedimensionthuscreatingaxisalignedsplits.Inorder
(a) (b)
(c)
totrainthisdecisionfunction,ateachnode,asubsetofallfeaturedimensions israndomlychosenandforeachfeature, n thresholdsaregenerated,separating theincomingsamples Δ intoleftandrightsubsets Δl and Δr .Foreachofthese splits,aninformationgainiscomputed:
where H isanentropyfunctiondependingonthekindofrandomforestand |·| denotesthenumberofelementsinaset.Thefinaldecisionfunction gν ∗ ,τ ∗ isgivenbyfinding argmaxν,τ (Iν,τ ).Thisprocessisrepeatediterativelyuntila leafnodeisreached,whichisdefinedbythefollowingcriteria:(i)themaximum depthisreached,(ii)aminimumnumberofsamplesisundercutor(iii)the informationgainfallsbelowacertainthreshold.Intheleafnodes,allincoming samplesareusedtocomputeaposteriordistributionwhichdependsdirectlyon thekindofforesttrained.
3.3OrientationClassification
Thegoaloftheorientationclassificationistoassignthecurrentforegroundmask toitscorrespondingclustercontainingalltheposesofaspecificorientationin relationtothecamera.Toachievethis,clustersarecreatedusingthemotion capturedataacquiredforeachposeandaclassificationrandomforestistrained toclassifyeachpixelintothecorrectcluster.
Fig.2. (a)30Orientationclustersobtainedwithk-meansclustering.Forsuchalarge numberofclusters,theposesaredividedbyorientationbutalsobroadlyintoarmand legmovements.(b)Orientationclassificationresultsfordifferentsizesofthegridlike featureanddifferentnumberoforientationcluster.
GenerationofOrientationClusters. Theclustersaregeneratedinanunsupervisedmanner,usingthemotioncapturedatafromthetrainingdataset.Foreach pose,theanglesbetweenallneighboringjointsarecomputed.Clusteringisdone usingthek-meansapproachonthesejointangles.Incasek-meansisrunonthe euclideandistancesofjointpositionsin3Dspace,thealgorithmnotonlyseparatesposesintermsofjointanglesbutalsopeopleofdifferentheights.Byusing onlythejointanglesanddeliberatelyomittinglimblengths,wegetconsistent
(a)
(b)
clustersfordifferentposeswithregardtotheoverallorientationoftheperson. K-meansreliesoninitialseedstocreateclustersandresultscanvarydepending onthoseseeds.Inordertoachieveacertainlevelofindependencefromthis,we run100instancesofk-meansandchooseaclustercombinationwhichismost oftenreachedduringthisprocess.TheinfluenceofthenumberofclustersisanalyzedintheSec4.Althoughotherclusteringalgorithms,e.g.meanshift[12] weretested,theydidn’tgivesatisfactoryresults.Sincefixingthebandwidthof meanshiftbyhandisnottrivial.K-meanswasthefinalchoiceforclustering.
ClassificationofOrientationClusters. Theclassificationrandomforestistrained usingthegrid-basedfeaturetoclassifyeachpixeltothecorrectcluster.Shannon’sentropyisusedfortheinformationgain.Additionally,weusetherandomfieldbasedreweightingfunctiondescribedin[13].Thisreweightingscheme takesintoaccounttheclassdistributionofthefulltrainingdataset,insteadof reweightingonlythesamplesinthecurrentnode,whichwasshowntoyieldmore accurateresults.Theinformationgain I isrewrittenas:
where Δ0 isthetotaltrainingset, n (c,Δi )isthenumberofoccurrencesof classcinthesubset Δi ,and wc =
C n(k,Δ0 ) n(c,Δ0 ) istheweightobtainedby dividingthetotalnumberofsamples k inthedataset Δ0 bythenumberof occurrencesofclassc.Itislowestforthemostrepresentedclassandviceversa. Z (Δi )= k∈C wk n (k,Δi )isanalogoustothepartitionfunctioninarandom fieldandrepresentstheweightofagivensubset Δi .Itreplacestheweight |Δl | |Δ| inEquation2.Thedetailedderivationofthisformulafromthestandard Shannon’sentropyispresentedintheworksofKontschiederetal.[13]wherethis newinformationgainwasfirstintroduced.Theleafnodesstorethedistribution ofclassesofallincomingpointsasahistogram.
3.4PoseEstimationPerOrientationCluster
Oneregressionforestistrainedfortheposeestimationofeachcluster.For eachtree,thetrainingsetconsistsofpixelsobtainedfromabootstrapofthe trainingimagesbelongingtoagivencluster.Thegroundtruthjointpositions areprovidedbyamotioncapturesystem,aswillbeexplainedinSec.4.1.The trainingdatasetconsistsofpairsofpixel-wisefeaturesasdescribedinSec.3.1and labelscontainingtheoffsetfromthegivenpixeltoalljointpositions.Foragiven pixel pi (xi ,yi ,zi )andthepose J = {j1 ,...,jN } consistingof N joints,thelabel isgivenby Ψ = {ψ1 ,...,ψN },witheach ψk =(jk,x xi ,jk,y yi ,jk,z zi ). Duringtrainingweiterativelysearchfortheoptimalsplitineachnode.Asshown in[2],thebodyjointscanbemodeledusingamultivariategaussiandistribution. Followingthisidea,wecanmodeltheinformationgainbasedonthedifferential entropyofgaussiansandassumeindependencebetweenthedifferentjoints.The
entropyfunctionHintheinformationgainfunctioncanthusbereducedto:
where Σ isthediagonalofthecovariancematrixofthejointpositionsand N isthenumberofjoints.Oncealeafnodecriterionisfulfilled,themeanshiftis computedonallincomingpointsforeachjointandthemainmodeisstoredwith itsweight,equaltothenumberofpointsvotingforthemainmode.
3.5PredictionPipeline
Foreachimagewithanunknownpose,thegrid-basedfeatureiscomputedfora randomsubsetofpixelsfromtheforegroundmask.Theyarethensentthrough alltreesoftheorientationclassificationforest.Thehistograms,containingthe distributionoverorientationclusters,extractedfromallleafsareaveragedover allpixelsandtrees.Weretainthethreebestorientationsfortheposeestimation. Intheposeestimationstep,allpixelsaresentthroughtheforestsbelonging tothosethreebestorientationclusters.Thefinalposeaggregationisdoneby applyingmeanshifttothepredictionsforeachjointseparatelyandchoosingthe mainmodeasthepredictionoutcome.
4ExperimentsandResults
4.1DataAcquisition
Inordertobeabletotestouralgorithm,wehavecreatedanewdataset,using astereocameraandamotioncapturesystem.Sincethemocapsystemdoesnot workoutdoors,thetrainingdatawasacquiredindoors.Thetrainingsetconsists ofsequencesdepicting10peopleperformingvariouswalkingandarmmovement motions.Duringtheacquisitiontheactorswerewearing14markerswhichreflect theinfraredlightemittedby8infraredcamerasandareusedtoprovideground truthskeletonpositionsforeachframe.Thedatasetconsistsof25000frames.
4.2OrientationClassification
ProposedDataset: Inthisparagraph,weanalyzetheorientationclassification part,describedinSection3.3.Theevaluationistwofold,firstweanalyzehow thenumberofclustersaffectstheclassificationoutcome,thenweevaluatethe influenceofthenumberofcellsofthefeaturevectorandcomparetothedepth comparisonfeature.Thenumberofclustersweresetto10and30duringthe experiments.Forthefeaturevector,weperformanevaluationprogressivelyincreasingthenumberofcellsfrom3 × 3to11 × 11instepsof2.Themaximum allowedtreedepthissetto20,andeachforestconsistsof5trees.Allresultsare averagedoveracrossvalidation.Foreachvalidation,theforestsweretrainedon 8peopleandtestedontheremaining2.ResultscanbeseeninFig.2(b).The
bestresultsareachievedfor30clusters.Therearetwoimportantobservations regardingthefeaturevector.Firstly,dividingthefeatureintotoomanycells, especiallyalongthey-axis,decreasesthe overallperformanceofthefeature.Especiallyforsideviewsandposeswherealllimbsareclosetothebody,afine gridalongthey-axisnegativelyeffectsthenoisereductionpropertiesforwhich thefeaturewasdesigned.Secondly,thefeaturevectorseemstoperformbestif theratiobetweenthenumberofrowsandcolumnsisclosertotheheightversus widthratioofthehumanbody.
Inordertocomparethegrid-basedfeaturetothefeatureusedin[1,2,3], wetrainedarandomforestsampling500featurecandidatesand20thresholds ateachnodewithamaximumprobeoffsetof1.29pixel-meters,identicalto thoseproposedin[1].Allotherparameterswerekeptidenticaltotheother experiments.Thegrid-basedfeatureachieved81.4%and89.9%for10and30 clustersrespectivelycomparedto64.6%and72.3%forthedepthcomparison featureusedin[1].
Fig.3. Evaluationofthegrid-basedfeaturevectorwithregardtothenumberofclusters andthenumberofcellsinthegrid.(a):Theaccuracyperjoint(b):errorincmper joint.
DaimlerPedestrianSegmentationBenchmark: Inordertoshowthattheapproachalsoworksoutdoors,weevaluatetheorientationclassificationonthe publiclyavailabledatasetofFlohrandGavrila[5],consistingof785singledisparityimagesofpedestriansatvariousdistancesfromthecamera.Thisdataset containsannotatedgroundtruthfortheforegroundmasksofthepedestriansbut doesnotcontainorientationinformation.Toevaluateourapproach,weseparatetheorientationclustersofourapproachinto8directionswithregardtothe camera {front,front-left,left,back-left,back,back-right,rightandfront-right}, choosingforeachofthegeneratedclustersthedominanttorsoorientation.Since thegroundtruthposeisnotavailableforthisdatasettodeterminethecorrect cluster,wechoosevisuallytheclosestorientationusedforthemanuallylabeled clusters.Testswererunforthe30-clustertrainingsetupusingthebestfeature fromthepreviousexperiments,achieving78%accuracy.
(a)
(b)
Itisnoteworthythatmostofthedisparityimagesprovidedbythedatasetare muchsmallerinsizethanthetrainingimages.Inonlyabouthalfoftheprovided images,theheightoftheforegroundmaskishigherthan120pixels,whichis roughlyhalfoftheaverageheightofthetrainingimages.Thisshowsthatour algorithmandespeciallythefeatureworkwellevenifthesizeoftestingimages isafractionofthesizeofthetrainingimagesInFig.4,weshowsomeexample imagesfromthedatasetwiththedeterminedorientation.
Fig.4. Exampleimagesfromthedatasetof[5].Thegroundtruthlabelisdenoted ingreenandthepredictioninred.Theyellownumberdisplaysthepercentageof foregroundpixelsvotingforthepredictedcluster.Weshowtheoriginalimageinstead ofthedepthimage,asitisvisuallymorehelpful.
4.3PoseEstimation
Theevaluationoftheposeestimationisdoneforclustersizesof10and30.For eachscenario,weusethebestfeaturefromthepreviousevaluationandapply thecompletepredictionpipelineasdescribedinSection3.5.Firsttheclassificationforestdeterminesthecorrectorientationcluster,thentheregressionforests fromthethreemostprobableclustersareusedtopredictthepose.Weconsider ajointtobecorrectlyestimatedifitiswithinaradiusof10centimetersofthe groundtruthjointposition.Thisfollowstheevaluationcriteriaestablishedby severalrelatedworks[1,2].ResultsareshowninFig.3(a).Fig.3(b)showsthe medianerrorperjoint.Weexplicitlyusethemedian,asanerrorintheorientationclassificationispropagatedtotheposeestimationproducingwrongposes withperjointerrorsofupto1m.Bydisplayingthemedianerror,wecanshow thatifthecorrectorientationhasbeendetermined,theposepredictionproduces goodresultsforalldifferentorientations.Examplesareshowninsupplementary materialsvideo.Tocompareourgrid-basedfeaturetothedepthcomparisonfeatureof[1],wetrainregressionforestsforeachclusterusingthesameparameters ashavebeendescribedfortheorientationclassification.Forafaircomparison betweenbothfeaturesintermsofposeregression,weusetheoutputoftheclassificationforesttrainedwiththegrid-basedfeature.Thisway,wedonotpenalize errorsofthedepthcomparisonfeatureintheorientationclassificationstep.The grid-basedfeatureachieved75 8%and84 9%for10and30clusters,compared to71 3%and80 0%forthedepthcomparisonfeature.
Thepredictionpipelineincludingfeaturecomputation,orientationclassificationandtheposepredictionruninreal-timeat35fpsonanIntel(R)Core(TM) i5-2540CPU.
5Conclusion
Weproposeanewalgorithmforhumanposeestimationinstereoimagesconsistingoftwostagesprocedure,wherewefirstclassifyglobalorientationandthen predictthepose.Weintroducedanewgrid-basedfeaturevectorandprovedits effectivenesscomparedtothecommonlyuseddepthcomparisonfeatureof[1]. Thisfeatureisalsousedinourtwo-stageprocedurewherefirstaclassification forestwasusedfororientationpredictionandthenaregressionforestisusedfor poseestimation.Inthefuture,wewanttoincludethecolorinformationprovided bythestereocameraandconsidertemporalinformationtocopewithisolated wrongpredictions.
References
1.Shotton,J.,Fitzgibbon,A.,Cook,M.,Finocchio,T.S.M.,Moore,R.,Kipman,A., Blake,A.:Real-timehumanposerecognitioninpartsfromsingledepthimages. In:CVPR(2011)
2.Girshick,R.,Shotton,J.,Kohli,P.,Criminisi,A.,Fitzgibbon,A.:Efficientregressionofgeneral-activityhumanposesfromdepthimages.In:ICCV(2011)
3.Taylor,J.,Shotton,J.,Sharp,T.,Fitzgibbon,A.:Thevitruvianmanifold:Inferringdensecorrespondencesforone-shothumanposeestimation.In:2012IEEE ConferenceonComputerVisionandPatternRecognition(CVPR),pp.103–110. IEEE(2012)
4.Pons-Moll,G.,Taylor,J.,Shotton,J.,Hertzmann,A.,Fitzgibbon,A.:Metricregressionforestsforhumanposeestimation.In:BMVC2013(2013)
5.Flohr,F.,Gavrila,D.M.:Pedcut:Aniterativeframeworkforpedestriansegmentationcombiningshapemodelsandmultipledatacues.In:BMVC2013(2013)
6.Sun,M.,Kohli,P.,Shotton,J.:Conditionalregressionforestsforhumanposeestimation.In:IEEEComputerVisionandPatternRecognition(CVPR),pp.3394–3401 (2012)
7.Pl¨ankers,R.,Fua,P.:Articulatedsoftobjectsformulti-viewshapeandmotion capture.IEEETrans.PatternAnal.Mach.Intell.25(10)(2003)
8.Urtasun,R.,Fua,P.:3dhumanbodytrackingusingdeterministictemporalmotion models.In:Pajdla,T.,Matas,J(G.)(eds.)ECCV2004.LNCS,vol.3023,pp.92–106. Springer,Heidelberg(2004)
9.Bernier,O.,Cheung-Mon-Chan,P.,Bouguet,A.:Fastnonparametricbeliefpropagationforreal-timestereoarticulatedbodytracking.ComputerVisionandImage Understanding113(1),29–47(2009)
10.Keskin,C.,Kıra¸c,F.,Kara,Y.E.,Akarun,L.:Handposeestimationandhandshape classificationusingmulti-layeredrandomizeddecisionforests.In:Fitzgibbon,A., Lazebnik,S.,Perona,P.,Sato,Y.,Schmid,C.(eds.)ECCV2012,PartVI.LNCS, vol.7577,pp.852–863.Springer,Heidelberg(2012)
11.Enzweiler,M.,Gavrila,D.M.:Integratedpedestrianclassificationandorientation estimation.In:IEEEComputerVisionandPatternRecognition,CVPR(2010)
12.Comaniciu,D.,Meer,P.:Meanshift:Arobustapproachtowardfeaturespace analysis.IEEETrans.PatternAnal.Mach.Intell.24(5)(2002)
13.Kontschieder,P.,Kohli,P.,Shotton,J.,Criminisi,A.:Geof:Geodesicforestsfor learningcoupledpredictors.In:CVPR2013(2013)
Another random document with no related content on Scribd:
—— (Dr. G.), 1020, 1032
Macfarren (Sir G. A.), 1030
Mackail (J. W.), 1018
Mackinnon (J.), 1006
Macleod (H. D.), 1016
Macpherson (Rev. H. A.), 1012
Madden (D. H.), 1013
Maher (Rev. M.), 1016
Malleson (Col. G. B.), 1005
Marbot (Baron de), 1007
Marquand (A.), 1030
Marshman (J. C.), 1007
Martineau (Dr. James), 1032
Maskelyne (J. N.), 1013
Maunder (S.), 1025
Max Müller (F.), 1007, 1008, 1015, 1016, 1022, 1030, 1032
—— (Mrs.), 1009
May (Sir T. Erskine), 1006
Meade (L. T.), 1026
Melville (G. J. Whyte), 1022
Merivale (Dean), 1006
Merriman (H. S.), 1022
Mill (James), 1015
—— (John Stuart), 1015, 1017
Milner (G.), 1031
Miss Molly (Author of), 1026
Moffat (D.), 1013
Molesworth (Mrs.), 1026
Monck (W. H. S.), 1015
Montague (F. C.), 1006
Montagu (Hon. John Scott), 1012
Moore (T.), 1025
—— (Rev. Edward), 1014
Morgan (C. Lloyd), 1017
Morris (W.), 1020, 1022, 1031
—— (Mowbray), 1011
Mulhall (M. G.), 1017
Nansen (F.), 1009
Nesbit (E.), 1020
Nettleship (R. L.), 1014
Newdigate - Newdegate (Lady), 1008
Newman (Cardinal), 1022
Ogle (W.), 1018
Oliphant (Mrs.), 1022
Oliver (W. D.), 1009
Onslow (Earl of), 1011
Orchard (T. N.), 1031
Osbourne (L.), 1023
Park (W.), 1013
Parr (Louisa), 1026
Payne-Gallwey (Sir R.), 1011, 1013
Peek (Hedley), 1011
Pembroke (Earl of), 1011
Phillipps-Wolley (C.), 1010, 1022
Pitman (C. M.), 1011
Pleydell-Bouverie (E. O.), 1011
Pole (W.), 1013
Pollock (W. H.), 1011
Poole (W. H. and Mrs.), 1029
Poore (G. V.), 1031
Potter (J.), 1016
Praeger (S. Rosamond), 1026
Prevost (C.), 1011
Pritchett (R. T.), 1011
Proctor (R. A.), 1013, 1024, 1028
Quill (A. W.), 1018
Raine (Rev. James), 1004
Ransome (Cyril), 1003, 1006
Rauschenbusch-Clough (Emma), 1008
Rawlinson (Rev. Canon), 1008
Rhoades (J.), 1018
Rhoscomyl (O.), 1023
Ribblesdale (Lord), 1013
Rich (A.), 1018
Richardson (C.), 1012
Richman (I. B.), 1006
Richmond (Ennis), 1031
Richter (J. Paul), 1031
Rickaby (Rev. John), 1016
—— (Rev. Joseph), 1016
Ridley (Sir E.), 1018
Riley (J. W.), 1020
Roget (Peter M.), 1016, 1025
Rolfsen (N.), 1008
Romanes (G. J.), 1008, 1015, 1017, 1020, 1032
—— (Mrs.), 1008
Ronalds (A.), 1013
Roosevelt (T.), 1004
Rossetti (Maria Francesca), 1031
—— (W. M.), 1020
Rowe (R. P. P.), 1011
Russell (Bertrand), 1017
—— (Alys), 1017
—— (Rev. M.), 1020
Saintsbury (G.), 1012
Samuels (E.), 1020
Sandars (T. C.), 1014
Sargent (A. J.), 1017
Schreiner (S. C. Cronwright), 1010
Seebohm (F.), 1006, 1008
Selous (F. C.), 1010
Sewell (Elizabeth M.), 1023
Shakespeare, 1020
Shand (A. I.), 1012
Sharpe (R. R.), 1006
Shearman (M.), 1010, 1011
Sinclair (A.), 1011
Smith (R. Bosworth), 1006
Smith (T. C.), 1004
Smith (W. P. Haskett), 1009
Solovyoff (V. S.), 1031
Sophocles, 1018
Soulsby (Lucy H.), 1026, 1031
Spedding (J.), 1007, 1014
Sprigge (S. Squire), 1008
Stanley (Bishop), 1024
Steel (A. G.), 1010
—— (J. H.), 1010
Stephen (Leslie), 1009
Stephens (H. Morse), 1006
Stevens (R. W.), 1031
Stevenson (R. L.), 1023, 1026
‘Stonehenge’, 1010
Storr (F.), 1014
Stuart-Wortley (A. J.), 1011, 1012
Stubbs (J. W.), 1006
Suffolk & Berkshire (Earl of), 1011
Sullivan (Sir E.), 1011
—— (J. F.), 1026
Sully (James), 1015
Sutherland (A. and G.), 1006
—— (Alex.), 1015, 1031
Suttner (B. von), 1023
Swinburne (A. J.), 1015
Symes (J. E.), 1017
Tacitus, 1018
Taylor (Col. Meadows), 1006
Tebbutt (C. G.), 1011
Thornhill (W. J.), 1018
Thornton (T. H.), 1008
Todd (A.), 1006
Toynbee (A.), 1017
Trevelyan (Sir G. O.), 1006, 1007
—— (C. P.), 1017
—— (G. M.), 1006
Trollope (Anthony), 1023
Tupper (L.), 1020
Turner (H. G.), 1031
Tyndall (J.), 1007, 1009
Tyrrell (R. Y.), 1018
Tyszkiewicz (M.), 1031
Upton (F. K. and Bertha), 1026
Van Dyke (J. C.), 1031
Verney (Frances P. and Margaret M.), 1008
Virgil, 1018
Vivekananda (Swami), 1032
Vivian (Herbert), 1009
Wakeman (H. O.), 1006
Walford (L. B.), 1023
Walker (Jane H.), 1029
Wallas (Graham), 1008
Walpole (Sir Spencer), 1006
Walrond (Col. H.), 1010
Walsingham (Lord), 1011
Walter (J.), 1008
Warwick (Countess of), 1031
Watson (A. E. T.), 1010, 1011, 1012, 1013, 1023
Webb (Mr. and Mrs. Sidney), 1017
—— (T. E.), 1015, 1019
Weber (A.), 1015
Weir (Capt. R.), 1011
Weyman (Stanley), 1023
Whately (Archbishop), 1014, 1015
—— (E. Jane), 1016
Whishaw (F.), 1023
White (W. Hale), 1020, 1031
Whitelaw (R.), 1018
Wilcocks (J. C.), 1013
Wilkins (G.), 1018
Willard (A. R.), 1031
Willich (C. M.), 1025
Witham (T. M.), 1011
Wood (Rev. J. G.), 1025
Wood-Martin (W. G.), 1006
Woods (Margaret L.), 1023
Wordsworth (Elizabeth), 1026
—— (William), 1020
Wyatt (A. J.), 1020
Wylie (J. H.), 1006
Youatt (W.), 1010
Zeller (E.), 1015
History, Politics, Polity, Political Memoirs, &c.
Abbott.—A H G .
By E A , M.A., LL.D.
Part I.—From the Earliest Times to the Ionian Revolt. Crown 8vo., 10s. 6d.
Part II.—500–445 B.C. Crown 8vo., 10s. 6d.
Acland and Ransome. A H O P H E 1896. Chronologically Arranged. By the Right Hon. A. H. D A , M.P., and C R , M.A. Crown 8vo., 6s.
Amos.—P E C G . For the Use of Colleges, Schools, and Private Students. By S A , M.A. Cr. 8vo., 6s.
ANNUAL REGISTER (THE). A Review of Public Events at Home and Abroad, for the year 1897. 8vo., 18s.
Volumes of the A R for the years 1863–1896 can still be had. 18s. each.
Arnold. I L M H . By T A , D.D., formerly Head Master of Rugby School. 8vo., 7s. 6d.
Ashbourne. P : S C H L T . By the Right Hon. E G , L A , Lord Chancellor of Ireland. With 11 Portraits. 8vo., 21s.
Baden-Powell.—T I V C . Examined with Reference to the Physical, Ethnographic, and Historical Conditions of the Provinces; chiefly on the Basis of the RevenueSettlement Records and District Manuals. By B. H. B P , M.A., C.I.E. With Map. 8vo., 16s.
Bagwell. I T . By R B , LL.D. (3 vols.) Vols. I. and II. From the first invasion of the Northmen to the year 1578. 8vo., 32s. Vol. III. 1578–1603. 8vo., 18s.
Ball.—H R L S I , from the Invasion of Henry the Second to the Union (1172–1800). By the Rt. Hon. J. T. B . 8vo., 6s.
Besant.—T H L . By Sir W B . With 74 Illustrations. Crown 8vo., 1s. 9d. Or bound as a School Prize Book, 2s. 6d.
Brassey (L ).—P A .
N M . 1872–1893. 2 vols. Crown 8vo., 10s.
M M N , 1871–1894. Crown 8vo., 5s.
I F C 1880–1894. Cr. 8vo., 5s.
P M . 1861–1894. Crown 8vo., 5s.
Bright. A H E . By the Rev. J. F B , D.D.
Period I. M M : A.D. 449–1485. Crown 8vo., 4s. 6d.
Period II. P M . 1485–1688. Crown 8vo., 5s.
Period III. C M . 1689–1837. Crown 8vo., 7s. 6d.
Period IV. T G D . 1837–1880. Crown 8vo., 6s.
Buckle.—H C E . By H T B . 3 vols. Crown 8vo., 24s.
Burke. A H S from the Earliest Times to the Death of Ferdinand the Catholic. By U R B , M.A. 2 vols. 8vo., 32s.
Chesney. I P : a View of the System of Administration in India. By General Sir G C , K.C.B. With Map showing all the Administrative Divisions of British India. 8vo., 21s.
Corbett.—D T N , with a History of the Rise of England as a Maritime Power. By J S. C . With Portraits, Illustrations and Maps. 2 vols. 8vo., 36s.
Creighton.—A H P G S S R , 1378–1527. By M. C , D.D., Lord Bishop of London. 6 vols. Crown 8vo., 6s. each.
Cuningham.—A S I F : a Senate for the Empire. By G C. C , of Montreal, Canada. With an Introduction by Sir F Y , K.C.M.G. Crown 8vo., 3s. 6d.
Curzon. P P Q . By the Right Hon. L C of Kedleston. With 9 Maps, 96 Illustrations, Appendices, and an Index. 2 vols. 8vo., 42s.
De Tocqueville. D A . By A T . Translated by H R , C.B., D.C.L. 2 vols.
Crown 8vo., 16s.
Dickinson. T D P N C . By G. L D , M.A. 8vo., 7s. 6d.
Froude (J A.).
T H E , from the Fall of Wolsey to the Defeat of the Spanish Armada.
Popular Edition. 12 vols. Crown 8vo., 3s. 6d. each.
‘Silver Library’ Edition. 12 vols. Crown 8vo., 3s. 6d. each.
T D C A . Crown 8vo., 3s. 6d.
T S S A , and other Essays. Cr. 8vo., 3s. 6d.
T E I E C . 3 vols. Cr. 8vo., 10s. 6d.
E S S C . Cr. 8vo., 6s.
T C T . Crown 8vo., 3s. 6d.
S S G S . 4 vols. Cr. 8vo., 3s. 6d. each.
C : a Sketch. Cr. 8vo, 3s. 6d.
Gardiner (S R , D.C.L., LL.D.).
H E , from the Accession of James I. to the Outbreak of the Civil War, 1603–1642. 10 vols. Crown 8vo., 6s. each.
A H G C W , 1642–1649. 4 vols. Cr. 8vo., 6s. each.
A H C P . 1649–1660. Vol.I. 1649–1651. With 14 Maps. 8vo., 21s. Vol. II. 1651–1654. With 7 Maps. 8vo., 21s.
W G P W . With 8 Illustrations. Crown 8vo., 5s.
C ’ P H . Founded on Six Lectures delivered in the University of Oxford. Cr. 8vo., 3s. 6d.
T S ’ H E . With 378 Illustrations. Crown 8vo., 12s.
Also in Three Volumes, price 4s. each.
Vol. I. B.C. 55–A.D. 1509. 173 Illustrations. Vol. II. 1509–1689. 96 Illustrations. Vol. III. 1689–1885. 109 Illustrations.
Greville.—A J R K G IV., K W IV., Q V . By C C. F. G , formerly Clerk of the Council. 8 vols. Crown 8vo., 3s. 6d. each.
HARVARD HISTORICAL STUDIES.
T S A S T U S A , 1638–1870. By W. E. B. D B , Ph.D. 8vo., 7s. 6d.
T C R F
C M . By S. B. H , A.M. 8vo., 6s.
A C S N S C . By D. F. H , A.M. 8vo., 6s.
N E O U S . By F W. D , A.M. 8vo., 7s. 6d.
A B B M H , G P R . By C G , Ph.D. 8vo., 12s.
T L F S P N W . By T C. S , Ph.D. 8vo, 7s. 6d.
T P G E C N A . By E B G . 8vo., 7s. 6d.
⁂ Other Volumes are in preparation.
Hammond. A W ’ P R . By Mrs. J H H . Crown 8vo., 2s. 6d.
Historic Towns.—Edited by E. A. F , D.C.L., and Rev. W H , M.A. With Maps and Plans. Crown 8vo., 3s. 6d. each.
Bristol. By Rev. W. Hunt. Carlisle. By Mandell Creighton, D.D. Cinque Ports. By Montagu Burrows. Colchester. By Rev. E. L. Cutts. Exeter. By E. A. Freeman. London. By Rev. W. J. Loftie. Oxford. By Rev. C. W. Boase. Winchester. By G. W. Kitchin, D.D. York. By Rev. James Raine. New York. By Theodore Roosevelt. Boston (U.S.) By Henry Cabot Lodge.
Hunter. A H B I . By Sir W W
H , K.C.S.I., M.A., LL. D.; a Vice-President of the Royal Asiatic Society. In 5 vols. Vol. I.—Introductory to the Overthrow of the English in the Spice Archipelago, 1623. 8vo., 18s.
Joyce (P. W., LL.D.).
A S H I , from the Earliest Times to 1603. Crown 8vo., 10s. 6d.
A C ’ H I . From the Earliest Times to the Death of O’Connell. With specially constructed Map and 160 Illustrations, including Facsimile in full colours of an illuminated page of the Gospel Book of MacDurnan, A.D. 850. Fcp. 8vo., 3s. 6d.
Kaye and Malleson. H I M , 1857–1858. By Sir J W. K and Colonel G. B. M . With Analytical Index and Maps and Plans. 6 vols. Crown 8vo., 3s. 6d. each.
Lang (A ).
P S : or, The Incognito of Prince Charles. With 6 Portraits. 8vo., 18s.
T C P : Being a Sequel to ‘Pickle the Spy’. With 4 Plates. 8vo., 16s.
S . A . With 8 Plates and 24 Illustrations in the Text by T. Hodge. 8vo., 15s. net.
Lecky (The Rt. Hon. W E. H.)
H E E C .
Library Edition. 8 vols. 8vo. Vols. I. and II., 1700–1760, 36s.; Vols. III. and IV., 1760–1784, 36s.; Vols. V. and VI., 1784–1793, 36s.; Vols. VII. and VIII., 1793–1800, 36s.
Cabinet Edition. E . 7 vols. Crown 8vo., 6s. each. I . 5 vols. Crown 8vo., 6s. each.
H E M A
C . 2 vols. Crown 8vo., 12s.
H R I S
R E . 2 vols. Crown 8vo., 12s.
D L .
Library Edition. 2 vols. 8vo., 36s.
Cabinet Edition. 2 vols. Cr. 8vo., 12s.
Lowell. G P C E . By
A. L L . 2 vols. 8vo., 21s.
Macaulay (L ).
T L W L M . ‘Edinburgh’ Edition. 10 vols. 8vo., 6s. each.
C W .
Cabinet Edition. 16 vols. Post 8vo. £4 16s.
Library Edition. 8 vols. 8vo., £5 5s.
‘Edinburgh’ Edition. 8 vols. 8vo., 6s. each.
‘Albany’ Edition. With 12 Portraits. 12 vols. Large Crown 8vo., 3s. 6d. each.
H E A J S .
Popular Edition. 2 vols. Cr. 8vo., 5s.
Student’s Edition. 2 vols. Cr. 8vo., 12s.
People’s Edition. 4 vols. Cr. 8vo., 16s.
‘Albany’ Edition. With 6 Portraits. 6 vols. Large Crown 8vo., 3s. 6d. each.
Cabinet Edition. 8 vols. Post 8vo., 48s.
‘Edinburgh’ Edition. 4 vols. 8vo., 6s. each.
Library Edition. 5 vols. 8vo., £4.
C H E , L R , etc., in 1 volume.
Popular Edition. Crown 8vo., 2s. 6d.
Authorised Edition. Crown 8vo., 2s. 6d., or gilt edges, 3s. 6d.
‘Silver Library’ Edition. With Portrait and 4 Illustrations to the ‘Lays’. Cr. 8vo., 3s. 6d.
C H E .
Student’s Edition. 1 vol. Cr. 8vo., 6s.
People’s Edition. 2 vols. Cr. 8vo., 8s.
‘Trevelyan’ Edition. 2 vols. Cr. 8vo., 9s.
Cabinet Edition. 4 vols. Post 8vo., 24s.
‘Edinburgh’ Edition. 3 vols. 8vo., 6s. each.
Library Edition. 3 vols. 8vo., 36s.
E , which may be had separately, sewed, 6d. each; cloth, 1s. each.
Addison and Walpole.
Croker’s Boswell’s Johnson.
Hallam’s Constitutional History.
Warren Hastings.
The Earl of Chatham (Two Essays).
Frederick the Great.
Ranke and Gladstone.
Milton and Machiavelli.
Lord Byron.
Lord Clive.
Lord Byron, and The Comic Dramatists of the Restoration.
M W
People’s Edition. 1 vol. Cr. 8vo., 4s. 6d.
Library Edition. 2 vols. 8vo., 21s.
S P .
Popular Edition. Crown 8vo., 2s. 6d.
Cabinet Edition. 4 vols. Post 8vo., 24s.
S W L M . Edited, with Occasional Notes, by the Right Hon. Sir G. O. Trevelyan, Bart.
Crown 8vo., 6s.
MacColl. T S P . By the Rev. M
M C , M.A., Canon of Ripon. 8vo., 10s. 6d.
Mackinnon. T U E S : S I H . By J M . Ph.D. Examiner in History to the University of Edinburgh. 8vo., 16s.
May. T C H E since the Accession of George III. 1760–1870. By Sir T E M , K.C.B. (Lord Farnborough). 3 vols. Cr. 8vo., 18s.
Merivale (C , D.D.), sometime Dean of Ely.
H R E . 8 vols. Crown 8vo., 3s. 6d. each.
T F R R : a Short History of the Last Century of the Commonwealth. 12mo., 7s. 6d.
G H R , from the Foundation of the City to the Fall of Augustulus, B.C. 753–A.D. 476. With 5 Maps. Crown 8vo, 7s. 6d.
Montague. T E E C H . By F. C. M , M.A. Crown 8vo., 3s. 6d.
Ransome. T R C G
E : being a Series of Twenty Lectures on the History of the English Constitution delivered to a Popular Audience. By C
R , M.A. Crown 8vo., 6s.
Richman.—A : P D P L
I -R . A Swiss Study. By I B. R , ConsulGeneral of the United States to Switzerland. With Maps. Crown 8vo., 5s.
Seebohm (F ).
T E V C . Examined in its Relations to the Manorial and Tribal Systems, etc. With 13 Maps and Plates. 8vo., 16s.
T T S W : Being Part of an Inquiry into the Structure and Methods of Tribal Society. With 3 Maps. 8vo.,
12s.
Sharpe. L K : a History derived mainly from the Archives at Guildhall in the custody of the Corporation of the City of London. By R R. S , D.C.L., Records Clerk in the Office of the Town Clerk of the City of London. 3 vols. 8vo. 10s. 6d. each.
Smith. C C . By R. B S , M.A., With Maps, Plans, etc. Cr. 8vo., 3s. 6d.
Stephens. A H F R . By H. M S . 8vo. Vols. I. and II. 18s. each.
Stubbs. H U D , from its Foundation to the End of the Eighteenth Century. By J. W. S . 8vo., 12s. 6d.
Sutherland. T H A N Z , from 1606–1890. By A S , M.A., and G S , M.A. Crown 8vo., 2s. 6d.
Taylor. A S ’ M H I . By Colonel M T , C.S.I., etc. Cr. 8vo., 7s. 6d.
Todd. P G B C . By A T , LL.D. 8vo., 30s. net.
Trevelyan. T A R . Part I. 1766–1776. By the Rt. Hon. Sir G. O. T , Bart. 8vo., 16s.
Trevelyan. E T W . By G M T , M.A. 8vo.
[In the Press.
Wakeman and Hassall. E I S E C H . By Resident Members of the University of Oxford. Edited by H O W , M.A., and A H , M.A. Crown 8vo., 6s.
Walpole.—H E C G W 1815 1858. By Sir S W , K.C.B. 6 vols. Crown 8vo., 6s. each.
Wood-Martin. P I : A S . A Handbook of Irish Pre-Christian Antiquities. By W. G. W M , M.R.I.A. With 512 Illustrations. Crown 8vo., 15s. Wylie. H E H IV. By J H W , M.A., one of H.M. Inspectors of Schools. 4 vols. Crown 8vo. Vol. I., 1399–1404, 10s. 6d. Vol. II., 1405–1406, 15s. Vol. III., 1407–1411, 15s. Vol. IV., 1411–1413, 21s.
Biography, Personal Memoirs, &c.
Armstrong. T L L E J. A .
Edited by G. F. S A . Fcp. 8vo., 7s. 6d.
Bacon. T L L F B , O W . Edited by J S . 7 Vols. 8vo., £4 4s.
Bagehot. B S . By W B . Crown 8vo., 3s. 6d.
Boevey.—‘T P W ’: being passages from the Life of Catharina, wife of William Boevey, Esq., of Flaxley Abbey, in the County of Gloucester. Compiled by A W. C B , M.A. With Portraits. 4to., 42s. net.
Carlyle. T C : A History of his Life. By J A F .
1795–1835. 2 vols. Crown 8vo., 7s. 1834–1881. 2 vols. Crown 8vo., 7s.
Crozier. M I L : being a Chapter in Personal Evolution and Autobiography. By J B C , Author of ‘Civilisation and Progress,’ etc. 8vo., 14s.
Digby. T L S K D , by one of his Descendants, the Author of ‘Falklands,’ etc. With 7 Illustrations. 8vo., 16s.
Duncan. A D . By T E C . With 3 Portraits. 8vo., 16s.
Erasmus. L L E . By J A F . Crown 8vo., 6s.
FALKLANDS. By the Author of ‘The Life of Sir Kenelm Digby,’ etc. With 6 Portraits and 2 other Illustrations. 8vo., 10s. 6d.
