Tests for success. Experimental design and online conversion tracking
Nik Shah, May 2014
Nik Shah EMEA Retail Measurement Lead
CTR is not a key measure
“There is no correlation between CTR and ROI%” Nielsen NetEffect meta analysis
Correlation = -.07%
Source: Nielsen, Oct 2011
Moving towards a multi-touch attribution model A last-touch attribution model attributes entire value of a conversion to the last ad clicked or seen
✖ Miscalculates ROI by assuming the consumer doesn’t see or click any other ads along the way
A multi-touch attribution model attributes value to observed touch point along the path to conversion
Provides a more realistic assessment of ROI by capturing a holistic picture of the customer journey across channels
Last Click Attribution First Only Prefer First
Facebook undervalued by
12% to 30%
Divide Equally
using Last Click
Prefer Last U-Shaped
Last Ad measurement undervalued Facebook adver4sing rela4ve to each of the five alternate a:ribu4on models.
Source: Quantifying the Impact of Multi-touch Attribution across hundreds of millions of clicks, Retail and FinServ verticals Kenshoo Research, August 2013
More people using more devices
> 60% of people in the UK use at least two devices every day and over 20% use three devices
> 40% of all online adults sometimes start an activity on one device and finish it on another
Multi-Device Usage Study by GfK, Nov - Dec 2013 (commissioned by Facebook). Survey of 2,018 UK online adults.
Even Multi-touch misses the mobile opportunity Last Click
Multi touch
Multi device
Custom Audiences
Experimental design helps cut through this jungle Nov
Nov
16
17
Usage
36% 64%
desktop
mobile Wiggle Events Calendar Wiggle Super Series
Blog ‘help make London a safer place to cycle’
` Gsearch ‘Evans ride it’ Evans ‘Find a ride’
Nov
19
Blog ‘The Right of Way’
‘London to Paris bike ride
Blog ‘The Right of Way’
6h 01m
19
0.4
15.5
Time per day
Categories visited
Searches per day
Social media visits per day
A typical medical trial Pre
Post
Treatment
P Placebo
Essential elements of a lift-test Create control group
Run campaign to test group
Measure Sales among both groups
€€€€€
Test group
€€ Control group
Ways to create a control group Custom Audiences FB holdout functionality Test group
Remember: Randomisation Statistical power Control group
How to run a campaign
Bid Type
Sufficient reach
Other media
Ways to link sales to exposure
€€€€€
FB conversion pixel
FB mobile app SDK
€€
Direct sharing of anonymised transaction information
Online/Multichannel conversion measurement
SDK
1 CLIENT: Identify target audience, e.g. using Custom Audiences
2 FB: Impose holdout on target audience and run ads only at test group
3 FB/CLIENT: Conversion data passed to FB either via SFTP, conversion pixel or SDK
4 FB: Differences in conversion between test and control groups are analysed and uplifts reported
Key takeaways
Comprehensive cross-device data is needed for proper attribution
To measure true incrementality, you need experiments
FB has great experimental design tools – come use them!