Dynamics of user experience: How the perceived quality of mobile phones changes over time

UX WS NordiCHI'06: COST294-MAUSE

Dynamics of user experience: How the perceived quality of mobile phones changes over time Margeritta von WilamowitzMoellendorff Social psychology and decisionmaking Darmstadt University of Technology Alexanderstraße 10, 64283 Darmstadt, Germany wilamowitz@psychologie.tudarmstadt.de Tel: +49 (0) 06151 - 163066

Marc Hassenzahl Economic Psychology, Psychology of Work- and Social Behavior Campus Landau Im Fort 7 76829 Landau, Germany hassenzahl@uni-landau.de

ABSTRACT

Axel Platz Siemens AG, Corporate Technology, CT IC 7 User Interface Design, OttoHahn-Ring 6, 81730 Munich, Germany axel.platz@siemens.com

INTRODUCTION

The way we use and experience an interactive product is a dynamic process. Surprisingly, only a very small number of studies in the context of Human-Computer Interaction (HCI) attempt to track and explain changes in behavior and experience over time. The present paper introduces CORPUS (Change Oriented analysis of the Relationship between Product and USer), an interview technique to reconstruct changes in user experience over a period of more than one year. Preliminary results (addressing the domain of mobile phones) revealed characteristic changes. Whereas pragmatic perceptions remained stable (utility) or even improved over time (usability), hedonic perceptions (stimulation, beauty, communicate identity) exclusively deteriorated. Stimulation showed an accelerated deterioration, caused by increasing habituation, whereas deterioration of beauty and identity was induced by comparisons with other peoples' products. All in all, we argue for a more dynamic perspective to user experience.

Experiencing an interactive product is a dynamic encounter.

Usage episodes stretch over time, each made of a series of single moments. At each moment, we may love the product or hate it; we may experience a usability problem or may find a useful new function; we may feel excited and stimulated or utterly bored. Moreover, as long as an episode consists of more than one moment, we may experience love, hate, excitement or boredom – all within a single usage episode.

This change over time – the dynamics of experience – is an important aspect, which is, however, virtually neglected by current approaches to user experience – or even more broadly – Human-Computer Interaction (HCI). The study of differences between novices and experts [e.g., 6] may be the closest to this objective. However, current approaches to user experience, which emphasize the importance of socalled hedonic needs or benefits (e.g., novelty, change, selfexpression) as a complement to pragmatic needs or benefits (utility, usability), almost never address dynamic issues [4].

Author Keywords

Mobile phone, quality dimensions, time, user experience, hedonic qualities, interview, change incidents

The present study's aim is to bring together the idea of temporal aspects and different quality dimensions, that is, patterns of change in pragmatic and hedonic quality perceptions. It takes up questions such as how the perception of a product (for example as usable) changes over time and which processes or incidents are responsible for change.

ACM Classification Keywords

Categories and Subject Descriptors: H.5.m [Information interfaces and presentation]: Miscellaneous General Terms: Design, Human Factors

The paper starts with a brief review of studies in HCI addressing time as a variable. After this, the CORPUS (Change Oriented analysis of the Relationship between Product and USer) interview technique is introduced and preliminary results of a study with users of mobile phones are presented.

74

UX WS NordiCHI'06: COST294-MAUSE

CHANGE-ORIENTED STUDIES IN HCI

a working environment and are therefore an important source of information. However, such an approach exclusively focuses on usability problems, thereby excluding hedonic quality aspects important for user experience.

Few studies in HCI explicitly addressed time as an important variable [e.g., 1, 5].

The most prevalent are cross sectional studies, which cover only one point in time, but with participants with different levels of experience. For example, Pr端mper and colleagues [6] studied the way novices and experts handle errors when using the computer for standard office tasks. They measured the number of errors and error handling time. Interestingly, experts did not commit fewer errors compared to novices; however, they spent less time with handling them.

This is just one example for a cross-sectional study, which compares different groups of users (novices and experts) to address change in user behavior. Albeit the most prevalent, this cross-sectional approach is problematic. Change is not directly studied, but only inferred from differences between groups of participants. As long as not the same participants are monitored over time, one cannot differentiate between differences due to time or due to interpersonal variation

This overview of studies in HCI demonstrates the lack of studies, taking change over time and hedonic quality dimensions into account. In addition, it highlights general problems with longitudinal study, such as the enormous investment for studies covering months or even years of usage. In the following section, we suggest a timeline oriented interview technique as a viable alternative to longitudinal studies.

THE 'CORPUS'-INTERVIEW AS AN ALTERNATIVE TO LONGITUDINAL STUDIES

The CORPUS (Change Oriented analysis of the Relationship between Product and USer) interview technique focuses on change in the nature of the user-product relationship. It addresses 'expert' users/owners (having used the product for at least 1 to 2 years) and combines the characteristics of two distinct interview forms: partly standardized and episodic interviews.

Longitudinal studies are more suitable to study change.

They cover a particular period of time, with two or more observations or measurements taken from them same participants. They can be further distinguished by the time period covered, the frequency of observation/measurement and the number of participants. The present paper focuses primarily on the time period covered. In the context of HCI, it is useful to distinguish between a micro perspective (e.g., an hour), a meso perspective (e.g., 5 weeks) and a macro perspective, with a scope on years of use and the idea to map the whole product lifecycle.

The more standardized part of the interview is directed at identifying change in the perception of different quality dimensions during the last one to two years. It starts from the now and goes back to the beginning of the relationship with the product. If change occurred, its direction and shape is assessed (e.g., accelerated improvement, steady deterioration). In addition, the interview switches to a more open, episodic technique to further explore underlying conditions and causes of change. Specifically, participants are instructed to tell little stories related to the change. These narratives, so-called "change incidents", give additional insights into underlying causes and conditions for change.

A typical study in HCI with a micro perspective is a usability test. Participants work on a series of tasks, often for an hour or two, while being observed constantly and maybe questioned repeatedly. Although interesting, it seems questionable whether the time period covered by a micro perspective is long enough to really witness changes in behavior or judgment.

To guarantee broad coverage in line with current approaches to user experience, CORPUS elicits change in five distinct quality perceptions (dimensions). They are primarily based on Hassenzahl's [2, 3] model of user experience.

He distinguishes pragmatic and hedonic quality aspects.

Pragmatic dimensions are directly related to task accomplishment. They address "do goals", such as "making a telephone call", "sending an SMS". In contrast, hedonic dimensions are self-referential. They address "be-goals", such as "being cool", "being fashionable". Recently, beauty became of interest as a further quality dimension [7]. Although not directly addressed in Hassenzahl's model, beauty can be understood as a hedonic dimension as well [4].

Studies with a meso perspective are already becoming rare.

An exception is a study by Mendoza and Novick [5], who observed 32 users of software for creating websites over a period of eight weeks. They focused on the general level of frustration and validated a categorization scheme for frustrating usage episodes. The level of frustration decreased over time. In addition, user's reactions to frustrating episodes changed markedly. Whereas in the beginning, participants tended to ask other colleagues more often, later they frequently found ways to fix the underlying problem themselves. These results already hint at dynamic processes.

Specifically the following two pragmatic and three hedonic dimensions were addressed:

Studies with a macro perspective are nearly non-existent in HCI. This is certainly due to the effort, which has to be put into these studies. Sometimes macro data is available in the field of remote usability, where usability problems are collected by so called "report incident buttons" over a longer period of time [1]. Critical incidents provide information about problems encountered while working on real tasks in

75

-

Utility (UT): the ability of a product to provide the necessary functions for given tasks.

-

Usability (US): the ability of a product to provide the functions in an easy and efficient way.

UX WS NordiCHI'06: COST294-MAUSE -

Trend: stable, improving or deteriorating

Stimulation (S): the ability of a product to surprise, to foster curiosity and to provide opportunities for the perfection of knowledge and skills

-

Beauty (B): the ability of a product to evoke a feeling of “beauty�

-

Communicate identity (I): the ability of a product to communicate self-serving symbols to relevant others

In general, the perception of a quality dimension can remain stable (e.g., the product appears today as useful as in the beginning), it can improve (e.g., it appears more useful today than in the beginning), or deteriorate (e.g., it appears less useful today than in the beginning) Table 1 (printed in italics) shows the absolute frequency of the different trends for each quality dimension and the global rating. The gray shade signifies cells with the highest absolute frequency for each dimension.

Additionally, a global evaluation (G) of the product was included as a sixth dimension.

In the structured part of the interview, participants were shown a graph with a timeline as x-axis and a 10-point scale ranging from "not at all" to "very much" as the y-axis. They were asked about their perception of the product on a particular quality dimension (e.g., usability) at the moment ("How much usability does the product has on a 0 to 10 scale?"). This was followed by the same question, however, referring to the beginning of usage. If the two assessments (current and initial perception) differed, the particular shape of change was further explored, for example, by asking, "whether the change occurred in a specific month", "more at the beginning" etc.

Trend / Shape

The interview started with the global evaluation (omitting the elicitation of change incidents), followed by the five quality dimensions in random order.

accelerated

7

steady

1

stable improving accelerated

deteriorating

jump

What are the causes for change? Is it possible to identify particular incidents, which lead to change?

I

G

4

2

3

1

4

1

4

1

1 1

8

5

5

6

1

2

2

4

2

4

1

Shape: steady, accelerated, or jumping

The CORPUS technique asks participants to assess the shape of change for each quality dimension (if change occurred). Three general shapes emerged: accelerated (i.e., sharp improvement/deterioration at the beginning of usage, 0-2 months), steady (i.e., continuous improvement/deterioration), or jump (i.e., sudden increase or decrease due a specific event)

Concerning change over time, three important questions can be answered with the help of CORPUS data:

3.

B

Interestingly, the global rating deteriorated as well (6 out of 8). From this, we may tentatively conclude that in the case of mobile phones global evaluation was rather driven by hedonic than pragmatic considerations

All participants owned and used their mobile phone for at least 12 months. Median usage was 20 months. An interview took about 45 minutes.

Shape: What is the shape of the trend? Is it steady?

Or does it predominantly take place in the beginning or at the end?

8

S

Table 1. Absolute frequency of trends and shape for change in quality perceptions Overall, pragmatic dimensions remained stable or improved, whereas hedonic dimensions deteriorated. Specifically, utility (UT) remained stable; usability (US) remained stable or improved. Stimulation (S) uniformly deteriorated.

Beauty (B) and identity (I) deteriorated (5) or remained stable. Only a single individual reported an improving beauty-

Eight individuals participated in the study (7 female). The participants were students of psychology. They received no compensation for participation.

2.

US

G

jump

Although CORPUS accommodates to all interactive products, we ran our first study in the domain of mobile phones.

For mobile phones both pragmatic and hedonic attributes are considered important and the average product lifecycle is between 18 to 24 months. Later studies will use other products (product categories) and may, thus, be able to reveal product or product category specific patterns of change over time.

Trend: Is there a general trend for each quality dimension? Which quality perceptions improve over time, which deteriorate?

UT

Hedonic

steady

STUDY

1.

Pragmatic

Table 1 shows the absolute frequency of the three different shapes for improvement and deterioration.

In the four cases, where a pragmatic dimension (namely usability) changed over time, participants reported an accelerated improvement. The hedonic dimensions considerably differed. Stimulation revealed an accelerated deterioration,

76

UX WS NordiCHI'06: COST294-MAUSE

whereas the deterioration of beauty was predominantly jumping, i.e., related to specific events. For identity all possible shapes were observed.

Category

Definition

Example

Familiarity

A process of getting to know the product and getting used to it.

“I needed some time to orient, but after two weeks I knew how everything worked” (positive)

Figure 1 summarizes the typical trend and shapes of the five quality dimensions on the basis of the mean elicited ratings.

10 Strengh of perception

“This was really fast, after some months there was nothing new to try” (negative)

utility usability

8

beauty

6

Comparison Users compared their mobile phone to an other one (new, friend's phone) and devaluated it

e

4

identity stimulation

2 0

“My boyfriend got a new one, one you can open with a click and it is more round than mine. I’d like this one more”

Table 2: Definitions and statement examples for the two broad categories

Ti me of use

There are two interesting points to make. First familiarity is obviously two-edged: It increases usability, while at the same time leading to a strong reduction of stimulation. In both cases the self is the focus. Second, beauty and identity are more related to others, specifically to comparison with other. In the present domain, comparisons are exclusively upward, leading to a deteriorated perception of the own product.

Figure 1. Dynamics of each quality dimension (Start and end point of each dimension are based on the mean elicited ratings) Explication

CORPUS asks participants to explicate any change in quality dimensions. To do so, participants told little stories related to or leading to change, so-called change incidents.

Category

Altogether 24 change incidents where gathered. Two broad categories emerged from the data (see Table 2). One group of incidents referred to the familiarity with the product. The other revolved around comparison. Familiarity was further subdivided into "positive", if referring to an improvement, and "negative", if referring to deterioration. Comparison always led to deterioration.

Pragmatic UT

Familiarity (negative) Familiarity

The two groups further differ in their focus, namely the self or others. Positive and negative familiarity is related to the self-oriented process of learning and habituation. Comparisons, however, more specifically focuses on product user or ownership in relation to other persons.

Hedonic US

S

4

B

I

1 7

(positive) Comparison

5

7

Table 3. Number of reasons/change incidents for the different quality dimensions over time.

Table 3 shows the absolute frequency of change incidents in each quality dimension. The gray shade signifies cells with the highest absolute frequency per dimension. Improving usability is in general explained by a growing familiarity with the product. Stimulation was related to negative familiarity (habituation). For beauty and identity all but one change incident was a comparison.

CONCLUSION

Only few studies take the dynamics of quality perceptions into account and – if so – they tend to focus on usability (pragmatic aspects) solely. From this, one may conclude that HCI regards the product-user-relationship as more or less stable over time. This may not be the case.

We argue for a more dynamic, change-oriented approach to user experience. To this objective, the present study combined the idea of pragmatic and hedonic quality dimensions with a focus on how according quality perceptions change over time. The resulting CORPUS interview technique is, thus, specifically designed to track and explain the dynam-

77

UX WS NordiCHI'06: COST294-MAUSE

The present study revealed how mobile phones change their character over time: from the more hedonic to the more pragmatic. Other products (e.g., investment goods) may have other dynamics. CORPUS gives the opportunity to gain first – and admittedly primarily exploratory – insights into the way user experience changes over time. This adds an important, but often neglected aspect to the study of user experience.

ics of user experience. In a second step the results could be improved by comparing the results with longitudinal or cross-sectional studies.

Our preliminary results revealed interesting differences in the dynamics of hedonic and pragmatic quality dimensions, which are in line with common sense and model predictions (e.g., derived from [3]). First, pragmatics improved over time, whereas hedonics deteriorated. Interestingly, usability and stimulation were driven by the same, underlying cause, namely familiarity. Familiarity led to a better understanding of the product and, thus, to a better perceived usability. At the same time, however, the product lost its power to surprise and stimulate. Note, that this does not necessarily imply a causal relation between both dimensions. Very different features of a product may make it usable or stimulating.

Additionally, the exact time course may be of further interest. Our crude reconstruction of the shape of change showed stimulation to deteriorate much stronger (and faster) than usability to improve. Future criteria for a "good" interactive product may be rather the way (speed, strength) a quality dimension improves or deteriorates over time, than its absolute value. In other words, a product with a slightly lower initial perceived usability, but a fast and strong "learning curve", may be "better" than a product with a slightly higher initial usability, but no improvement over time at all.

ACKNOWLEDGMENTS

This research is part of the project "Dynamic quality dimensions in interactive systems" done in collaboration with and supported by Siemens, Corporate Technology (CT IC 7), Munich.

REFERENCES

1. Hartson, H.R., & Castillo, J.C. Remote evaluation for post-deployment usability improvement. Proceedings of AVI 98, (1998), 22-29.

2. Hassenzahl, M., Platz, A., Burmester, M. & Lehner, K. Hedonic and ergonomic quality aspects determine software’s Appeal [technical paper]. Proceedings of the CHI 2000 Conference on Human Factors in Computing, New York: ACM (2002), 201-208.

3. Hassenzahl, M. The thing and I: understanding the relationship between user and product. In Funology: From Usability to Enjoyment, M. Blythe, C. Overbeeke, A.F. Monk and P.C. Wright (Eds), (2003), 31-42.

Second, Hassenzahl [3] argued that stimulation is primarily self-related, whereas identity (and beauty) is rather a social phenomenon. Indeed, causes for deteriorating hedonic dimensions differed accordingly. Whereas causes for a deteriorating stimulation were self-related (familiarity, habituation), causes for a deteriorating identity and beauty were based on comparisons with others. This supports the assumed different nature of stimulation and identification/beauty.

4. Hassenzahl, M., & Tractinsky, N. User experience – a research agenda. Behaviour and Information Technology, 25(2), (2006), 91-97.

5. Mendoza, V. Novick, D.G. Usability over time, Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information, (2005).

Interestingly, the dynamics of the global evaluation resembled the hedonic dimensions rather than the pragmatic dimensions. This may lead to the tentative conclusion that the steady deterioration of hedonics (induced by comparisons) is the driving force behind a weakened user-productrelationship. Indeed, Hassenzahl [3] argued that hedonics rather than pragmatics drive bonding to a product (i.e., strong positive relationship), which is supported by the present findings.

6. Prümper, J., Zapf, D., Brodbeck, F. C., & Frese, M. Some surprising differences between novice and expert errors in computerized office work. Behaviour & Information Technology, 11, (1992), 319-328.

7. Tractinsky, N., Katz, A. S. & Ikar, D. 'What is beautiful is usable', Interacting with computers13 (2), (2000), 127-145.

78