Mobile Softwareteknologier Interaktionsdesign og usabilityevaluering

2 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

3 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

4 Status fra sidste gang Kort status pr. gruppe Videre arbejde med øvelsen

5 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere User-Centred Design Prototyper Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

6 Involvering af brugere Mål: involvere brugerne aktivt i udviklingsarbejdet Brugerne er med til at tage designbeslutninger User-Centred Design henføres til Norman (1986) UCD ændrede usability engineering fra evaluering af menneskelig performance og fejl til en sammenkobling af design- og evalueringsaktiviteter i en samlet udviklingsproces Mange har arbejdet med det siden

7 Adoption of UCD Techniques Venturi et al Investigated UCD adoption through a web survey Respondents: 83 The early involvement of UCD practitioners in the product life cycle is more frequent compared to ten years ago The methods and the techniques employed have shifted their focus from summative evaluation to rapid development cycles and from quantitative to qualitative evaluation methods.

8

9

10 Prototype Definition: En repræsentation af et design. Kan bruges til at afprøve kvaliteten af dette design i forhold til bestemte kriterier To kategorier – ud fra nærhed til produkt Low fidelity: designskitser på papir High fidelity: emulator, værktøj

11 Evaluering af prototyper Grundlæggende: samme procedure som andre evalueringer Afhænger af kategorien TypeAdvantagesDisadvantages Low fidelityLess timeLittle use for usability test Lower costNavigation and flow limitations Evaluate multiple conceptsFacilitator driven Useful for communicationPoor detail in specification Address screen layout issues High fidelityPartial functionalityCreation time-consuming InteractiveInefficient for proof-of-concept User-drivenBlinds users to major representation flaws Clearly defined navigation schemeUsers may think prototype is 'real' Use for exploration and test Marketing tool

12 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Design og evaluering Begreb, aktiviteter og resultater Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

13 Design og evaluering: Samspil User interface design provides design products: The system An operational prototype A user interface specification A paper prototype … Usability evaluation provides feedback to user interface design This feedback forms the basis for redesign and further development This is called formative evaluation

14 Usability: Concept Usability Effective to use (effectiveness) Efficient to use (efficiency) Safe to use (safety) Have good utility (utility) Easy to learn (learnability) Easy to remember how to use (memorability) Experience Satisfying Enjoyable Fun Entertaining Helpful Motivating Aesthetically pleasing Supportive of creativity Rewarding Emotionally fulfilling A usability problem is something that reduces the usability of the system

15 Usability Evaluation A usability evaluation is a set of activities that produces a coherent assessment of the usability of a software system A usability test is a limited activity that contributes to the evaluation, e.g. one user applies the system to solve a number of tasks A usability evaluation is documented in some way

16 Activities in a Usability Evaluation When in development process: Exploratory Assessment Validation Comparison Location and equipment Participants and roles Determine basics Plan process Create test situation Interpret data Conduct test Recreation of context Selection of test subjects Design of tasks Test plan: Purpose Key questions User profile Test method Task list Context and equipment Test monitor role Data to be collected Report structure Introduction Task solving Data collection Logging Debriefing Transcription of log files Data summary Data analysis Documentation (report)

17 Result of a Usability Evaluation The result of a usability evaluation is typically documented in a usability report Characteristics: pages usability problems: A list and a detailed description of each Problems are identified through user- based tests Problems are categorized (cosmetic, serious, critical) Additional: TLX and task completion times Log files transcribed from the video ( pages) This report is the feedback 1. Executive summary 2. Method a) Purpose b) Procedure c) Test participants d) Test procedure e) Location & equipment f) Identification & categorization of problems 3. Results a) Workload (NASA-TLX) b) Time used c) Problem overview d) Detailed description of problems 4. Conclusion 5. Appendix a) Tasks b) Interview guide c) Questionnaires d) Video log-files e) System log-files f) Task solutions 1. Executive summary 2. Method a) Purpose b) Procedure c) Test participants d) Test procedure e) Location & equipment f) Identification & categorization of problems 3. Results a) Workload (NASA-TLX) b) Time used c) Problem overview d) Detailed description of problems 4. Conclusion 5. Appendix a) Tasks b) Interview guide c) Questionnaires d) Video log-files e) System log-files f) Task solutions

18 Key Result: Usability Problems

19 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium MobileWard: Is it worth the hassle? NetMill: It’s worth the hassle! Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

20 Two Evaluation Approaches Field experiments Realistic use context Difficult to control Complicated data collection Complex and time consuming Safety and ethical issues Laboratory experiments Experimental control High quality data collection Lack of realism

21 Laboratory vs. Field Most usability evaluations of mobile systems are currently conducted in laboratory settings A recent literature study revealed that… 41% of mobile HCI research involve evaluation 71% of this is done in laboratory settings It is a widely adopted point of view that mobile systems require field evaluations, but… It is difficult to conduct field evaluations The added value of testing in the field is unknown Additional problems come at a high cost (time & effort)

22 MobileWard: Method Laboratory evaluation Lab at Aalborg University, Denmark 6 test subjects (trained nurses) Tasks derived from user study Laboratory furnished as hospital, divided into two wards + corridor Field evaluation Frederikshavn Hospital, Denmark 6 test subjects (trained nurses) No specified tasks Involving real work activities

23 Findings (1) 37 different usability problems Lab evaluation resulted in 36 problems 8 critical, 18 serious, and 10 cosmetic Field evaluation resulted in 23 problems 7 critical, 10 serious, and 6 cosmetic Primarily more serious and cosmetic problems

24 Findings (2) More problems per session 18.8 (2.0) problems versus 11.8 (3.3) problems (U=2.651, p<0.01) Critical: 5.3 (1.2) and 4.5 (2.2) problems Serious: 7.5 (1.0) and 4.5 (0.8) problems Cosmetic: 6.0 (0.9) and 2.8 (1.0) problems Identified significantly more serious (U=2.79, p<0.01) and cosmetic problems (U=2.84, p<0.01)

25 Conclusions Was it worth the hassle? Not really, at least not for usability problem identification However, the real use situation provided additional information on use Replicating the context – always possible? Lab evaluation without context replication Field evaluation with task assignments

26 NetMill: Method Two user-based usability evaluations: A usability laboratory A field-based setting System: mobile system used by tradesmen and workers to register use of time and materials Test subjects: 14 tradesmen students on a technical high school Tasks: Nine tasks, identical for the two settings, except for a single task where the field evaluation included a physical aspect in order to complete the task. A pre-questionnaire was made to gather data of the participant’s experience with different types of information technology A NASA-TLX test A post-questionnaire to reveal the participant’s subjective opinion Two separate teams with a test monitor and a logger conducted the two evaluations

27 Data Collection in the Two Settings Field Laboratory

28 Findings (1) Same amount of critical and serious problems More cosmetic problems in the field

29 Findings (2) Signifikant forskel på kritiske problemer

30 Conclusion Two evaluations conducted in the field and in a usability laboratory using identical data collection techniques The field-based evaluation was more successful in uncovering usability problems: It identifies significantly more problems It was only in this evaluation we identified problems related to cognitive load and interaction style 58% of the identified usability problems were unique to either of the two settings, but the more severe a problem was, the more likely it was to be identified in both evaluations. The context of use influences the usability of a system, which indicates that more realistic context settings can help provide more valid information about the overall usability of a system The overall conclusion of is that it is worthwhile conducting user-based usability evaluations in the field, even though it is more complex and time consuming

31 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

32 Øvelse 1: Planlægning Planlæg en usability-evaluering af jeres indkøbsvogn Overvej og fastlæg de basale forhold Lav en testplan Lav de opgaver, brugeren skal udføre Præsenter eventuelle vanskeligheder

33 Det mobile usabilitylaboratorium: problemet

34 Det mobile usabilitylaboratorium: løsningen (1)

35 Det mobile usabilitylaboratorium: løsningen (2)

36 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

37 Activities in a Usability Evaluation When in development process: Exploratory Assessment Validation Comparison Location and equipment Participants and roles Determine basics Plan process Create test situation Interpret data Conduct test Recreation of context Selection of test subjects Design of tasks Test plan: Purpose Key questions User profile Test method Task list Context and equipment Test monitor role Data to be collected Report structure Introduction Task solving Data collection Logging Debriefing Transcription of log files Data summary Data analysis Documentation (report)

38 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

39 Hvad er kontekst? Fysisk kontekst: hvordan bruges systemet Fysisk lighed, for eksempel indretning som brugskonteksten (en bank, en forretning, …) Teknologisk kontekst: den teknologi, som er tilgængelig for brugeren For eksempel netværk, devices Social kontekst: den sociale situation, som systemet bruges i Andre kontekster De 5 W'er (fem former for kontekst) Who Where What When Why

40 Teknikker til evaluering af mobile systemer Pirhonen et al. (2002) Windows Media Player på en PDA Gestures som interaktionsform Lavede en usability-evaluering I evalueringen forsøgte man at genskabe en mobil kontekst

41 To teknikker Teknik 1 Forhindringer på en gang Lærred med opgaver Teknik 2 Trappemaskine Videooptagelse

42 New Techniques: Two Experiments Kjeldskov & Stage (2004) Two experiments comparing techniques for lab- and field-based usability testing of mobile systems were conducted Experiments explored different techniques requiring… Different levels of physical movement Divided cognitive attention Example application: use of Short Message Service (SMS) on PDAs and mobile phones

43 Five Laboratory Techniques Attention needed to navigate NoneConscious Body Motion None1. Sitting at a table or standing n/a Constant2. Walking on a treadmill with constant speed 4. Walking at constant speed on a changing track Varying3. Walking on a treadmill with varying speed 5. Walking at varying speed on a changing track

44 The Laboratory Experiments 5 conditions (6 test subjects per condition) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

45 The Field Experiment 1 condition: walking in a pedestrian street (6 test subjects) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

46 Comparison: Usability Problems TechniquesTotal Lab 1Lab 2Lab 3Lab 4Lab 5Field Critical Serious Cosmetic Total No technique identified all problems Most problems found when seated at table (34) Statistical significance Comparable numbers of critical problems found (3-4) More than double the number of cosmetic problems were found while seated compared to the other lab techniques Number of identified usability problems categorized by severity

47 Comparison: Workload Techniques Lab 1Lab 2Lab 3Lab 4Lab 5Field Mental demands Physical demands Effort Overall workload Sitting at a table (lab 1) required significantly less mental activity compared to all other techniques but lab 2 Overall, sitting or walking at constant speed is experienced significantly less demanding than any other technique Subjective experience of workload with the different techniques

48 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

49 Øvelse 2 Lav usability-evalueringen af jeres indkøbsvogn Resultater: Problemliste Erfaringer med at teste mobilt Lav det på slides til præsentation (10 min. pr. gruppe) Demo af systemet Fundne usability-problemer Lessons learned

50 Kursusgang 6: Usability-evaluering af mobile apparater Status fra sidste gang Involvering af brugere Planlægning af en usability-evaluering Felt eller laboratorium Øvelse 1: Planlægning af usability-evaluering Gennemførelse af en usability-evaluering Teknikker til (gen)skabelse af kontekst Øvelse 2: Udførelse af usability-evaluering Præsentation af resultater

51 Præsentation af resultater Præsentation: kort demo af systemet Fundne usability-problemer Lessons learned