Table of Contents
How to Use This Book vi
Frequently Asked Questions x
Foreword xvii
Introduction xix
Chapter 1 Creating the World We Want to Live In 1
The Criticality of Context 2
Defining Multimodality 5
Fluidity over Rigidity 7
Fix the System, Not the Person 7
Leading with Anti-neutrality and Anti-racism 12
A Gender Equity Lens 15
Where No Design Has Gone Before 16
Apply It Now 17
Chapter 2 Capturing Customer Context 19
Blending Theatricality with Design 20
Storytelling for Design 23
The Building Blocks of Storytelling 23
Story as Shared Understanding 36
Apply It Now 39
Chapter 3 Understanding Busy Humans 43
Turn Contextual Awareness into Activity Models 44
Passive Activities: Zoning Out 47
Sustained Activities: Passing the Time 48
Discrete Activities: Getting Things Done 49
Focused Activities: Going with the Flow 51
Live Activities: Can't Miss a Thing 52
Apply It Now 54
Chapter 4 Activity, Interrupted 55
When Interruptions Attack 57
Leaving an Activity Behind 59
Build an Interruption Matrix in Three Easy Steps 61
Putting It All Together 64
Apply It Now 68
Chapter 5 The Language of Devices 69
Visual Output: Displays and Indicators 71
Auditory Output: Sound Effects, Speech, and Music 76
Haptic Output: Touch and Force 82
Kinetic Output: Movement and Motion 87
Ambient Output: The World Around You 89
Apply It Now 91
Chapter 6 Expressing Intent 93
Auditory Input 94
Visual Input- 101
Haptic Input 106
Kinetic Input 111
Ambient Input 114
Apply It Now 116
Chapter 7 The Spectrum of Multimodality 121
Dimensions of Multimodal Experiences 124
Mapping the Multimodal Quadrants 128
Adaptive Interactions (Quadrant 1) 129
Anchored Interactions (Quadrant 2) 131
Direct Interactions (Quadrant 3) 132
Intangible Interaction (Quadrant 4) 132
Choosing Your Interaction Models 133
Orchestrating Multimodal Transitions 134
Apply It Now 137
Chapter 8 It's a (Multimodal) Trap! 141
Synchronization: Do You Have a Shared State? 143
Context: What Just Happened? 145
Discoverability: How Will People Learn? 148
Ergonomics: Are You Hurting People? 152
Identity and Privacy: Whose Turn Is It, Anyway? 155
Apply it Now 158
Chapter 9 Lost in Transition 161
Between Modalities 162
Between Network Connections 167
Between Devices 172
Apply It Now 177
Chapter 10 Let's Get Proactive 181
Taxonomy of Interruption 183
Manifesting Proactive interactions 186
Interruption and Action 196
Cross-Device Considerations 197
Apply It Now 201
Chapter 11 Breathe Life into the Unknown 205
Opti-Pessimism: Explore the Extremes 206
Storyboards: Bring Customer Context to Life 211
Playing Pretend: Put Yourself in Their Shoes 217
Prototyping: Proving Your Concept 220
Apply It Now 228
Chapter 12 From Envisioning to Execution 231
Interactions over Time 233
Voice Design: Visualizing the Invisible 239
Gesture Design: Documenting the Wave 240
Multimodal Design Systems 243
Apply It Now 252
Chapter 13 Beyond Devices: Human + Al Collaboration 257
Defining Al 259
Machine Learning 260
Responding with Confidence 262
The Why of Al 263
Exploring Algorithmic Bias 267
Question Everything 273
(Dis)Embodied Agents 274
Apply It Now 282
Chapter 14 Beyond Reality: XR, VR, AR, and MR 287
No Such Thing as New Ideas 288
An Extended Reality Lexicon 290
Dimensional Interactions 296
Input for Another Dimension 300
The Reality of Virtual Space 307
Apply It Now 311
Chapter 15 Should You Build It? 317
What Harm Could It Do? 320
P: Solve the Right PROBLEM 321
I: Pursue INCLUSION 326
C: CHANGE and Consequences 328
S: A SYSTEMS Perspective 332
Apply It Now 339
Index 343
Acknowledgments 362
About the Author 368
