1

Time-Turner: Data Engagement Through Everyday Objects in the Home

Abstract Families enjoy capturing digital media about their life and replaying moments, yet it is not always easy to do so. To explore this design space, we created a physical, ambient, and situated visualization prototype called Time-Turner specifically designed for a home setting that records video of family activities and allows families to review their past activities. We report our design requirements, design rationale and the implications of our work for future design researchers.

Author Keywords Ambient systems; situated visualization; data physicalization; tangible visualization.

ACM Classification Keywords H.5.2. Information Interfaces and presentation. User Interfaces – input devices and strategies, prototyping.

Introduction Many people capture and accumulate a massive number of photos and videos associated with family members. Presently, families capture these moments themselves, but in the future, it is likely that automated forms of capturing these moments will be present in the home [5,8]. For example, one way could be through always-on video recording allowing one to revisit their past moments and to reflect upon their

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. DIS'17 Companion, June 10-14, 2017, Edinburgh, United Kingdom © 2017 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-4991-8/17/06. http://dx.doi.org/10.1145/3064857.3079122

Samarth Singhal School of Interactive Arts and Technology Simon Fraser University 102 – 13450 102nd Avenue Surrey, BC, Canada samarths@sfu.ca William Odom School of Interactive Arts and Technology Simon Fraser University 102 – 13450 102nd Avenue Surrey, BC, Canada wodom@sfu.ca

Lyn Bartram School of Interactive Arts and Technology Simon Fraser University 102 – 13450 102nd Avenue Surrey, BC, Canada lyn@sfu.ca Carman Neustaedter School of Interactive Arts and Technology Simon Fraser University 102 – 13450 102nd Avenue Surrey, BC, Canada carman@sfu.ca

2

past experiences. However, one problem with the design of always-on video recording systems is that family members may not be able to search or filter their moments effectively because it could be hard to remember details of one’s past amidst the business of everyday life and with the continued passage of time. Additionally, the lack of material presence that digital archives often exhibit can complicate a family’s ability to richly engage with these materials in everyday life. Prior work has highlighted this as a major challenge for future research in the DIS community (e.g., [17]); however, few design examples have emerged that offer generative and productive ways forward.

Designing a visualization for a family in the context of a home is challenging. In contrast with the typical expert task-centric, focused approach to information visualization, this requires considering visualization methods that would be meaningful and contextually appropriate for family members. To this end, we explored alternatives to screen-based visualizations like physical visualizations, situated visualizations, and ambient displays in our design work.

We created Time-Turner, a video recording system along with a physical situated ambient visualization representing temporal and spatial data of activities within a family’s home. The prototype is composed of three square shaped table-coasters as shown in Figures 1, 2 and 3. Each coaster represents the household activity data at a different granularity (month, day, and second). Family members can explore, query and filter data by time by sliding the edges of the coasters against each other.

Background and Related Work Ambient information systems visualize abstract interpretations of data in the environment or the user’s periphery of attention [1]. Ambient systems such as AmbientRoom [11], Information Percolator [7] and everyday objects like Post-it notes [19] map digital information to output modalities that blend in to the physical environment . These systems illustrate how information can be embedded and visualized in the user’s periphery using output mechanisms that are appealing and yet not distracting. In a situated data representation, the physical representation of data is dependent on the physical referent(s) [24]. Examples include the Dangling String [23] created by artist Natalie Jeremijenko and PinWheels [10]: both use physical motion to convey network traffic in users’ periphery. Situated physicalizations enhance the richness of data by allowing one to examine and extract additional information not present in the original data.

There is a rich history of research in HCI on domestic technology design to better manage everyday life [4]. Our focus is on how families manage data containing recordings of personal or family memories. Kaye et al. [13] describe that digital archives can help people with self-reflection. Taylor et al. [22] highlight the importance of artifacts in the home to organize household routines. Elliot et al. [5] show the vital role of location in providing richness to data. We build on this research by focusing on the design of physical artifacts for families that are meant to be lived with and experienced in the context of routines over time. This is in contrast to other visualization approaches that focus on screen-based designs.

Figure 1: Coaster 1 representing data for a month

Figure 2: Coaster 2 representing data for 24 hours of a day.

Figure 3: Coaster 3 showing the video feed for a timestamp.

3

The Design of Time-Turner We departed from traditional visualization design with a different approach that combined elements of physical, situated and ambient visualizations. Our focus was on a design that could be easily seen at-a-glance, be used throughout the home, and integrate itself within everyday activities of the family. This paper argues that screen-based visualization techniques do not always migrate to non-work environments [2,18].

Video Recording First, Time-Turner records video data of a home using one or more cameras that are set to be always on (the recording features builds on our previous design work and results from our previous study [3,25]). The system captures video along with metadata like the specific location in the house, the number of people in the view, and the activity level of the location. Activity level is comprised of ordinal data calculated in the interval of 0-1. This value is calculated by a pixel-wise comparison to see how much of the image has changed. A value close to zero indicates no activity while a value close to 1 indicates high activity or movement in the video. We selected activity level as the variable to be encoded into our visualization so that family members could know what periods in the past had the most or least activity in their home.

Data Visualization Second, a series of three drink coasters provide physical, ambient, and situated visualization artifacts to support family members’ practices of reflecting and remembering their memories. We selected coasters to embody our system in the form of everyday objects that could fluidly revolve around the house and be

shared by all family members. Each coaster represents information at a different granularity.

Coaster 1 represents data of the entire month. The layout of the LED’s is like the layout of a calendar, placed in a grid of 5x7 (7 days of the week and max 5 weeks) as shown in Figure 1. Coaster 2 represents the data for the entire day like a 24-hour clock aligned in a circle as shown in Figure 2. Coaster 3 contains the most detailed and rich data in the form of images from the always-on-video system as shown in Figure 3. We made the three coasters of varying sizes to give them a stacking order based on the granularity of the data. We start from the smallest coaster at the top which represents information at a low granularity or the overview first. Then, as we move down the stack of coasters, the granularity of the data increases. The size of the coasters indicates the stacking order and hence determines the granularity of the data. Our design rationale is informed by Shneiderman’s visual information mantra of overview first, zoom and filter, then details on demand [21].

REPRESENTATION FIDELITY We decided to use luminance to embody and represent ordinal data associated with activity levels in our data. We used light emitting diodes (LED) for this purpose on our coasters. Our design rationale echo’s Janesen et al. [12] where users can quickly identify which physical variables convey information and how. High luminance represents high activity levels (close to 1) and low represents values close to zero or no activity. We converted this ordinal data into bins to help users easily distinguish the changes in luminance. We use a blinking LED to depict the selected date and hour on coasters.

Figure 4: Coasters stacked per their granularity level.

Figure 5: A user sliding the day coaster along the edge of the month coaster to move forward and backward in the month.

4

By default, this is mapped to the current date and current time exactly one year ago, and keeps transitioning per the current date and time. This transitioning alert falls in to the category of change blind [14] which consumes no conscious awareness of the users but may still affect their behavior [9].

SPATIAL INDIRECTION The coasters change the data being shown as the users move them to different areas of the house, making them situated visualizations. Thus, the coasters show the data for the nearest camera, detected using Wi-Fi triangulation. We feel presenting the data to the user in the context of the physical referent allows the user to gain additional insight and attach value to the location. Previous research has suggests that playing video back in the same location adds an emotional connection to video [6,20], which might enhance the experience.

EXPLORATION BY INTERACTION To support interactivity and allow family members to view their data at different time periods, we added interactivity to the first and second coaster with a novel technique of spatial sliding. This allows users to query and filter their data by the same granularity level of the coasters i.e. by a) date, and b) time of the day. A user can change the data point on a coaster by magnetically attaching the next coaster along its right edge and sliding it upwards or downwards as shown in Figure 5. If it slides in the upward direction, the data point on the left coaster moves backwards in time while sliding downwards causes it to move forward in time. The current selection of these data points on the coasters is indicated by a blinking LED on each coaster. As soon as the coasters are magnetically separated from each other, they return to their default state of displaying

the current date and current hour from one year ago. The goal is to reflect on what was happening at that point in time one year earlier as most of the family rituals (birthdays, anniversaries & festivals) are annual.

Discussion and Conclusion Time-Turner provides family members with ways to explore their past by interacting with two or more coasters. While our focus was on family moments, the design could similarly be used to present other forms of data of spatial-temporal data where meta-data could be encoded by the different properties of the LEDs (hue, luminance, saturation). For example, a family could use the coasters to track their energy consumption [15] by encoding this data by different granularities and locations and later interact with the coasters to look for their usage. In these cases, the video data could then allow users to understand what might have been happening at the moments associated with this data.

We envision that an extensive study of Time-Turner could provide us with insights into the design shaped family members practices of reflecting on their past. To date, we have collected one and half years of video from one of the author’s homes where the family has been exploring their past moments using Time-Turner. The family has found the design invites curiosity around why different days’ glow brighter than others and the design is often a conversation-starter when guests are at the house. Our future work involves transforming Time-Turner into a research product [16] to investigate people’s experiences of use over long periods of time.

Acknowledgements This research was funded by NSERC. We thank Brendan Matkin and Yasamin Heshmat for their contribution.

Usage Scenario We offer a use-scenario to further explore and speculate on potential social experiences of Time-Turner.

Scenario: A father and mother used Time-Turner while having their daily morning coffee. The mother noticed that the activity for today was the highest for the entire month and decided to look more closely. Looking at the Day Coaster, she saw that the activity level was higher during the evening and so they decided to see what was happening in the evening. After interacting with the coasters and moving forward in the hour on the Hour Coaster they found that there was a celebration going on in the home exactly one year ago. They were celebrating their eldest son’s first football game win. The mother and father decided to celebrate this past event. Later in the evening, they surprised their son with the party and used Time-Turner to view their moments of celebration from last year.

5

References

1. Saskia Bakker, Elise van den Hoven, and Berry Eggen. 2015. Peripheral interaction: characteristics and considerations. Personal and Ubiquitous Computing 19, 1: 239–254.

2. Lyn Bartram, Johnny Rodgers, and Kevin Muise. 2010. Chasing the Negawatt: Visualization for Sustainable Living. IEEE Computer Graphics and Applications 30, 3: 8–14.

3. Brendan DeBrincat and Carman Neustaedter. 2015. Moments: Family Video Recording Right Here, Right Now, on That Day. Extended Proceedings of Graphics Interface New York, NY, USA, ACM.

4. Andy Crabtree, Richard Mortier, Tom Rodden, and Peter Tolmie. 2012. Unremarkable networking: the home network as a part of everyday life. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 554-563. DOI=http://dx.doi.org/10.1145/2317956.2318039

5. Kathryn Elliot, Carman Neustaedter, and Saul Greenberg. 2005. Time, ownership and awareness: the value of contextual locations in the home. In Proceedings of the 7th international conference on Ubiquitous Computing (UbiComp'05), Michael Beigl, Stephen Intille, Jun Rekimoto, and Hideyuki Tokuda (Eds.). Springer-Verlag, Berlin, Heidelberg, 251-268. DOI=http://dx.doi.org/10.1007/11551201_15

6. Kathryn Elliot, Mark Watson, Carman Neustaedter, and Saul Greenberg. 2007. Location-dependent information appliances for the home. In Proceedings of Graphics Interface 2007 (GI '07). ACM, New York, NY, USA, 151-158. DOI=http://dx.doi.org/10.1145/1268517.1268543

7. Jeremy M. Heiner, Scott E. Hudson, and Kenichiro Tanaka. 1999. The information percolator: ambient information display in a decorative object. In Proceedings of the 12th annual ACM symposium on User interface software and technology (UIST '99). ACM, New York, NY, USA, 141-148. DOI=http://dx.doi.org/10.1145/320719.322595

8. John Helmes, Caroline Hummels, and Abigail Sellen. 2009. The other brother: re-experiencing spontaneous moments from domestic life. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI '09). ACM, New York, NY, USA, 233-240. DOI: https://doi.org/10.1145/1517664.1517715

9. Stephen S. Intille. 2002. Change Blind Information Display for Ubiquitous Computing Environments. In G. Borriello and L.E. Holmquist, eds., UbiComp 2002: Ubiquitous Computing. Springer Berlin Heidelberg, Berlin, Heidelberg, 91–106.

10. Hiroshi Ishii, Sandia Ren, and Phil Frei. 2001. Pinwheels: visualizing information flow in an architectural space. In CHI '01 Extended Abstracts on Human Factors in Computing Systems (CHI EA '01). ACM, New York, NY, USA, 111-112. DOI=http://dx.doi.org/10.1145/634067.634135

11. Hiroshi Ishii, Craig Wisneski, Scott Brave, Andrew Dahley, Matt Gorbet, Brygg Ullmer, and Paul Yarin. 1998. ambientROOM: integrating ambient media with architectural space. In CHI 98 Conference Summary on Human Factors in Computing Systems (CHI '98). ACM, New York, NY, USA, 173-174. DOI: http://dx.doi.org/10.1145/286498.286652

12. Yvonne Jansen, Pierre Dragicevic, Petra Isenberg, Jason Alexander, Abhijit Karnik, Johan Kildal,

6

Sriram Subramanian, and Kasper Hornbæk. 2015. Opportunities and Challenges for Data Physicalization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3227-3236. DOI: https://doi.org/10.1145/2702123.2702180

13. Joseph 'Jofish' Kaye, Janet Vertesi, Shari Avery, Allan Dafoe, Shay David, Lisa Onaga, Ivan Rosero, and Trevor Pinch. 2006. To have and to hold: exploring the personal archive. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, USA, 275-284. DOI=http://dx.doi.org/10.1145/1124772.1124814

14. Tara Matthews, Anind K. Dey, Jennifer Mankoff, Scott Carter, and Tye Rattenbury. 2004. A toolkit for managing user attention in peripheral displays. In Proceedings of the 17th annual ACM symposium on User interface software and technology (UIST '04). ACM, New York, NY, USA, 247-256. DOI=http://dx.doi.org/10.1145/1029632.1029676.

15. Carman Neustaedter, Lyn Bartram, and Aaron Mah. 2013. Everyday activities and energy consumption: how families understand the relationship. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1183-1192. DOI: https://doi.org/10.1145/2470654.2466153

16. William Odom, Ron Wakkary, Youn-kyung Lim, Audrey Desjardins, Bart Hengeveld, and Richard Banks. 2016. From Research Prototype to Research Product. In Proceedings of the 2016 CHI

Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2549-2561. DOI: https://doi.org/10.1145/2858036.2858447

17. William Odom, John Zimmerman, and Jodi Forlizzi. 2014. Placelessness, spacelessness, and formlessness: experiential qualities of virtual possessions. In Proceedings of the 2014 conference on Designing interactive systems (DIS '14). ACM, New York, NY, USA, 985-994. DOI: http://dx.doi.org/10.1145/2598510.2598577

18. Zachary Pousman, John Stasko, and Michael Mateas. 2007. Casual Information Visualization: Depictions of Data in Everyday Life. IEEE Transactions on Visualization and Computer Graphics 13, 6 (November 2007), 1145-1152. DOI=http://dx.doi.org/10.1109/TVCG.2007.70541.

19. Kathrin Probst, Michael Haller, Kentaro Yasu, Maki Sugimoto, and Masahiko Inami. 2014. Move-it sticky notes providing active physical feedback through motion. In Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction (TEI '14). ACM, New York, NY, USA, 29-36. DOI=http://dx.doi.org/10.1145/2540930.2540932

20. Jason Procyk and Carman Neustaedter. 2014. GEMS: the design and evaluation of a location-based storytelling game. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14). ACM, New York, NY, USA, 1156-1166. DOI=http://dx.doi.org/10.1145/2531602.2531701

21. Ben Shneiderman. 1996. The Eyes Have It: A Task by Data Type Taxonomy for Information

7

Visualizations. Proceedings of the 1996 IEEE Symposium on Visual Languages, IEEE Computer Society, 336–.

22. Alex S. Taylor and Laurel Swan. 2005. Artful systems in the home. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 641-650. DOI=http://dx.doi.org/10.1145/1054972.1055060.

23. Mark Weiser and John Seely Brown. 1996. Designing Calm Technology. POWERGRID JOURNAL 1.

24. Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2017. Embedded Data Representations. IEEE Transactions on Visualization and Computer Graphics 23, 1 (January 2017), 461-470. DOI: https://doi.org/10.1109/TVCG.2016.2598608

25. Yasamin Heshmat, Carman Neustaedter, and Brendan DeBrincat. 2017. The Autobiographical Design and Long Term Usage of an Always-On Video Recording System for the Home. Proceedings of the Conference on Designing Interactive Systems.