Alexander, J., Roudaut, A., Steimle, J., Hornbaek, K., Alonso, M. B.,Follmer, S., & Merritt, T. (2018). Grand Challenges in Shape-Changing Interface Research. In CHI 2018: Proceedings of the 2018CHI Conference on Human Factors in Computing SystemsAssociation for Computing Machinery (ACM).https://doi.org/10.1145/3173574.3173873

Peer reviewed version

Link to published version (if available):10.1145/3173574.3173873

Link to publication record in Explore Bristol ResearchPDF-document

This is the author accepted manuscript (AAM). The final published version (version of record) is available onlinevia ACM at https://dl.acm.org/citation.cfm?doid=3173574.3173873. Please refer to any applicable terms of useof the publisher.

University of Bristol - Explore Bristol ResearchGeneral rights

This document is made available in accordance with publisher policies. Please cite only thepublished version using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/user-guides/explore-bristol-research/ebr-terms/

Grand Challenges in Shape-Changing Interface ResearchJason Alexander Anne Roudaut Jürgen Steimle

Lancaster University University of Bristol Saarland UniversityLancaster, UK Bristol, UK Saarbrücken, Germany

j.alexander@lancaster.ac.uk roudauta@gmail.com steimle@cs.uni-saarland.de

Kasper Hornbæk Miguel Bruns Alonso Sean Follmer Timothy MerrittUniversity of Copenhagen TU Eindhoven Stanford University Aalborg University

Copenhagen, Denmark Eindhoven, The Netherlands Palo Alto, CA, USA Aalborg, Denmarkkash@di.ku.dk mbruns@tue.nl sfollmer@stanford.edu merritt@cs.aau.dk

Figure 1. Example shape-changing interfaces: (a) Morphees, a shape-changing mobile phone [?]; (b) An elastic deformable display [?]; (c) inFORM, aself-actuated pin-array [?]; (d) ShapeClip, a prototyping toolkit for shape-changing interfaces [?].

ABSTRACTShape-changing interfaces have emerged as a new methodfor interacting with computers, using dynamic changes in adevice’s physical shape for input and output. With the ad-vances of research into shape-changing interfaces, we see aneed to synthesize the main, open research questions. Thepurpose of this synthesis is to formulate common challengesacross the diverse fields engaged in shape-change research, tofacilitate progression from single prototypes and individualdesign explorations to grander scientific goals, and to drawattention to challenges that come with maturity, includingthose concerning ethics, theory-building, and societal impact.In this article we therefore present 12 grand challenges forresearch on shape-changing interfaces, derived from a three-day workshop with 25 shape-changing interface experts withbackgrounds in design, computer science, human-computerinteraction, engineering, robotics, and material science.

ACM Classification KeywordsH.5.2. Information Interfaces and Presentation (e.g. HCI):User Interfaces

Author KeywordsShape-changing interfaces; Grand Challenges; ResearchAgenda.

INTRODUCTIONShape-changing interfaces provide the opportunity to funda-mentally transform human interaction with computing ma-chines. Users will change from reading and touching flat glassdisplays to physically manipulating interfaces that transformtheir shape and materiality to represent the underlying contentand context (see Figure 1). Examples of such interfaces arenumerous, and include handheld mobile devices [?, ?], table-top surfaces [?, ?], furniture [?], and architecture [?]. Each ofthe examples in the literature are unique, and quite diverse intheir goals, fidelity, input/output capabilities, and complexityof construction. However, the overarching challenges faced intheir development are often common.

While prior surveys have discussed challenges from a fewdifferent disciplinary perspectives [?, ?, ?], the overarchinggrand challenges facing the development of shape-changinginterfaces have not been systematically explored. With thematuring of the field, it becomes increasingly important toidentify a roadmap for future research, as new questions haveemerged. These questions result from the acceleration of thetechnologies for implementing shape-change (e.g. materials-science, electrical- and mechanical-engineering, bio-science),the increasing number of disciplines involved in shape-changeresearch (e.g. design, psychophysics, philosophy), the increas-ing number of real-world deployments, and a range of ethicaland societal implications that are becoming pertinent as thisresearch grows. In short, the challenges facing the field mightboth be expanding (through new disciplines and technology)and converging (through the advances in the field).

To help form a research agenda and to give cross-disciplinaryresearchers exposure to the diverse array of challenges, thisarticle synthesizes the grand challenges that must be addressed

for shape-changing interfaces to move from vision to success-fully embedding themselves in everyday life. These challengeswere generated, explored, and distilled during a three-dayworkshop with 25 experts in shape-changing interfaces, withbackgrounds in design, computer science, human-computerinteraction, engineering, robotics, and material science.

We envision researchers and practitioners from various fieldswill use this article to: (1) identify areas of opportunity in thefield where they can contribute; (2) situate their work withinthe larger shape-changing interfaces research agenda; (3) setout new directions for researchers in their own fields; (4)identify current knowledge and capabilities; (5) allow policymakers to better understand the community, state-of-the-art,and potential applications.

WHAT ARE SHAPE-CHANGING INTERFACES?Vision and DefinitionThe vision for shape-changing interfaces is for interactive com-putational devices to transform into any shape or materialityrelevant to the context of use. This vision has its roots inSutherland’s Ultimate Display where he describes a computerthat can “control the existence of matter” [?]. Shape-changinginterfaces change our fundamental approach to interaction de-sign, expanding interactive systems to include our perceptualmotor skills to support the same direct interaction our body haswith the everyday world. They take advantage of our hapticand kinaesthetic senses, our instinctive perception of physical3D forms, and provide inherent support for multi-user interac-tion. More specifically, we define a shape-changing interfaceto:

1. Use physical change of shape or change in materiality asinput and/or output [?].

2. Be interactive and computationally controlled.3. Be self-actuated [?] and/or user-actuated.4. Convey information, meaning, or affect.

These scoping statements describe the fundamental nature ofa shape-changing interface (1) and are widely inclusive ofdifferent forms of interaction (3). We limit this inclusivityto ensure inanimate objects (such as slow-response memoryfoam) are not included (2) and that these devices must have apurpose or benefit by conveying meaning (e.g. through data,purpose, or affordance) (4).

Describing Shape-Changing InterfacesThe design space of shape-changing interfaces is vast; to cap-ture this, several taxonomies define elements of shape-change.Coelho and Zigelbaum [?] describe their technological prop-erties, including the power requirements, ability to memorizenew shapes, input stimulus or ability to sense deformations.Rasmussen et al. [?] present a review of existing work onshape-changing interfaces, identifying eight types of deforma-tion (orientation, form, volume, texture, viscosity, spatiality,adding/subtracting, permeability). Roudaut et al. [?] proposethe term shape-resolution to extend the definition of displayresolution to shape-changing interfaces. Finally, Sturdee andAlexander [?] categorise shape-changing interface prototypeswith a view to informing their design and application. Thereader is referred to these works for full reviews of the field.

Figure 2. Five purposes of shape changes in end-user interactive devices.

Shape-Changing Interfaces as a Research FieldShape-changing interface research has its roots in TangibleUser Interfaces [?] and Organic User Interfaces [?] and drawson expertise from a range of disciplines. Current work hasmany different foci including engineering (e.g. robotics, mate-rial engineering), behavioural science, and (interaction) design.Each of these individual fields have goals of their own. Forexample, haptics researchers strive to generate and ‘display’kinaesthetic and tactile stimuli to the user [?]; while roboticsresearchers seek to build autonomous agents that accomplishgiven tasks through interactions with the environment [?].These foci allow researchers to investigate the problem spacefrom several directions, often combining diverse methodologi-cal approaches.

However, research into shape-changing interfaces has its owngoals and constraints that make it a unique field. These spanthe technical challenges of complexity and robustness, to users’emotional response, and the device’s wider impact on soci-ety. Our belief is that the unique challenges, constraints, andthe interactive nature of shape-changing interfaces will drivescientific breakthroughs in technology, perception, and HCIthat would be fundamentally different than if only approachedfrom one of these other fields. The field of shape-changinginterfaces has a great opportunity to bridge between these dif-ferent disciplines, contributing not only new research, but alsoto the cross pollination of concepts and techniques.

PURPOSES AND BENEFITSTo invest resources into addressing the grand challenges out-lined in this article, researchers and target user-groups mustsee clear benefits in the development of shape-changing inter-faces. This section briefly explores the purposes and benefitsfrom the literature (summarized in Figure 2). We see ‘bene-fits’ as tightly interwoven with the device’s (current) intendedpurpose. The categories are adapted and updated from thepurpose categories presented by Rasmussen et al. [?], usingthe output from a three-day workshop (detailed later).

Adaptive AffordanceBy adapting its shape to specific contexts of interaction, ob-jects can increase their usefulness and extend their reusabilitythrough changing their physical affordances. The literature dis-cusses these changes as dynamic affordance [?, ?], just-in-timeaffordance [?] or dynamic ergonomics [?].

Adaptation to the task involves shape-change to facilitate theexecution of an action, such as a mobile phone that mutatesinto a game controller [?], increasing device reusability. Adap-tation to the user(s) involves shape change to facilitate theexecution of an action by a specific user(s), for instance, hid-ing screen content from a passerby [?]. Adaptation to theenvironment involves shape-changes to support or influencea specific environment; for example, quiet shape-changingmessage notifications that avoid disrupting a meeting [?, ?].

Augment UsersAugmentation allows the end-user to physically change ele-ments of or within their body. While examples of technolo-gies implanted inside the user’s body are rare, we envisioncochlear implants becoming interactive to allow adaptationwithout surgery. External augmentations actuate or becomeextensions of the user’s body. Many examples of externalaugmentation come from the robotics community (e.g. pros-thesis or the sixth finger device [?]) to enhance manipulationdexterity. Conversely, smart fabric can change its stiffnessaround the user’s joints to restrict their movement in gaming,training, or rehabilitation [?].

Simulate ObjectsShape-changing interfaces have a rich set of applicationsaimed at simulating static or interactive objects. There areseveral classes of simulations: (1) Recreating existing realworld objects (e.g. car seat adjustment [?] or for remotesurgery); (2) Simulating objects that do not yet exist (e.g.in a shape-changing CAD system); (3) Exposing objects thathumans could not normally understand (e.g. molecules [?]).Simulating these objects can be achieved through virtual sim-ulation, where users are immersed in virtual content withshape-changing devices providing physical haptic feedback.Physical simulation produces a visual and physical represen-tation of an object that the user can manipulate [?] (e.g. amodular construction kit to represent a beating heart [?]).

Communicate InformationShape-changing interfaces encode and communicate infor-mation to their users through combinations of visual, haptic(tactile and/or kinaesthetic), and shape animation (transitionsbetween states). A key example of this communication is thenew field of data physicalization [?] that extends informationvisualization into the physical space; for example, by allowingusers to physically manipulate a data-set using a dynamic barchart [?, ?]. This physicalization allows end-users to exploittheir lifelong training in visual and haptic perception.

The haptic element of shape-change can also enable eyes-freeinteraction—a handheld shape-changing device can informthe user of the directions to take when navigating a map [?, ?]or of incoming messages [?]. Shape may also communicatethe state of virtual content [?] in multi-user scenarios.

Hedonic and Symbolic PurposesShape-change can be used for purely hedonic or symbolicpurposes—those that have aesthetic, sensorial, or ‘fun’ goalsrather than functional goals [?]. Example hedonic devicesinclude shape-changing garments [?], architecture [?] andworks of art [?]. These interfaces are typically used to increasethe pleasure or enjoyment of the user(s).

GRAND CHALLENGESOur aim is to identify grand challenges in research on shape-changing interfaces. We are inspired by similar attempts inother fields—for example, robotics [?, ?], data physicaliza-tion [?], and cross-device interactions [?]. We believe formal-ising these challenges is timely for this field, whilst acknowl-edging the discussion around their usefulness [?].

The research challenges in this document were produced overa half-year process. The process was started with a 3-dayworkshop involving 25 researchers who were experts in shape-changing interfaces, from a variety of disciplines, includingdesign, computer science, human-computer interaction, en-gineering, robotics, and material sciences. The workshopaimed to discuss the current state-of-the-art in shape-changinginterfaces, explore the research challenges of this emergingfield, and develop a research agenda for the area. In particular,it included sessions on challenges for the field, benefits ofshape-change, and worst-case scenarios. It was synthesizedinto a preliminary report by the four organizers of the seminarand provided the basis for the current paper. In this way, theprocedure is similar to a Delphi study [?] but with more se-lective reporting. In particular, the challenges presented wereregarded of high importance to driving research in this area.We interpret ‘high importance’ challenges as those that arefundamental to the field’s development, and real-world appli-cation and deployment. It specifically omits challenges thatare common across research areas (for example, the challengesof cross-disciplinary research) and focusses on those that areunique to shape-changing interfaces.

In total, we report 12 grand challenges. The cross-disciplinarynature of this research area is reflected in our categorisationof the challenges into technological, behavioural, design, andsocietal challenges. Figure 3 shows a summary of the grandchallenges we extracted and organized. During the workshopwe attempted to prioritise the challenges. However, it quicklybecame apparent that the strong interconnectivity betweenthe challenges meant it was difficult to isolate and prioritiseon “importance”; this article therefore does not offer such aprioritisation. Other classifications, such as on pragmatics ofresearch are of course possible.

TECHNOLOGICAL CHALLENGESDeveloping shape-changing interfaces entails numerous tech-nical challenges on multiple dimensions and in multiple areas.We describe the challenges around toolkits, miniaturisation,integration of shape-change with other I/O modalities, andnon-functional requirements.

Toolkits for Prototyping of Shape-changing HardwareWe see a need to develop toolkits that help prototype thesoftware and hardware components of shape-changing inter-

Figure 3. Grand challenges for shape-changing interfaces span technological, behavioural, design, and societal concerns.

faces. This is challenging because prototyping shape-changinginterfaces requires knowledge of complex electronics and me-chanical engineering that go beyond that typically required inother areas of interactive computing—software programmingor simple electronics. This intricate combination of technolo-gies will require further development in physical programmingtools [?], rapid prototyping of shape-changing interfaces [?, ?],or technologies such as 4D printing [?]. Furthermore, specificuse contexts may require careful selection among actuationapproaches [?], that cover electromechanical, pneumatic, orhydraulic actuation, or smart materials [?]. Each of these mainclasses comprise a multitude of different approaches meaningit is currently excessively difficult to prototype shape-changinginterfaces.

We imagine several strands of research to address this chal-lenge: (1) a standard platform for hardware prototyping, deal-ing with some aspects of actuation; (2) a cross-platform soft-ware layer for applications and; (3) tools for end-user program-ming. Recent research has started to address this challengeat all three layers [?, ?, ?]. The next steps are to better un-derstand requirements from existing prototypes and taxonomypapers before creating a platform that enables the sharing ofknowledge and code across the whole community.

The purpose of toolkits is to dramatically lower the barrierto implementation. A suitable goal is to reduce the imple-mentation effort of classic interfaces (e.g. InForm [?] or Pin-Wheels [?]) by at least a factor of 10 in time and cost.

Miniaturized Device Form Factors and High ResolutionA significant challenge remains in the availability of smallform-factor, minimal weight, and high resolution actuators.In line with more general computing trends, shape-changinginterfaces are moving from stationary to mobile to wearableform factors [?, ?], and from rigid to flexible to stretchable andeven floating shapes [?, ?, ?, ?]. The use of electromechanicalactuators, as is common in many systems (e.g. inFORM [?],

PinWheels [?], the BMW Kinetic sculpture [?]), often resultsin large, heavy, and immobile setups that are not compatiblewith these demands. In addition, end-user expectations of cur-rent interactive systems will demand high-resolution output—Humans’ haptic and visual perception still far exceeds thatpossible in shape-changing interfaces [?]. However, increas-ing the shape resolution [?] tends to considerably complicatethe technical setup.

We believe this challenge can be successfully addressed bybuilding on recent results from soft- and modular-robotics, andsmart materials. First activities should focus on soft robotics,as this field is the closest to supporting working prototypes.Soft robotics are often based on pneumatic approaches [?, ?, ?,?], which allow for varied geometries, slim shapes, and evenstretchable form factors, while offering strong output forces.Recent advances in computational design and digital fabrica-tion show great potential in simplifying the implementation ofcustom form factors and customized actuation patterns. Thenext important steps consist of increasing the resolution andthe actuation speed.

Smart materials can further help to miniaturize form factorsto slim films or sheets. Coelho [?] demonstrated the viabilityof shape memory alloys for use in HCI prototypes, but theyhave high energy consumption, slow response time, and heatspread. These limitations can be overcome by identifyingdesigns that carefully fine-tune mechanical behaviour [?]. Di-electric elastomers [?] can support more form factors, whileoffering fast response times. It will be important to identifyways to increase their actuation strengths, for instance by com-bining multiple layers of elastomers. Lastly, fully printedactuators [?] promise to speed up the fabrication of slim actu-ators, but at the cost of limited actuation speed and strength.

The modular robotics approach, where interfaces are formedby assemblies of units, provides a promising approach forscalability in shape-change. In current systems, the individual

units are still at the cm-scale [?, ?, ?, ?]: the key challengefor shape-changing interfaces is to downscale the individualunits, such that future interfaces will allow users to interactwith “stuff rather than things” [?] and hence support a muchhigher resolution and expressiveness of shape-change.

A realistic goal is to develop actuation systems that are 1–2 orders of magnitude slimmer and more lightweight thancurrent systems, while they respond in real time and produceforces comparable to the electromechanical actuators that areused in most current shape-changing interfaces. The outputresolution should be in the mm-scale rather than in the cm-scale. Given current progress, modular actuation systemsshould scale to 100’s or even 1000’s of units, rather than the10’s demonstrated today.

Integration of Additional I/O ModalitiesToday’s shape-changing interfaces typically focus on shapeinput and output, with less emphasis on other modalities—forinstance, visual output is often realised by projection map-ping or low resolution LEDs. While this is acceptable forearly research prototypes, successful real-world interfaceswill demand additional input and output modalities. Thisincludes high-resolution touch sensing, in-place visual out-put, adjustable material properties and accurate sensing of thedevice’s physical shape. Without this core integration, shape-changing interfaces will fail to realise their full potential.

We see multiple important directions to address this challenge.First, conventional off-the-shelf sensors and active displays(e.g. LCD, OLED) should be integrated in shape-changinginterfaces. Second, advances in flexible sensors and displaysopen up a path toward integrating conformal touch sensorsand displays that are directly integrated with the actuator. Forinstance, elastic conductors can be printed to capture touchinput on deforming surfaces [?], while electroluminescentmaterials [?] or printed OLEDs actively emit light.

In addition to high-resolution visual output, we believe thatfuture shape-changing interfaces should render haptic materialproperties, in addition to shape alone. Recent research hasprovided first evidence of the feasibility of rendering variousbasic material properties even with a pin-type display [?].

A suitable goal is to realize shape-changing interfaces that cap-ture touch input and provide high-resolution visual output ina quality comparable to today’s handheld computing devices,while having the capability to dynamically modify their hapticmaterial properties such that they feel realistic to the user.

Non-functional RequirementsEnergy consumption is a significant challenge in actuated sys-tems, particularly in mobile or wearable solutions which mustbe self-contained. Today’s shape-changing interfaces are usu-ally tethered; if mobile, they typically use large batteries orhave short battery life-spans that would be prohibitive in re-alistic use settings. While there is little room for improvingthe energy consumption of basic actuation technologies, wesee significant potential in systems design that reduces powerconsumption by offering (bi-)stable states [?, ?]. In these

cases, the system only requires energy for transitioning be-tween states, but does not consume any power when residingin a state. Further, there is potential for applying environmen-tal energy harvesting to shape-changing interfaces—ambientlight, thermal radiation, and user movement seem promisingsources. Notwithstanding these techniques, devices that canself-charge [?] will ensure the burden of charging is removedfrom end-users. A suitable goal for this challenge is to realizemobile and wearable shape-changing interfaces that can beused for at least a full day without recharging.

Safety and Compliance are also significant challenges to ad-dress before placing actuated objects in users’ hands. Roboticsresearchers are now moving beyond traditional robotic arms,wheeled platforms and humanoids to explore a wide varietyof different form factors, especially influenced by the goal ofbringing robots out of the industrial setting into closer proxim-ity to humans. For example, soft robots made of flexible andelastic silicone, with pneumatic actuation can deform in sizewhile remaining soft and compliant, so can be used close to,or even on people’s bodies.

USER BEHAVIOUR CHALLENGESThis section describes research challenges concerning users’understanding of, interaction with, and behaviour aroundshape-changing interfaces. Addressing these challenges willproduce empirical and theoretical insight about behaviour.

Understanding the User Experience of Shape-changeWe should seek to understand the user experience (UX) ofshape-change. Evaluating the user experience and usefulnessof new technology is challenging in general, but shape-changeposes a specific set of challenges. First, shape-change com-bines modalities (e.g. visual, haptic)—isolating their relativeeffects is hard. Second, the experience of using shape-changespans diverse goals such as communicating affect, reducingtask completion time, and task fit. Third, current systems areoften not robust enough for in-depth evaluations. So despitecalls for more studies, this challenge remains open.

By understanding the user experience of shape-changing inter-faces we will be able to characterise their value, understandin which domains and for what tasks they are beneficial, andsupport their design and construction. We see two parts to thischallenge: suitable evaluation and the isolation of outcomes.

In-situ, Comparative, and Replicated UX EvaluationsCurrent research in shape-changing interfaces is limited by thecomplexity of hardware (as described earlier) which results inmany devices being fragile, hard to replicate, and not suitablyrobust for long term use. One consequence is that in-situevaluations of shape-change with real tasks are rare (but doexist [?, ?, ?]). Such evaluations will help to assess suitablecontexts of use, the fit between tasks and interfaces, and issuesaround the cultural appropriation of shape-change.

Further, there is a need for more comparative and controlledstudies that allow us to better understand the benefits anddrawbacks of shape-changing interfaces compared to exist-ing modailites of interaction, including direct manipulation

in GUIs, un-actuated TUIs, gestural interaction in spatial vir-tual reality, and single point haptic feedback. Research in theevaluation of TUIs has proven insightful (e.g. Tuddenham etal. [?] and Zuckerman and Gal-Oz [?]). Comparing physi-cal shape-change to visual-only changes is a first importantstep, but we need to move beyond this to justify the cost andcomplexity of shape-changing interfaces.

Finally, replications of work on shape-change are rarely con-ducted. Most dissemination venues favour novelty and theunique one-off prototypes; the cost of (re-)building previouslyreported systems prevent independent replication studies (al-though advances in toolkits would significantly help). Thevalue of replications are well-argued within and outside ofHCI [?, ?]. An important first step is to see if findings ob-tained with non-functional prototypes (e.g. videos [?]) can bereplicated with physical prototypes.

Isolating Factors and Outcomes of Shape-changeMost evaluations of shape-change concern an entire system.To deepen evaluation we must isolate the factors and outcomesof different aspects of these systems. Key factors to considerare the experience of different transformations among shapes,the impact of using shape-change as it unfolds, and interactingnot only above or on shape-changing interfaces, but also withinthem (through deformation and pressure). The impact of scaleand resolution on experience is also a key question: a first stepwould be just-noticeable-difference studies of motion resolu-tion (compared to Shimojo et al.’s static investigation [?]) andunderstanding the influence of scale on experience.

Further, there is a need to explore detailed outcomes of usingshape-change: learnability and naturalness of interaction areclaimed as benefits, but to our knowledge they are not stud-ied so far. Many papers have linked shape-change and theexperience of particular feelings [?, ?, ?]. However, severalof these papers remain unclear about their basic claims ofaffect [?]. For instance, are claims being made about coreaffect (a non-reflective, always available feeling) or emotion(involving cognitive appraisal and attribution)?

Similarly, our field would benefit from concepts identified inHuman Robot Interaction (HRI). HRI researchers often in-vestigate human perception of anthropomorphism, animacy,likeability, perceived intelligence, and perceived safety inrobots [?] or improved techniques for robots to convey in-tent through non-verbal motion. An important next step willconsist of investigating to what extent these predominantlyagent-based findings transfer to shape-changing interfaces.

Shape-Change Theory BuildingBehavioural sciences often construct theories in an attempt tointegrate and explain empirical results, to identify gaps in ourunderstanding, and to make predictions. Shape-change’s grandchallenge is to integrate and explain our empirical results andmake predictions about the use of future shape-changing in-terfaces. We are unaware of theorizing in this sense aboutshape-change (as opposed to research in tangible user inter-faces, where work has begun [?, ?]). The benefits of suchwork would be to generalize and predict the user experienceand usefulness of specific examples of shape-change.

Current use and development of theory in shape-change re-search is rare. While examples include biological theoriesof motion [?, ?], most theory use in shape-change has con-cerned affordance [?, ?, ?, ?]. The notion of affordance hashelped drive technical development, analyse empirical data,and generate experimental prototypes. It has not yet, to ourknowledge, been used to formulate an integrated account ofempirical data about shape-change, nor to make predictions.Nor are we aware of the application of theories about graphicalperception, biological growth, or movement control—to takea few examples—in shape-change research.

Current research needs integrated accounts of what shape-change is, why it works, what experiences it can engender, andwhen it is useful. We imagine as a first step that this could useDubin’s [?] view of theory-building. A key part of his workis propositions—statements linking concepts through laws ofinteractions and boundary conditions. Propositions containactual, committing statements describing how concepts in-fluence each other. For shape-change research, we need todevelop theoretical statements that articulate propositions onhow shape-change affects interaction. This would help us todevelop scientific claims about shape-change as well as predicthow users might react to new shape-changing interfaces.

DESIGN CHALLENGESThis section describes the design research challenges. Thebiggest differences to the traditional field of design is theresponsiveness of shape-changing interfaces, the design andform of the artefacts, and their dynamic qualities. Shape-changing interfaces are unique in the integration of movementand transformation through actuation based on direct userinput. Three key design challenges are identified: designingfor temporality, integrating the artefact and interaction, anddeveloping applications and content.

Designing for TemporalityShape-changing interfaces require temporal design: there isan important challenge in translating behavioural sketches andfunctional descriptions of behaviour into actual designs. Whilesketches and prototypes of static forms provide representationsthat are visually and tangibly comparable by multiple peoplesimultaneously, dynamic form has temporal aspects that aredifficult to compare in parallel. This makes traditional methodssuch as design critique [?] more difficult to perform. In partic-ular, the direct interaction a user has with a shape-changinginterface offers a unique experience to that person; there is notyet a language that supports the articulation of properties, ex-periences, or sensations, such as colour or material propertiesin traditional product design.

The many disciplines of design have started the first stepsin addressing this challenge. Dance and movement [?] hasinspired work into active form [?], dynamic form [?], and tem-poral form [?]. Designers use ‘acting out’ and choreographyto sketch dynamic form languages and implement them intoexperiential prototypes [?, ?], building on the body of knowl-edge available in animation (e.g. in the computational designof mechanical characters [?]). Inspiration may also be drawnfrom music, where notation systems use symbols to indicate

music both concretely (notes) and more ambiguously throughexpressions such as ‘crescendo poco a poco’ [?]. Other di-rections can be found in modelling platforms with multipledegrees of freedom that replicate different movement patternsin parallel, although interaction will again lead to diverse ex-periences [?]. A key goal is the development of techniquesand methodologies that allow the design, construction, anddirect comparison of temporal forms.

Integrating Artefact and InteractionWith shape-changing interfaces, designers will be challengedto develop devices that are satisfying both in the form anddynamics of interaction between the artefact and the peopleinteracting with it [?]. The usability and aesthetics will beinextricably linked, with the aim of designing shape-changingmaterials that engage the body as well as our mind [?].

This interplay of properties increases the complexity ofthe design exponentially, and consequently the training re-quired by designers. Their expanding duties that werepreviously separated—the notion of a “master builder” [?]in architecture—reflects the increasing responsibility andpurview of the designer. As part of this, designers need tounderstand theory, heuristics, and dynamic affordances whereappearance and actions serve as carriers of meaning [?]. Ex-panding their competences is therefore essential for the futuredevelopment of this technology. A first step is to develop moretools that support the design process (as per the technologychallenges) and design methods that couple action and reac-tion [?]. A suitable measure of success is the development ofdesign tools and methods that intrinsically integrate artefactand interaction in the design process; and designers whosetraining means they can successfully handle this complexity.

Application and Content DesignThere is a significant challenge in the ‘when’, ‘what’, and‘how’ of the design and implementation of applications andcontent. We can break this challenge into four parts: (1) Whenshould we apply shape-change? (2) What shape-changesshould we apply? (3) What applications should we build?(4) How do we design the content for those applications?

When to apply shape-change: While shape-changing inter-faces provide many advantages, they are clearly not suitablefor all interaction scenarios. Possibly due to the lack of the-oretical understanding, the community has yet to explore orunderstand under which circumstances physically-dynamic in-terfaces are appropriate and when traditional interfaces shouldbe used. However, first broad steps are underway, with re-searchers presenting exemplars at large public events [?], shar-ing prototypes with the broader public [?], and developingspeculative scenarios and designs [?, ?]—all of which triggerdiscussions on acceptance and desirability. A obtainable goalis to develop frameworks and design principles that describewhen shape-change should be applied and when traditionalinterfaces are more appropriate.

What shape-changes to apply: One of the key benefits ofshape-changing interfaces is their ability to transform intomultiple shapes or forms, potentially for quite diverse tasksor situations. Consequently, the designer can choose shapes

that are adapted to the human instead of technical, manufac-turing, or transportation limitations. However, there is a lackof understanding as to what types of change should occur inmore abstract or generic circumstances, where shape may notnaturally represent content. Without clear guidelines of use,there is the risk that end-users will become confused by seman-tics across different shape-change applications. Using designguidelines, researchers should aim to reach a consensus onshape-change semantics that can apply across many devices.

Applications: The literature contains a vast range of specificshape-changing interface prototypes (see Figure 2) with sev-eral initial attempts made to identify categories of applica-tions [?, ?]. This diverse set of applications allows researchersto explore and test the design space; but discovering the ‘killer-apps’ and domains for shape-changing interfaces is what willdictate its ultimate success—without this, the technology maynever make it into consumer’s hands. Researchers, in con-junction with industrialists, should identify and explore keyapplication domains that provide clear benefits and routes toend-user engagement. Without more targeted application do-mains (such as Data Physicalization [?]), the field runs the riskof becoming too diverse and being unable to cohesively tacklemany of the grand challenges described here.

Content design: All shape-changing interfaces will requirecareful consideration of the end-user content. This is partic-ularly relevant for shape-changing interfaces that exploit thedominant visual sense through a shape-changing display. Suchcontent must address the visual and physical configurationof the display and user input. First steps have already be-gun to address non-traditional display formats (e.g. spherical,volumetric displays, and 3D UI’s [?] and non-rectangulardisplays [?, ?]). Research must then address content designfor displays that change their shape. Researchers should aimto develop toolkits that facilitate content design on any size,shape, or form-factor interface.

POLICY, ETHICS, AND SUSTAINABILITY CHALLENGESThe regulatory, ethical and sustainability implications of thedevelopment of shape-changing interfaces will impact thelong-term adoption of this technology, therefore also its long-term development.

Policy and EthicsWe see a significant challenge for policy-makers to createlegislation that ensures the safe and ethical operation of shape-changing interfaces. It is particularly challenging to ensurethat these interfaces are suitably regulated without curtailingtheir innovation potential, especially with the large diversityof devices that may become available.

Shape-changing interfaces pose a number of safety and ethicalrisks that without suitable regulation may result in rejection byend-users or widespread bans by government. The key issuesfor shape-changing interfaces are: (1) Safety: they must bebuilt so as not to harm users or bystanders or cause damageto physical property; (2) Security, privacy, and control: theymust be subject to the same security and privacy expectationsof other digital systems (both digitally and physically). End-users must ultimately always have control over the forms their

interface may take; (3) Content and appropriate use: theymust prevent inappropriate content being shown to particularuser groups (e.g. children). Legislation should also provideguidance on inappropriate use cases based on societal, cultural,and criminal values; (4) Ownership and responsibility: indi-vidual users are likely to own the hardware for these devices(just as we own mobile phones today) with remote serviceproviders delivering content. The law must clearly distinguishthe responsibility of different parties for the output and itsconsequences; (5) Implications of non-permanency: today’sshape-changing objects do so in a predictable and repeatablefashion. The radical transformations proposed for the futuremay impact on areas of society that require a known ‘history’of form; e.g. the untraceable transformation of a weapon usedin an attack could hinder crime-solving teams.

Governments often struggle to legislate on new technologicaladvances quickly enough. However, our field can learn lessonsfrom other step-change technologies (e.g. drones, autonomouscars) on how to rapidly and suitable legislate to avoid stiflinginnovation. To the best of our knowledge, no such explorationsexist for shape-changing interfaces.

Sustainability and the EnvironmentShape-changing interfaces are a ‘double-edged sword’ for sus-tainability. The combination of physical actuation components,display surfaces, and electronics, pose an increased demand onnatural resources, and increased challenges for recycling, andlater re-use beyond those already present in consumer-levelelectronics [?]. Addressing these resource challenges are es-sential for the long-term viability of shape-changing interfacesand the health of our environment.

However, a shape-changing interface’s morphing abilityshould ultimately reduce the need for multiple instances of sim-ilar devices and therefore reduce long-term resource require-ments. For example, instead of consumers needing an array ofdifferent form-factor computing devices (laptop, tablet, smart-phone, smartwatch), a single shape-changing device couldtransform into the desired size, shape, and weight on demand.

To reach such a state, researchers should place emphasis on de-veloping ‘life-time’ shape-changing interfaces: those that havethe life-span of decades (similar to a house) rather than the fewyears typical in today’s consumer electronics [?][?, pg. 258].To achieve this, researchers must strive for modularity [?] toallow the repair, self-repair [?], or replacement of broken com-ponents, and support upgrades as higher ‘shape resolution’ [?]components are developed. Standardization that can survivethe fundamental redesign of these interfaces will be essential.Finally, by supporting interoperability between software andhardware, developers will further reduce the burden on theresources necessary to produce these devices.

CONCLUDING REMARKSThe field of shape-changing interfaces has greatly expandedduring the last few years, showcasing novel technologies andembracing new disciplines. At the same time, the field hasmatured considerably, and is beginning to converge on keyquestions surrounding technology, design, theory-building,and societal implications. In this paper we have identified and

discussed what we consider the 12 grand challenges currentlyfacing researchers. Although there are many examples ofshape-changing interfaces that provide delightful experiencesand offer better ergonomics, the challenges identified need beaddressed to help deliver the full benefits of shape-change.

It is important that we also recognize and deal with limita-tions of shape-changing interfaces and temper this optimism.Shneiderman [?] discussed the ten plagues of the informationage, many of which apply directly to shape-changing research.In research on proxemics, researchers have discussed sinister“dark patterns” [?], that is, applications of proxemics whereusers are manipulated and deceived. We appreciate the trans-parency and open dialogue encouraged by such work, whichhighlights how things can go wrong. We envision similar neg-ative and even dangerous appropriations of shape-changinginterfaces will also occur; understanding and minimising thatimpact crosses many of the presented challenges.

We are excited about the potential of shape-changing inter-faces and how they will transform interaction with computingsystems. We see the identification of these grand challengesas a positive step in focussing research efforts towards thiscommon goal. These challenges will, no doubt, evolve as tech-nology matures, society changes, and our understanding ofusers increases. We hope you will join us in developing theseinterfaces to shape the future of human-computer interaction.

ACKNOWLEDGEMENTSThis work originated from the discussions held at DagstuhlSeminar #17082 on Shape-Changing Interfaces; we’d liketo acknowledge and thank all attendees at this workshopfor their input. The authors in this paper are partially sup-ported by grants from: MORPHED (EPSRC #EP/M016528/1),GHOST (EC FET-open #309191), Automorph (EPSRC#EP/P004342/1), The Leverhulme Trust, InteractiveSkin (ERCStarting Grant #714797), and Body-UI (ERC ConsolidatorGrant #648785).