Pencils and Pixels:An Exploration of Physical and Digital Sketch Techniques for Tangible Interaction Design

Solomon BiskerMethodologies of Visualization (Design Drawing I) Fall 2009, Carnegie Mellon University

All of the works in this project were created using a hybrid of physical pencil, pen and marker sketching techniques learned in this class, and digi-tal design tool techniques learned during my career as an Interaction Designer. The objective of my project was to explore how blending physical and digital sketching techniques could boost my repertoire of sketching skills as I pursue my Masters degree in Tangible Interaction Design. My hypothesis in starting this project was that sketching tangible interactions poses unique challenges that neither hand-sketching nor digital mock-ups alone are currently suited to tackle. This felt like it would be particularly true in interactions that have environmental context (“mobile” or built environment scenarios) and interactions that involve posing and positioning of the the human body (touch screens, hand-held objects, and the like.) To that end, I came up with simple scenarios of tangible interactions that I felt might benefit strongly from these hybrid techniques. Then, for each scenario, I attempted to use both hand-sketching and digital manipulation of sketches to play to the strengths and weaknesses of both medi-ums in developing final concept sketches for each scenario.

Before we dive into these interaction scenarios, there are few general tips worth noting that are useful to anyone creating or digitizing hand-drawn sketches for later digital manipulation. By reading these general tips upfront, the reader will hopefully be able to follow my individual techniques at home without making the same mistakes I did.

If possible, use a high-quality color scanner for your work. The scanned image on the left was from a professional color scanner. The colors in the image are much richer (and truer to the original sketch) than those captured by the “consumer” color scanner in the image on the right.

If you intend to scan your drawings in black and white to eliminate the paper color, make sure no marker bleeds through from the other side of the page. In this case, a light marker was translated to pitch black, rendering the lines under that marker unusable.

Drawings of actual interfaces will not stretch or skew cleanly to match the ac-tual dimensions of your screen. This means that to get an interface drawing to drop into a photo clearly, you will want to measure out the proper sspect ratio of the device and translate that directly to the paper before sketching (perhaps practicing to try to achieve dimensions close to that ratio from memory, if you are looking to digitize quick skteches from brainstorming sessions.)

Digital designs often play to the strengths of digital design tools. One such strength is the ability to make pixel-perfect lines and shapes, which can be used to emphasize simplicity of form. This is seen in public sig-nage, where “bathroom man” figures are recognizable and perfect in form.And yet, the bathroom man invari-ably feels almost *too* perfect. This reflects the inability of digital tools to let one sketch the subtilty of a person in a pose. Here, I’ve used sketching to capture a “pose” of a “bathroom man” doing a real action (in this case, one of my professors giving a speech). Then, digital tools add precision without los-ing the humanity of the pose.

I began by capturing the signage form off of a bathroom sign, and then applying that form to various people I encountered.

Here, I was working through how shoulder height and angle influences attitude.

I experimented with sitting poses of students, but decided that having to then stylize the student’s chairs would detract from their poses.

As my professors stood before my classes, I realized that the signage form left much room for capturing a pose’s spirit in arm movement and head position.

I attempted to recreate the purity of the “bathroom man” by overlaying basic circles, lines and other shapes on top of my raw sketch. In this way, I was able to recapture my original angles, both of the head tilt and of the arm gesticula-tion. Once I brough the shapes into the right poses to capture the sketch, the shapes had to be resized in order to better capture the idealized propor-tions of a signage figure. In this sense, while my sketch might have captured the *pose* more faithfully, I then needed technology to help me tighten those overly thick thighs, pull up that torso and otherwise make my person more “idealized.” The end result is “the ideal man,” albeit in an oddly human pose. The biggest failing I saw was one of perspective - most sign people are drawn head-on, and it was difficult for me to capture the pose while maintaining that head-on feel. All digital work done in Adobe Illustrator.

I recreated the angles and positioning of the body parts in Illustrator as a series of rectangles with rounded ends, eventually adding a skewed rectangle as a torso that would capture shoulder position.

Once the shape began to take form, I began manually correcting the proportions on my sketch by adjusting shape sizes. It was crucial working with vectors in Illustra-tor (as opposed to bitmaps in Photoshop) to maintain the smoothness in the shapes as these scaling tweaks were made.

I found myself getting more and more nitpicky with the form as the shape entered a“uncanny valley” of idealized form. I ultimately wound up referencing a real bathroom sign to bring it to an idealized state.

Interaction design increasingly needs to incorpate the context of physical space into the design of digital products. It is often not the interface at all that has to be envi-sioned, but rather the motions the body goes through - and how the digital aspects of product relates to that physical motion - that needs to be sketched and storyboarded. A colleague’s research on the impact of pico-projectors on the virtual pet movement inspired me to sketch out a vision for walking a virtual dog. In this sketch, my physical hand drawings had to capture the tactile feeling of “leading” a dog, while the digital alterations to the sketch needed to approximate the virtuality and hyper-realism of a “projected pet” in a realistic manner.

I started by trying to capture the shape of the hand,”clenching” the hand to emulate holding a leash and working on thumb grip.

I then took my improved hand drawings and refined then in the context of projection, explor-ing how a person would “project” their pet.

Finally, I explored how to frame the context itself - realizing that people would want to project the pet at a certain size, and in proper relation to their walking.

I decided to focus on projecting a “real” pet, using Flickr to find royalty-free pictures of dogs in approximate poses appropriate for the story. I then used Adobe Photoshop to digitally edit the dog out of its original photo and to scale it appropriately to fit into the user’s projection space both size-wise and perspective-wise. Finally, tweaks was added to emulate the pico-projector’s actual operation.

After finding a dog in an appropriate pose on Flickr, the Photoshop “Magic Wand” tool was used to remove much of the grass from the scene (with hand-erasing removing the rest.)

I discovered that the picture of the dog was captured at too low of a resolution to fit nicely with the resolution of the drawing. Since the drawing was a stylized image, I lowered the resolution of the drawing so the dog would appear “realistic” in the context of the scene.

A white blur was added to the scene where the projector would eminate light out of the user’s grip.

The final edited dog picture was made slightly transparent, to re-flect potential weakness in the projector’s strength.

This envisionment was done with similar motivation to the pi-co-projector sketches, except that I now wanted to see how my existing 2D interaction design tools (OmniGraffle, Visio, etc) could be used in sketches of a real world interaction. Rather than a “user’s eye view”, a straight on third person perspective was used in my sketch such that the 2D inter-face wouldn’t need to be skewed to match the scene. My hope was that the sketch would let me see whether a human body’s proportions would be compatible with a touchscreen interface for finding rideshares I had created for city use.

I started by exploring the human form, trying to faithfully sketch a person facing away from me, with his arm raised as if to touch the touchscreen.

This process took some time - it turns out I had trouble imagining just how the back of ones body looks, in terms of shape, angle and proportion.

Eventually, my sketches converged on an acceptable body form, and I explored the use of marker to add clothing which would emphasize that form.

Again, Adobe Photoshop was the appropriate tool - but this time, it was needed to digitaly modify the sketch itself. Specifically, the portion of the per-son intersecting the screen was digitally removed, with areas of the sketch without body parts being rendered transparent. This allowed me to paste my screenshot to scale directly into the scene.

The sketch showed me that my interface was sufficient for an urban touch screen context, but arguably did not take enough advantage of the screenspace I had been provided. My figure’s arm seems to have a higher precision of point-ing than I would have expected at this scale and distance from the screen. As a result, I suspect the entire interface could have either packed in more informa-tion or been placed into a smaller form factor without sacrificing usability.

The part of the body figured that intersected with the scene was copied and pasted onto a new Photo-shop file in order to render the background transparent.

The 2D screen mockup was then dropped on where the person had previously been. The screen was stretched slightly to fit the drwaing, as the aspect ratio was inconsistent between my UI and the drawing.

Finally, the top of the person was reattached to his body on top of the screen,as if to reconstruct the designed scene by layers.

My last envisionment tried to take an opposite approach to my typical interface sketching technique - drop-ping a hand-drawn interface into a digital photo or other context (as opposed to using Visio to mock it up and dropping that into the photo.) I thought perhaps this might be useful to test out interfaces very, very early in a sketching process, where it is unclear whether or not it’s worth the time to even mock up an idea in Visio. In this example, I tried to create a fake mobile Treemap news interface for placing onto a photo of my holding an iPhone screen.

I tried to improve the boundaries of the display screen by focusing on keeping the lines parallel, but I found that made me pay less attention to giving the screen a proper width-height ratio.

Once the shape began to take a proper form, I began to play with drawing the interface itself in a “sketchy” manner to focus on issues of layout and size. I did this by blurring out system elements and smaller interface text.

Most of my issues in sketching this interface came from sketching the outline of the screen.An incorrect shape would look wrong immediately in the context of a photographed photo.

I created this rendering in Adobe Photoshop of my interface projected onto a photo to see how the interface might feel in the palm of my hand. This seems to quickly let me see how interface elements might relate to, say, the shape of my hand and my thumb. It seemed clear from this composite sketch that buttons on a Treemap interface would be clickable on this scale, and this might have encour-aged me to explore the concept for a design further in Visio. However, I realized quickly that for hand-drawn interfaces, it becomes that much more important to draw the interface aspect ratio correctly. Visio-created interfaces - especially those done in vector - might stretch to fit your drawings, but your drawn interfaces will not stretch as cleanly to match your photographs.

Above: An attempt at shrinking the photo to match the drawn screen’s aspect ratio. Below: An attempt opposite of that above - trying to stretch the interface drawing itself to match the photograph. In this case, stretching the interface works better than shrinking the picture, but neither looks ideal.

Photoshop’s Magic Lasso tool was used to select and copy the interface sketch so it could be pasted into the photograph. Similar to the dog photo earlier, our photo is too high resolution for our sketch to fit the scene - so I scaled down the photo resolution to match the drawing.

Then, the image was cleaned up with the Eraser tool before Free Transform was used to rotate the sketch to match the angle of the screen in the user’s palm. An additional Free Transform on the photo itself was then used to scale it to the size of the drawing.

The first attempt (the upper of the two final results shown at right) looked too squished, so I tried again, this time using the original photo and applying a second Free Transform to the drawing to match the device screen’s actual aspect ratio.