iFlixis246 Multimedia Metadata Final ProjectMilestone 5: Complete Description of our iFlix DesignCecilia Kim, Nick Reid, Rebecca Shapley

Table of Contents

IFLIX 1

Table of Contents 2

Introduction to iFlix 3What is iFlix? 3Relationship to the Apple iLife application suite 3The Keyframe Summary: content metadata 3Sorting and Browsing: object metadata 4A static surrogate to start a revolution 4

The iFlix audience 5

Our Motivations for iFlix 5

Relationship of iFlix to Previous & Current Work on Multimedia Information 6

About This Project 7

User Scenario 8Organize: The Catalog View 8Organize: Import metadata for a new flick 8Organize: Rendevous with Local Macs 8Play: Viewing a Flick 8Organize: Watching for New Flicks 9Summary: Editing a Flick Summary 9Organize: Share Flix Summaries and FlixLists 9

Screen Descriptions 10Organize – Import, Organize and Share Flix listings 10Summary – View and Edit a Flick’s Keyframe Summary 14Play – Watch a Flick 18

System Diagrams 19Information Flows including iFlix 19

Interviews with Users 21Methods 21Alpha Screenshots 22Conclusions 23

References 24User Preference for Static Summaries 24Keyframe Selection for Video Summaries 24Navigating video using keyframes 25Navigating among keyframes 25Music-recognition services 25On the audiences 25

Introduction to iFlixWhat is iFlix?iFlix is designed to be the next member of the Apple iLife suite of applications. iFlix is amedia player and cataloging application, but perhaps most interestingly, is the firstapplication to provide a way for consumers to get involved in media content annotationwith almost no learning curve. iFlix’s Keyframe Summary is a simple, flexible conceptfor annotating media content that is scalable to the user’s interest in annotation whilebeing easy to create, organize, and share.iFlix is a tool for organizing flicks, from Hollywood movies to homemade flash moviesshared over the Internet. Flicks are time based multimedia clip products of any commonfile type, including flash files (.swf), quicktime files (.mov), real player streaming pointerfiles (.ram), or "bookmark" pointer files (.url), located either on the user’s local harddrive or as a streaming movie on a remote server.

Relationship to the Apple iLife application suiteThe iLife applications are designed to bring everyday computer users into the world ofdigital media consumption and production. The users of these applications are motivatedby their desire to organize their personal collections of media and to share the resultswithin their social circles. Apple has intentionally set the learning curve quite low inthese applications, at the expense of features that professional media producers mightconsider unacceptable. Additionally, there is little room in these consumer –focusedapplications for creating and editing highly-structured, computer-parsable MPEG-7metadata about media content. However, we feel that iFlix will both serve a consumer’sgrowing need to organize their video media collections and introduce the concept ofannotating media content into the general computer using community whilesimultaneously providing the tool for doing so easily.

Audio Images VideoOrganize iTunes iPhoto iFlixProduce GarageBand (camera) iMovie

Throughout the interface, our design choices leverage users’ experience with many of theinteractivity concepts developed in the iLife suite applications : iTunes, iPhoto, iMovie,and even Safari. Novel to iFlix is the simple but powerful concept of the KeyframeSummary, which is designed to be exactly what the user wants it to be.

The Keyframe Summary: content metadataThere’s a reason why the iFlix box - an application to organize video - is the last to befilled in. To organize a collection will involve displaying lists of the colllection’s content.While commercial music is described with a standard metadata of title, artist and album,and images can be represented with a thumbnail, there isn’t an obvious answer to whatbest represents a video. The variety of length and variation in flicks is astounding, and

many intelligent people have done a lot of brilliant work to figure out how to summarizea video in a catalog listing.A Keyframe Summary consists of a selection of still images and associated text in aspecific order. iFlix automatically generates three-image shot-optimized KeyframeSummaries from the video (a.k.a. “flick”), accompanying the images with text createdby OCR of text in the flick and by speech recognition. By adding images and editing thetext, users can edit these Keyframe Summaries to their heart’s delight, collecting multipleversions of summaries for a given flick. Keyframe Summaries can be exchanged by e-mail, distributed via RSS feed, shared on a webpage, and downloaded from authoritativesources.

Sorting and Browsing: object metadataiFlix aims to identify the key simple object metadata users want to sort and search flickobjects by. This object metadata is either automatically generated by content analysis,contained in id3 tags with the file, entered by users locally or retrieved from a sharedGracenote-style online database. Examples of object metadata include filesize, date, title,author, distributor, names of actors, characters, and other participants in the flick’sproduction, “my rating,” and play count.While designing, we quickly discovered that there is no definite or default way ofcategorizing flicks. Unlike music there is generally not one definite named creation agent,or the creator’s name is unknown. Outside the world of Hollywood content, even the titleof a flick maybe ambiguous. Unlike personal pictures, the use of time to catalog flicksisn’t powerful either because the knowledge doesn’t exist or the creation date is onlymildly relevant to the media’s meaning.In our design, we’ve privileged Names and Roles as a way of browsing an iFlixcollection. Names include both those of real people and those of characters that appear inthe flick. Because this is such a huge list with a variety of implications, it is refined with aRole keyword. Selecting the Name Spike Lee will pull up the Roles Actor and Director,and all the flicks he has been involved in. Selecting a particular Role will narrow the listof flicks to browse.We feel this aspect of the design is still open to more refinement. The iLife applicationsare good at narrowing object metadata down to the specific essentials that users want andwill create or maintain.

A static surrogate to start a revolutionWe made a specific design choice that iFlix would create static summaries of the time-based cataloged visual media, feeling from our own experience and user interviews thatstatic surrogates can be reviewed at the user’s own pace. Research suggests that althoughpeople can interpret various time-based summaries, if they are not generated masterfully(like movie trailers) people prefer static summaries. The Keyframe Summary is scalableto the amount of effort the user wishes to invest (from nil to highly-customized) and canbe put to various uses, including cataloging, navigation, summarizing a video,bookmarking interesting moments, and more. We even specifically designed aninterface that doesn’t require the keyframes to stay in the same order as the original flick,allowing maximal storytelling flexibility such as that often leveraged by Hollywood

movie trailers! As users find value and satisfaction in interacting with contentannotations, we trust that future applications will be able to leverage a combination ofhuman and automatic information to eventually bring the computer-parsable mediarevolution to the average computer user.

The iFlix audienceiFlix is designed to be usable by people who enjoy flicks. There is a very broad range ofmedia consumers, but our ideal users for building a promising future for computablemedia are those who find sharing flick information to be satisfying expressions of socialconnectivity.The application supports a range of interaction with the user's collection of flicks, fromsimply cataloging, grouping, browsing, and searching among a collection, to addingvalue to the metadata about a particular flick, to sharing lists of flicks, metadata aboutflicks, and the location of flicks with friends. We expect iFlix to be used by:"The Organizer" – people who are active browsers of media content on the Internet orare in media production and have a need to organize the various video clips andstreaming file pointers on their computer“The Fan” – the small but highly-dedicated, highly-organized group of fans of variousmedia series, such as Star Wars or Star Trek fans, who currently keep an amazinglyextensive mental catalog of specific interesting moments in various episodes, and wouldbe able to externalize and organize this catalog."The Socializer" – perhaps college undergraduates (ages 18 to 25) - people for whomcollecting, commenting on and sharing video experiences is a social activity.Additional audiences for iFlix include sports fans sharing or selling annotations of keymoments in sports events footage; video ethnography production teams collectivelyannotating reams of footage; and interest groups seeking feedback from their membernetworks on proposed commercials or documentaries.

Our Motivations for iFlixMotivation behind iFlix is simply addressing the real problems that we have been facingwith iLife applications. We are all Mac users that have been impressed with powerful yetsimple designs of the various iLife applications, and we believe they are excellent entry-level applications. There is an “Organizer” and a “Socializer” on our design team, sowe’re designing for our own challenges with finding and keeping track of the variousversions of media that we have produced and consumed. Management and sharing ofmetadata hasn’t even been a possibility, until iFlix.As iTunes has been the solution for music files, we’d like iFlix to be the solution forvideo files. iFlix' features are inspired by 1) the visual and sharing features of iPhoto, 2)the cataloging, metadata, and sharing features of iTunes, and 3) the function of mediaplayer and the concept of identifying pointers and keyframes within a flick for whichiMovie pioneered a simple interface.How should users organize their media? Their way. For any time-based media catalogingsolution, some sort of summary representation of the media is required for browsing and

evaluating the contents of lists, or the results of searches. Much successful attention hasfocused on automated techniques for generating these representations for use in retrievingvideo from large collections, but many challenges remain. One of those challenges isbringing media interactivity to regular people, where the organization is informed bytheir own interests instead of an algorithm. We believe users are interested in non-time-based summaries that can be browsed and don’t need to be played. We are interested inproviding an interface that facilitates human users’ contributions to optimize thesummary representation of a flick towards their particular communication goal. Not onlydoes iFlix’ combination of automatic and human-moderated summarization serve a broaduser audience, it also explores new territory in terms of encoding semantics into videosummaries.

Relationship of iFlix to Previous & Current Work on MultimediaInformationWe take our inspiration from the envelope-pushing work being done within theInformedia project, the Open-Video Digital Library project, and FXPal’s Manga andMbase projects. These projects have shown us an impressive range of what is possiblefor interactivity with static summaries of multimedia content. iFlix wouldn’t exist withoutour ability to say with confidence, for example, that the task of generating an keyframesummary with text annotations automatically from video content in a way that is usefulfor a browsing human is a solved problem. However, these projects all focus onsearching within video libraries fed by automatically captured content, best described asthe large, impersonal, automated information retrieval system. In this informationretrieval context, automatic summarizing systems must seek the one summary torepresent a video.With iFlix, we’re interested in the personal and social digital video library. Personaldigital libraries means people already know they care about the content. Shared personaldigital library index means someone you know cares about the content. This applicationis NOT optimized for helping filter through long meeting videos for important moments,for example….although if someone has, they might share their index via iFlix. iFlix isabout being able to make a summary, and another, and another…each one appropriate fora given expression about the flick’s content.The Open-Video Digital Library project began to bring human mediation into theirsummary creation process, to improve the quality of the summaries. But this still remainsbehind the scenes; a task that the video librarians do while cataloging a video. iFlix aimsto make video content annotation with keyframes a front-of-house operation: easy,ubiquitous, and sharable.Our few user interviews supported what Manga also discovered: people think about statickeyframe summaries as a way to navigate within content, much like the chapters in aDVD. Manga also points to another approach to a static summary – the collage orstained glass summary – where keyframes are arranged in two dimensions and indifferent sizes on a single page. We considered this approach in iFlix during our designbrainstorming, envisioning an additional mode screen with a full-page customizableManga-type collage or poster for a video. It could be a creative outlet for generating a

collage from the still keyframe images, re-sizing them, even merging the boundariesbetween them to accomplish some of the effects of a movie poster.We decided to focus on the linear keyframe summary for a few reasons. First, werecognized that other applications already provide the artistic environment well, and wedidn’t want to re-develop Adobe Photoshop, for example. The keyframe summary issomething new and very flexible for a variety of users, and we have also provided pathsto and from iPhoto so keyframe images can be used in art, and art can be brought in as akeyframe. Second, we were concerned about the readability of complex poster frames ina listing, and yet having the keyframe visuals in the catalog listing is an important part ofthe purpose of iFlix. Finally, our aim for simplicity and low-learning curve in the iFlixdesign requires that the method for annotating multimedia content be revolutionary inpower but “Doh!”-obvious in concept, and similar to things users already know or do.Hence, we didn’t develop the single-page collage idea in this project.Mbase obviously has developed a powerful navigation tool, and we’ve designed asimilar, simpler one appropriate for the iLife target audience.The goal is to provide video users with an easy interface for marking which parts of thevideo they specifically care about, rather than guess what metadata everyone will careabout. iFlix is designed to support media content annotation becoming an easy andubiquitous activity. Within iFlix, the value in annotation comes through being able tonavigate to moments in the video that you find important, and sharing those with othersin your social networks.

About This ProjectThe design of iFlix’s user interface and features is being conducted as a final project for agraduate-level course in multimedia informatics.Our process to date has included:

• defining iFlix,• brainstorming designs,• composing the alpha versions of the Organize and Summary screens,• seeking feedback from our peers and through think-aloud interviews with

potential users,• revising to the beta/current versions of the Organize, Summary and Play screens• collecting more feedback from professional advisors to the multimedia course,

and• ensuring the current designs of screens and features are thoroughly communicated

as a foundation for future work.There is obviously much more work that should be done to further develop the iFlixexternal application design. There are many potential audiences for iFlix’ functionality,but its development will benefit from focusing on serving a particular audience well.Additional needs assessment work should be done to characterize user audiences andselect one. The proposed interaction for adding keyframes and comments to summaries

should be tested with users from the selected audience. Central features of iFlix such asthe Keyframe Summary editing tools and the sharing mechanisms should undergomultiple additional iterations.Thinking more broadly, our professional advisors envisioned that in the age of TiVo, akeyframe summary annotation interface might become an interesting and integral part ofwatching television: see a funny line in the Simpsons, grab the keyframe, send it to afriend to start watching from that point; close-to-realtime annotations of sportsbroadcasts; even a different way to collect Neilsen ratings or viewer impressions of showepisodes. A media-invested site like Yahoo! or MSN might also provide an onlineversion of iFlix –type functionality as a personalized interface for users to organize,consume, and recommend media available on the site.

User ScenarioOrganize: The Catalog ViewGeorgia just moved into a new group house at U. C. Berkeley. She’s met a few nice folkshere already, and Ryan has offered to share a flick with her. She plugs in his USBMinidrive to her iBook, copies over the flick’s file, and opens iFlix. Dragging the file’sicon over the Organize view, she adds Ryan’s flick to the rest of her listings of funnyshort flicks, mostly attachments that she and her friends have sent around by e-mail.

Organize: Import metadata for a new flickGeorgia clicks Yes when prompted about importing the Summary from the FMDB - adatabase of shared, user-entered metadata about flick files including authors, websitesource, or whatever else is available. iFlix automatically generates a three keyframesummary of Ryan’s flick, so Georgia can recognize it visually. She and Ryan watch theflick…another funny one!

Organize: Rendevous with Local MacsThe group house also has a LAN, and she now notices a bunch of additional flicks arelisted in her iFlix catalog. These additional flicks are iFlix metadata entries from her newhousemates’ computers. Symbols next to the listings distinguish the origins andaccessibility of the flicks: pointers to streaming files and websites on the internet,copyable files on a friend’s hard drive, and non-copyable files on a friend’s hard drive.Georgia can review the summary of a flick, and if she likes it, she can click to copy thepointer or copyable file into her iFlix library. She has a thing for penguins. She importsthe files and their metadata listings for a few copyable funny penguin flicks to her harddrive. She sees another penguin flick is posted at Ebaum’s World (a flick contentaggregation site), so she imports the listing. She also sees that a housemate has a copy ofAmelie, (which features a gnome, not a penguin, but she likes it anyway) and she sendsthem an e-mail, proposing a flick-watching night.

Play: Viewing a FlickShe selects the listing for the penguin flick from Ebaum’s World in the catalog, andclicks the “Play” button. iFlix switches to the Play screen. iFlix natively plays anyQuickTime format, or provides screenspace for other media players on Georgia’s

computer to display a file. The penguin flick is streamed using RealPlayer, and iFlixdisplays the RealPlayer player within the larger context of iFlix. Whatever controls areavailable through the media player are also available within iFlix’ Play mode.

Organize: Watching for New FlicksGeorgia had heard of Ebaums’ World, but never gone to visit the website. Having seenthe penguin flick and lots of other listings from there among her housemates, she goes tothe website to check it out. They have a special “Penguin-Lovers” theme, and she can’tresist…she wants to know about every penguin flick they have! She switches back toiFlix, clicks on Import, and adds Ebaums’s world’s Penguin-Lovers webpage to her listof flick sources to track. Now these flicks will appear as listings in her catalog just as herhousemates’ did, and she can import any new flicks to her personal library whenever theyappear.

Summary: Editing a Flick SummaryFond of the new penguin flick she discovered , she wants to share it with her friends. Tohelp them evaluate whether it’s worth downloading the flick to their mobile phone orover a dial-up connection, Georgia wants to send them the Summary with the weblink.iFlix has automatically generated a three keyframe summary of the flick. Looking at theSummary, she sees that one especially funny idea in the flick is missing. She thinks shecan add it in without giving away the punch line…She selects the penguin flick’s listing in the catalog, clicks Summary, and iFlix changesto the Keyframe Summary editing screen. She clicks the timeline in about the right place,watches the flick and stops it near the right frame. She types in an intriguing caption forthe keyframe, and clicks Save. The new keyframe appears on the Summary timeline, andshe reviews it…yeah, it works. She shows it to Ryan, who suggests an additionalkeyframe. They add that one, too.

Organize: Share Flix Summaries and FlixListsWith the flick selected, she clicks E-mail and sends the flick’s URL with the Summaryfile out to her other penguin-loving friends. Ryan is impressed with her exhaustive list ofpenguin flicks, and points out that the group house has a web server. She selects herpenguin FlixList and clicks Homepage, setting up a webpage for herself to share her listof penguin flicks with the whole world…and her new housemates.

Screen DescriptionsOrganize – Import, Organize and Share Flix listingsScreenshot

OverviewThe Organize view, similar to the iTunes interface or iPhoto’s Organize view, providesthe user an overview of the contents of their flix collection. By clicking within thehierarchical structure of the folders in the Sources column (left panel), the user can selectif the catalog reflects all flix available for playing including networked resources, or tolist only items on the user’s local hard drive. Flix can be searched for, browsed for,organized into playlists, and imported from various sources.iFlix provides extensive support for social sharing of video media within a networkedcommunity, while respecting ownership models for digital property. Support featuresinclude rapid search across the community network, and status icons that help the userunderstand if a flick is local or remote, importable or streamed-only, or available forpurchase.

Feature and Functionality DescriptionsImport: Bringing data into iFlix

Importing Flicks

Flicks from the user’s hard drive can be added with drag and drop onto the catalogwindow, or using the Import feature of the File menu. Flicks available at a webpage orvia a streaming media pointer file can be imported in the same fashion, using thebrowser’s drag-n-drop bookmark icon, the resulting .url or .ram file, or by pasting theURL into the menu-triggered Import dialogue box. Importable Flicks from networkresources may also be imported with a click on the status icon (see Status Icons).Generating Listing Entry

After import, iFlix will automatically check the FMDB (FlixMetaDataBase) for existingobject metadata and Keyframe Summary files, or a combination of the program and theuser’s input can generate title, Keyframe Summary, and other object and contentmetadata to populate the newly imported flick’s listing. Network resources may alsoprovide their listing data for a flick. We encourage the future developers of iFlix to useappropriate parts of the MPEG-7 standard, such as the navigation/summary descriptionstandards.Source Management Panel

The upper-left panel , entitled “Source”, has specifically broken with the two-tieredfolder convention from iPhoto and iTunes, creating a third “Library” tier. Now with oneclick, the catalog display can now encompass all local and LAN iFlix listing resources .The ability to focus the catalog display on just the locally maintained iFlix library or onany particular networked resource is maintained.Extending the idea of the iTunes music store, we envision a variety of sources fordiscovering and downloading or purchasing flicks can be added to this SourceManagement Panel. By including them under an additional tier “Stores,” search effortsmay be able to act across vendors and locate the desired media.Network Search and Catalog Sync

A Catalog Sync feature allows subscribing to networked resources’ iFlix listings tofacilitate rapid search for flicks across all available listings in the networkedenvironment. When a Catalog Sync is established with a frequent Rendevous partner, thepartner’s iFlix catalog (listed under “Shared” in the Source Management Panel) iscached locally, and updated in the background. The Catalog Sync feature is accessed andmanaged through menu commands.Understanding and Customizing the CatalogMode Menu

With a flick selected, click Play to enter media player mode and see the video, orSummary to view and edit the Keyframe Summary on the Summary editing screen.Flix Status Icons

Status icons help the user navigate a variety of possible situations. The listing of a flickavailable over the network could be a pointer file in a friend’s catalog, an actual file ontheir hard drive. The flick can be a file you can have a copy of, or a file you are onlyallowed to view while they are online.

A flick’s non-local status is indicated with the “streaming” icon from iTunes. Clicking onit will play it as a stream from the LAN or Internet networksource. If the flick isavailable to be copied to the user’s local hard drive, a > arrow icon will appear, and thefile can be imported by clicking and holding on the icon. When a networked flick isadded to a local playlist or its Keyframe Summary is edited, the metadata listing isimported. If the user requests to play it when the network resource is unavailable, a ! icon(also from iTunes) appears to indicate the absence of the original file.Making FlixLists

Just like iTunes and iPhoto, custom collections of flicks can be created, named, added toby drag-n-drop, and deleted. Flicks with particular themes, playlists for a flick-watchingparty, or “funny one-liners” can be brought together in memorable FlixLists, which canbe exported and shared with friends. For flicks with multiple summaries, a particularsummary can be specified to represent the flick in a given list (not shown). Meanwhile allflicks listings remain safely ever-present in the Personal Library.Search and Browse to find Flicks

Browsing in iFlix is inspired by iTunes. Click the Browse eye to show/hide the browsingpanel. The browsing panel has two sections, Names/Roles, and Type.Flicks may be full-length feature films, trailers, TV episodes, commercials, animationshorts, flash movies, and any number of additional types. Whichever of these types existin the metadata of the library or libraries selected on the Source Management Panel willappear on the Type browsing list. Select one of them to limit the list of flicks to only thattype.Similarly, Names/Roles pulls from the metadata in the selected source library. Thismetadata covers named people associated with flicks, and their role(s). For example,Robert Redford might be listed under Names. Associated with his name are the Roles“Actor” and “Director.” The names of the characters he played as an actor might alsoappear in the Names list. Click any of these Names or Name/Role combinations to filterthe flicks shown in the catalog listing.Search in iFlix is also inspired by iTunes and Mail.app, but with a small twist. Typingany text into the search box will start iFlix looking in all text fields for matches. The pulldown-menu allows a user to limit to the search to particular text fields. And the twist isthat the searched field is displayed in grey text in front of the search text, providing theuser feedback that the limit has been set.Customizing the Catalog Listing Display

By default, the catalog listing display shows certain columns, including displaying thefirst three keyframes in the Summary column. Use the icon size slider (from iPhoto! Andusing the Mac OS X Aqua rendering engine!) to show larger keyframes. Adjust columnwidths, for example to show more keyframes from summaries. Sort the listings by oneclick on the column header. Customize which columns appear in the catalog listingsusing the View>Columns menu settings (not shown, but modeled after those inMail.app).Finally, expose or hide the Browse panel by clicking the Browse eye.

Future designs should include an indicator of when the number of keyframes present in asummary is greater than the number shown in the Flix column. The Organize view shouldalso provide the ability to select which summary to display in a given list.

Sharing FlixLists and Keyframe SummariesSharing Tools

With one or more flicks selected in the catalog listing, the user can share the selection invarious ways. Where relevant, these features implement the iTunes model of intellectualproperty protection for flicks.Play is an additional way of entering Play mode with a selected flick.E-mail will compose an e-mail in your preferred e-mail application with the selectedmetadata file(s) or FlixList as an attachment. Summaries might also be shared via IM(not shown).Homepage facilitates publication of FlixLists or download pages for metadata files towebpages, including blogs. Like browsing someone’s bookshelf or movie collection inthe physical world, people share their movie titles on Friendster profiles, webpages andblogs as a way of identifying their perspective on the world. Sharing your collection ofkeyframe summaries can be a social activity just as sharing photos is.Import is an additional way of activating the import of flick(s) to your hard drive, or theirmetadata listing to your iFlix library, according to their status.Burn supports the creation of video CDs or DVDs from a FlixList or selected flick.

Summary – View and Edit a Flick’s Keyframe SummaryScreenshot

OverviewEach flick has an automatically generated summary composed of three keyframes andassociated text and music. The Summary mode provides for viewing, navigating, andenhancing Keyframe Summaries of a flick. Each flick can have multiple summaries, soyou can make your own and share them with friends!To create the automatic summary, three keyframes are selected from evenly spacedintervals in the flick, then optimized using the practices refined by the Informedia project.The associated text is generated with speech recognition on any audio track,supplemented with OCR of any video text. iFlix will also attempt to identify musicwithin the soundtrack, and provide a link to iTunes to purchase it.Flicks have a variety of different aspect ratios. iFlix will not perform pan-n-scan,letterboxing, or other adjustment to the flick’s aspect ratio. The Current Frame Screenand the Context Screens will automatically scale to fit into the are provided by thewindow size while adjusting for aspect ratios. Keyframes will adjust to fit the aspect ratiointo the same height of the Summary Display.

Add a Keyframe to a SummaryCustom summaries can be downloaded from friends and selected for viewing from theVersion bar. Make your own summary by adding to a copy of an existing summary.Simply playback the flick to an appropriate frame, type in any relevant text, and click“+”. The summary display shows a preview of where your keyframe will be added.Rearrange the keyframes on the timeline to create the most compelling summary.

Feature and Functionality DescriptionsMode MenuClick Organize to see the catalog listings, or click Play to start the media player with thecurrent flick. Clicking on a source or FlixList in the Source Management Panel to the leftwill exit Play and show the catalog in Organize view.

Frame Navigation PanelCurrent Frame Screen

The current frame or the active playback of the current flick is the main feature of theupper half of the Summary screen. While stopped to show a specific frame, the commenttext box is available for editing at the bottom of the image. Just below it is a linedisplaying the name of the music track, or that a music track has been detected near thisframe.Context Screens

To the right and left of the current frame screen, smaller displays show the frameimmediately preceding and following the current frame. This helps when deciding whichframe makes the perfect keyframe.Use the arrow button below each context preview to move the flick one frame forward orbackward at a time. Click and hold on a button to bring up a menu for selecting a numberof frames to jump greater than one.Scrub bar

The scrub bar presents a timeline view of the current flick. Light lines indicate shotbreaks detected automatically by iFlix, or imported as part of the flick’s metadata.A point-down triangle indicates the location of the currently-displayed frame within theflick’s timeline. Click and drag to relocate the playhead.Point-up triangles indicate where existing keyframes have been drawn from in the flick.Click one of these triangles to move the playhead to that point in the flick.Playback Buttons

Control the playback of the current flick with the Fast-forward, Play, and Fast-reversebuttons. Use the Fast-rate Controller in the Summary Tools panel below to adjust the rateat which the Fast-forward and Fast-reverse buttons skim through the flick.During playback, the comment text box and Context screens are inactivated.Add a new Keyframe to a Summary

With the flick’s playback paused, click the + button to add it to the summary currentlydisplayed in the Summary Display. A gap between two keyframes in the summaryindicates where the new keyframe will appear.

Summary DisplayVersions bar

Inspired by Safari’s Bookmarks Bar, the Summary Display’s versions bar shows thenames of the existing Keyframe Summaries for the current flick. By default, iFlix createsthe Auto three-keyframe summary, and places it first on the bar.If the user changes the Auto summary or clicks the + button, a new summary is created.E-mailed summary files can be added by drag-and-drop onto the catalog window or thiswindow. Menu actions (not shown) provide for the creation of automated summaries withdifferent defaults (more keyframes, or one per shot) and importing summaries from aURL or file location.Summary names highlight on rollover, and clicking the name will display that summary.The name of the currently-displayed summary is shown as an editable text box (exceptAuto). Summary names (except Auto) can be dragged into any order, or dragged off thebar to delete them.Whichever summary is selected when the user switches away from the Summary viewwill be the one used to represent the flick in the catalog listing.If the flick is streamed and the summary file has been imported, the Summary screen mayneed to take a minute to contact the flick source and collect the keyframe images beforedisplaying the summary.Future designs might tackle ways to display aggregate comments, addressing issues likehow to identify differences in comments or keyframe selection between highly-similarsummaries; ways to collate or view multiple summaries; and allowing users’ commentson a keyframe to reflect a social commenting chain, much like a chat room dialog orblog entry and comments.Keyframes on Display

Each summary is displayed as a series of Keyframe images and the first 15 characters ofthe comment text. A scroll bar will appear if all of the keyframes can’t fit on the screen.If music has been identified in the soundtrack around the keyframe, a music note iconwill appear (see Check Music).Keyframes can be re-ordered by drag-and-drop, just like on the iMovie clip orderingtimeline. Clicking the little X in the upper right corner will delete the keyframe, text, andmusic information from the summary.Navigating with a Summary

Clicking on any keyframe in the currently displayed summary will move the FrameLocation Panel’s Main Screen display to that keyframe’s location in the flick. KeyframeSummaries can be used as a way to provide DVD-like chapters for media.Keyframe location preview

Two keyframes will be spaced further apart to indicate where the currently displayedframe on the Frame Navigation Panel will go in the Summary sequence if it is added as akeyframe. By default, this will occur between the keyframe that most recently proceedsthe location of the playback head on the scrub bar, and the keyframe that follows it in thesequence.

Summary ToolsFast-Rate Control

Use this slider to adjust how fast “Fast” is for the Fast-Forward and Fast-Reverse controlson the Frame Navigation Panel. The tortoise on the right is the slowest setting, while thehare on the left is the fastest.Check Music

Check Music identifies any commercial music in the flick’s soundtrack, adding thisinformation to the current keyframe. Grabbing a few seconds of the music from beforeand after the current keyframe, Check Music uses an online music identification serviceto gather the name and other metadata about the music. This metadata is added in aspecial library in iTunes, and the keyframe will now display the track name and a musicnote icon to indicate a link to iTunes. iFlix uses this same tool on keyframesautomatically selected as part of the Auto Keyframe Summary.If iFlix does not detect any music within the soundtrack of the flick, this button will beinactivated.Get Text

Get Text uses optical character recognition (OCR) to identify text within the image.Titles or credits present in the image can be entered into the Comments with one click.iFlix uses this same tool on keyframes automatically selected as part of the AutoKeyframe Summary.Get Speech

Get Speech uses speech-to-text technology on any speech it recognizes within thesoundtrack, starting a few seconds before and ending a few seconds after the selectedkeyframe. This length is modified by the density of surrounding keyframes and thenatural phrasing of the speech to optimize the coherence of the text results. iFlix uses thissame tool on keyframes automatically selected as part of the Auto Keyframe Summary.If iFlix does not detect any speech within the soundtrack of the flick, this button will beinactivated.Slideshow

Because Keyframe Summaries may have more images than can comfortably be viewed atonce in the Summary viewer, the Slideshow functionality presents the images full-screen,with the comment text at the bottom of the picture. Moving the mouse during theSlideshow brings up an overlay menu with slideshow controls (like in iPhoto). Pausingthe slideshow allows the text for the current keyframe to be edited.iPhoto import/export

This tool provides various functionality for importing and exporting photos between iFlixand iPhoto. The images in an entire Keyframe Summary may be imported as a newalbum in iPhoto. Conversely, a photo from iPhoto can be imported to replace the visualimage of an existing keyframe, allowing the creation of custom poster frames or sectiontitles.

Play – Watch a FlickScreenshot

OverviewAny flick can be watched in Play mode. iFlix will either play the media for the formats itdirectly supports, or present the media player for the given file format seamlessly for theuser, including whichever controls the media player supports.

Feature and Functionality DescriptionsMode MenuClick Organize to see the catalog listings, or click Summary to view and edit theKeyframe Summary for the current flick. Clicking on a source or FlixList in the SourceManagement Panel to the left will exit Play and show the catalog in Organize view.

Playing a flick in iFlix media player

The iFlix media player is a QuickTime player, presented through the iFlix application andvery similar to iMovie. A point-down triangle on the timeline bar indicates where thecurrent frames are in the flick’s playback (not shown). Click and drag the triangle tochange the current location in the flick.Playback controls are located in the lower panel. Click the large right-pointing arrow toplay the flick. During playback , it will change to a square, and clicking it will stopplayback. Smaller buttons with double arrows will fast-forward or fast-rewind the flick.Click the + button to add the current frame to the flick’s Keyframe Summary.Activate full-screen playback by clicking the button with a soft rectangle around the playarrow (not shown). Moving the mouse in full-screen mode will bring up an overlay menuwith the playback controls.Future design work could optimize and streamline the ability to select and or share akeyframe with others directly from Play mode.

Playing a flick in another media playerFlicks in file formats other than those supported by QuickTime can often still be playedin iFlix. When entering Play mode, iFlix will check the player associated with the filetype and either open the player window and associated playback controls within iFlix’Play screen, or automatically start up the other player and switch to that application.Otherwise, iFlix may prompt the user to identify an application on the local hard drivethat can play the selected file. Full-screen playback and keyframe addition will besupported when possible.

System DiagramsInformation Flows including iFlixiFlix is an application designed to help collect, organize, and pass on information abouttime-based video assets within a community of users. Information flows through iFlix inthe following manner:

In more detail:

Interviews with UsersMethodsWe showed alpha screenshots of iFlix to potential users, and asked them to describe whatthey thought any given screen widget was for, and how they might use such anapplication. The potential users are all undergraduate UC Berkeley students between theages of 19 and 24, residents of a 150-person community house with an active media-sharing community over the local LAN. They are representatives of “The Socializer”audience for iFlix.In a separate document, we present a rough transcript of their comments. Our ownquestions or clarifications are in [ ]. Names have been changed to protect the innocent.

Alpha ScreenshotsOrganize

Summary

ConclusionsMuch of the search, browse, network/share, mediaplayer, and keyframe functionality isintuitive to potential users. They are able to translate effectively from their experiencewith iTunes and movie editing software.The static keyframe summary seemed to be a natural entity, and something many couldenjoy “consuming”, and a few might enjoy creating. Sharing them would be an essentialpart of the point of making them.The keyframe summary would be a nifty way of navigating through a flick.Collections of highly-specific custom keyframe summaries could be a way for fans torecord their preferred combinations of TV series moments.The music-recognition and purchasing feature is exciting to users, but not intuitivelypresented yet in the alpha version of the Summary mode screen.The Stain Glass is not named correctly, and although it seems cool, doesn’t seem to haveas much motivation to be created as the keyframe summary does. This could be anartifact of the absence of having a screenshot for it for these interviews, however.The iPhoto and Homepage features have potential; people have ideas about what theymight be, although those ideas aren’t as clearly formed as the keyframe summary.

These interviews were not a good test of the proposed interaction for actually editingkeyframe summaries. Future interviews should give users a sense of how OCR andspeech-to-text would help generate comments for keyframes.

ReferencesUser Preference for Static SummariesGary Marchionini’s workhttp://ils.unc.edu/~march/also http://www.open-video.org/

slide show from AVI '98http://ils.unc.edu/~march/avi98/sld016.htmSummary slide appears to indicate that users prefer static summary displays of video.

How Fast is Too Fast?How Fast is Too Fast? Evaluating Fast Forward Surrogates for Digital VideoACM/IEEE 4Joint Conference on Digital Libraries, May 29, 2003 (Talk for theVannevar Bush Best Paper Award)http://ils.unc.edu/~march/jcdl2003/How_Fast_files/v3_document.htm

Slide 7 has a list of different types of surrogates for video. Slide 17 lists a summary ofresults that suggests while people can understand video in extreme fast-forward, theydon’t like it. Also that users should have control of video summaries, but with appropriatedefaults.

Keyframe Selection for Video SummariesAutomatedfxpal work on Manga and MBasehttp://www.fxpal.com/?p=mangahttp://www.fxpal.com/?p=mbaseTime-Constrained Keyframe Selection Technique. Andreas Girgensohn, and JohnBoreczky. In Multimedia Tools and Applications, 11(3), pp. 347-358, 2000.,August 1,2000 http://www.fxpal.com/?p=abstract&abstractID=42Keyframe-Based User Interfaces for Digital Video Andreas Girgensohn, John Boreczky,and Lynn Wilcox. IEEE Computer, 34(9), pp. 61-67, September 1, 2001http://www.fxpal.com/?p=abstract&abstractID=6Video Manga: Generating Semantically Meaningful Video Summaries. ShingoUchihashi, Jonathan Foote, Andreas Girgensohn, and John Boreczky. In ProceedingsACM Multimedia, (Orlando, FL) ACM Press, pp. 383-392, 1999., October 30, 1999http://www.fxpal.com/?p=abstract&abstractID=136

Informedia

http://www.informedia.cs.cmu.edu/dli2/index.htmlThis project has also developed good principles for selecting appropriate phases among aspeech soundtrack to associate with a keyframe.

VirageAutomated plus human inputThe Open Video Digital Library

Gary Marchionini and Gary Geisler. The Open Video Digital Library. D-LibMagazine. December 2002 V8:12http://www.dlib.org/dlib/december02/marchionini/12marchionini.htmlFigures 1 and 2 show interfaces similar to those we are proposing. The article alsodescribes creating keyframe summaries through a combination of automated andmanual actions, as well as efforts to capture what others think about videos and whohas used a video for what.

Navigating video using keyframesfxpal work on Mangahttp://www.fxpal.com/?p=mangaAn Interactive Comic Book Presentation for Exploring Video. John Boreczky, AndreasGirgensohn, Gene Golovchinsky, and Shingo Uchihashi In CHI 2000 ConferenceProceedings, ACM Press, pp. 185-192, 2000., April 1, 2000http://www.fxpal.com/?p=abstract&abstractID=86

Navigating among keyframesfxpal work on MBasehttp://www.fxpal.com/?p=mbase

Music-recognition servicesMusicBrainz, an open-source music recognition initiative - http://www.musicbrainz.org/“Philips and Gracenote Launch Gracenote Mobile(SM) -- First Global Music Recognitionand Content Delivery Service for Mobile Phones” – Marketwire,http://www.marketwire.com/mw/release_html_b1?release_id=61431Shazam is a UK-based phone service provider that offers music recognition, facilitatingringtone and CD purchases. http://www.shazam.com/uk/do/home

On the audiencesThe “Fan” audience type is described in: Jenkins, H. Textual Poachers: TelevisionFans & Participatory Culture. Routledge, New York, 1992;

The “Socializer” audience type is a media-centered variation on the mobile-phone gift-giving social culture described in: Taylor, A. S. & Harper, R. (2003). The gift of

the gab? A design oriented sociology of young people's use of mobiles.Computer Supported Cooperative Work, 12, 267-296.