1516 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 43, NO. 6, DECEMBER 2013

Guest EditorialIntroduction to the Special Issue on

Modern Control for Computer GamesAbstract—A typical gaming scenario, as developed in the

past 20 years, involves a player interacting with a game us-ing a specialized input device, such as a joystic, a mouse, akeyboard, etc. Recent technological advances and new sensors(for example, low cost commodity depth cameras) have enabledthe introduction of more elaborated approaches in which theplayer is now able to interact with the game using his body pose,facial expressions, actions, and even his physiological signals.A new era of games has already started, employing computervision techniques, brain–computer interfaces systems, haptic andwearable devices. The future lies in games that will be intelligentenough not only to extract the player’s commands providedby his speech and gestures but also his behavioral cues, aswell as his/her emotional states, and adjust their game plotaccordingly in order to ensure more realistic and satisfactorygameplay experience. This special issue on modern control forcomputer games discusses several interdisciplinary factors thatinfluence a user’s input to a game, something directly linked tothe gaming experience. These include, but are not limited to,the following: behavioral affective gaming, user satisfaction andperception, motion capture and scene modeling, and completesoftware frameworks that address several challenges risen in suchscenarios.

The games industry has grown to be one of the mainstreammarkets in our days. At the beginning (around the 1970s) thegames industry constituted a focused market, highly dependenton specialized input sensors to enable the interaction betweena player and the game. Indeed, in typical games, the user had tobe familiar with an input device, such as a keyboard, a mouse,a joystic, or a console in order to properly communicate withthe game. Furthermore, the game had a predesigned plot thatwould progress along with the player’s actions in a predefinedway, giving the feeling of a nonexistent, in reality, controlover the game evolution. Moreover, in such a gaming scenario,several issues had to be tackled. The game had to be carefullydesigned and developed so as to allow real-time interactionwith the player and to ensure high-quality environment so thatthe immersion of the player in the game environment wouldbe as realistic as possible, and of course to employ devicesthat would be of affordable cost.

Recent technological advances have opened new avenuestoward more realistic human–machine interaction systems,thus enabling the introduction of games in which the player isnot only able to control the game but also can control thegameplot and gameplay experience without using an inputdevice, just by freely acting on his/her will. More specifically,the introduction of Microsoft Kinect has enabled the robustextraction of body poses and joint locations in real time, while

Date of publication October 17, 2013; date of current version November18, 2013.

Digital Object Identifier 10.1109/TCYB.2013.2283551

at the same time being of low cost. This did not only makethe gaming experience more realistic, but also opened newavenues in game design. Therefore, current games designersand developers can greatly benefit from its use to producerealistic systems.

Emphasis has now been given toward affective gamingscenarios. The term affective gaming corresponds to the intro-duction of affect recognition in games. These approaches triedto incorporate emotion recognition in their systems to extractthe emotional state of the player and to use it to control thegameplot and gameplay experience.

In this special issue, we are pleased to present seven papersthat are related to different aspects of modern control forcomputer games. All resulted from a careful review process(at least two rounds of reviews).

Emotional states do not only play a crucial role in enhancingthe game play but also in controlling the actual game (affectivegaming). Further to this, the paper entitled, “Fusing Visual andBehavioral Cues for Modeling User Experience in Games,”investigates the creation of an accurate model of player’s expe-rience based on a data corpus consisting of 58 subjects playinga popular game. Gameplay behavior, as well as player visualcharacteristics, was used to create models for engagement,frustration, and challenge. Subsequently, the derived playerexperience models were used to personalize the game level todesired levels of engagement, frustration, and challenge basedon the assumption that game content can be mapped to playerexperience through the behavioral and expressivity patterns ofeach player.

Surface motion capture (SurfCap) of a subject providesinformation useful for both game control and design. Thepaper entitled, “Animation Control of Surface Motion Cap-ture,” introduces techniques for interactive animation controlof SurfCap sequences that allow the flexibility in editingand interactive manipulation associated with existing toolsfor animation from skeletal motion capture (MoCap). To thisend, Laplacian mesh editing is extended using a basis modellearnt from SurfCap sequences to constrain the surface shapeto reproduce natural deformation. Three novel approachesfor animation control of SurfCap sequences are introducedand combined with high-level parametric control of SurfCapsequences in a hybrid surface and skeleton driven animationcontrol framework to achieve natural surface deformation withan extended range of movement by exploiting existing skeletalmotion capture archives.

Later in this special issue, two papers that interestinglyrelate unobtrusive control acquired using audio and visualcues with games are presented. In more detail, the paper

2168-2267 c© 2013 IEEE

ARGYRIO et al.: GUEST EDITORIAL 1517

entitled, “Multisignal Vision and Speech Interaction System ina Gaming Context,” introduces a novel system that integrateseye gaze estimation, head pose estimation, facial expressionrecognition, speech recognition, and text-to-speech compo-nents for use in real-time games. To this end, a novel head poseestimation algorithm is proposed combining scene flow witha generic head model, while a local binary patterns with threeorthogonal planes (LBP-TOP) approach applied on the 2-Dshape index domain is used to recognize facial expressions.

The paper entitled, “Depth-Color Fusion Strategy for 3-DScene Modeling With Kinect,” focuses on the use of MicrosoftKinect sensor and the inherent problems due to noise relatedto the provided depth data. The presence of noise significantlyaffects the game performance as in many games it considerablyreduces the accuracy. In this paper, accurate depth and colormodels of the background elements are iteratively built, andused to detect moving objects in the scene. An adaptive joint-bilateral filter that efficiently combines depth and color byanalyzing an edge-uncertainty map and the detected fore-ground regions is designed and implemented to this end. Theresults verify that the proposed approach reduces the Kinectdata problems such as spatial noise and temporal randomfluctuations, thus better refining the objects depth boundaries.

Visual and auditory cues are important facilitators of userengagement in virtual environments and video games. Not sur-prisingly, it has recently been suggested that auditory stimuliaffects human perception of fidelity (quality). Hence, studyingand understanding the exact effect of multimodal stimuli onvisual fidelity perception can potentially impact the design ofvirtual environments and games and lead to the creation ofmore engaging virtual worlds and scenarios. The next paperentitled, “The Effect of Sound on Visual Fidelity Perception inStereoscopic 3-D,” investigates the effect of auditory stimulion human perception of visual fidelity within a stereoscopic3-D environment.

Another issue that is raised is which level of controlprovides the desirable result in games. This is discussed in thepaper entitled, “How Much Control Is Enough?’ The authorsinvestigate on whether the amount of control can be modulatedso that a game can be fun without providing to the playerperfect control. To this end, experiments are conducted withusers trying to guide a hamster to the exit of a maze, witha varying level of control at each attempt. The correlation offrustration, fun, and control are thoroughly discussed.

Finally, a general software framework for the developmentand research in the area of multimodal human–computerinteraction (HCI) systems (such as games) is presented inthe paper entitled, “HCI2 Framework: A Software Frameworkfor Multimodal Human–Computer Interaction Systems.” Moreprecisely, the HCI2 Framework is built upon publish/subscribe(P/S) architecture. It implements a shared-memory-based datatransport protocol for message delivery and a TCP-basedsystem management protocol. The authors also present itsintegrated development environment that provides a completegraphical environment to support every step in a typical MHCIsystem development process, including module development,debugging, packaging, and management, as well as the wholesystem management and testing. An example of a full system

developed using the proposed HCI2 framework, which rep-resents a computer game based on hand-held marker(s) andlow-cost camera(s), is also provided.

Acknowledgment

The Guest Editors would like to thank all the authors whohave contributed to this special issue and all the reviewerswho have provided valuable comments. They would like tothank the IEEE Transactions on Cybernetics (formerlyIEEE Transactions on Systems, Man, and Cybernetics:

Part B) for supporting this special issue, with special thanksto the Editor-in-Chief Prof. E. Santos for his support. Theywould also like to thank the Editorial Assistants L. Cullenand T. Martin, as well as the Journal Coordinator M. Rafferty,for their professional support with the editorial matters duringthe preparation of this special issue.

Special Tribute to Prof. Maria Petrou

During the preparation of this special issue, one of the GuestEditors, Prof. M. Petrou, Imperial College London, London,U.K., and the Director of the Information Technologies In-stitute, Centre for Research and Technology Hellas, Hellas,Greece, passed away. The Guest Editors all have been deeplysaddened by the sudden loss of a great scholar, a colleague,and a friend. All the Editors are very much honored to haveserved with her as Guest Editors on this special issue.

VASILEIOS ARGYRIOU, Member, IEEE, Guest EditorDepartment of Computing, Information Systemsand MathematicsKingston University,Kingston upon ThamesSurrey KT1 2EE,U.K.(e-mail: vasileios.argyriou@kingston.ac.uk)

IRENE KOTSIA, Guest EditorSchool of Science and TechnologyMiddlesex University, The BurroughsLondon, NW4 4BT, U.K.(e-mail: i.kotsia@mdx.ac.uk)

STEFANOS ZAFEIRIOU, Guest EditorDepartment of ComputingImperial College London,London SW7 2AZ, U.K.(e-mail: s.zafeiriou@imperial.ac.uk)

MARIA PETROU, Senior Member, IEEE, Guest EditorDepartment of Electrical and Electronic EngineeringImperial College London, London, U.K.Informatics and Telematics InstituteCentre of Research and Technology - Hellas,Thessaloniki 57001, Greece(e-mail: maria.petrou@imperial.ac.uk)

1518 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 43, NO. 6, DECEMBER 2013

Vasileios Argyriou (M’10) received the B.Sc. degree in computer science from Aristotle University of Thessaloniki, Thessaloniki,Greece, in 2001, and the M.Sc. and Ph.D. degrees from the University of Surrey, Surrey, U.K., in 2003 and 2006, respectively, bothin electrical engineering.

From 2001 to 2002, he held a research position with the AIIA Laboratory, Aristotle University of Thessaloniki, involved in imageand video watermarking. From 2002 and 2006, he participated in many European projects for archive file restoration (PrestoSpace) andsubpixel motion estimation collaborating with Snell and Wilcox. He joined the Communications and Signal Processing Department,Imperial College, London, U.K., in 2007, where he was a Research Fellow involved in 3-D image reconstruction from photometricstereo. He is currently a Senior Lecturer with the Department of Computing, Information Systems and Mathematics, KingstonUniversity, London, U.K, where he is engaged in action recognition and AI for computer games.

Dr. Argyriou is a member of the IET.

Irene Kotsia received the B.Sc. and Ph.D. degrees from the Department of Informatics, Aristotle University of Thessaloniki,Thessaloniki, Greece, in 2002 and 2008, respectively.

From 2008 to 2009, she was a Research Associate and Teaching Assistant with the Department of Informatics, Aristotle Universityof Thessaloniki. From 2009 to 2011, she was a Research Associate with the Department of Electronic Engineering and ComputerScience, Queen Mary University of London, London, U.K., and since 2012, she has been a Research Associate with the Departmentof Computing, Imperial College, London, U.K., and a Lecturer in creative technology and digital creativity with the Department ofComputing Science, Middlesex University, London, U.K. She has co-authored over 35 journal and conference publications.

Stefanos Zafeiriou received the B.Sc and Ph.D degrees from Department of Informatics, Aristotle University of Thessaloniki, in2003 and 2007, respectively.

He is currently a Lecturer (equivalent to Assistant Professor) in pattern recognition/statistical machine learning for computervision with the Department of Computing, Imperial College, London, U.K. He has co-authored over 31 journal papers mainly onnovel statistical machine learning methodologies applied to computer vision problems such as 2-D/3-D face and facial expressionrecognition, deformable object tracking, human behavior analysis, etc., published in the most prestigious journals in his field ofresearch (such as the IEEE Transactions on Pattern Analysis and Machine Intelligence, International Journal of ComputerVision, the IEEE Transactions on Image Processing, the IEEE Transactions on Neural Networks and Learning Systems,the IEEE Transactions on Visualization and Computer Graphics, the IEEE Transactions on Information Forensics

and Security, etc.). He has participated in more than ten European Union, British, and Greek research projects. His students arefrequent recipients of very prestigious and highly competitive fellowships such as Google Fellowship, Intel Fellowship, and Qualcomm

Fellowship. He has over 900 citations to his work.Dr. Zafeiriou has been awarded one of the prestigious Junior Research Fellowships from Imperial College London in 2011 to start his own independent

research group. He currently serves as an Associate Editor in the IEEE Transactions on Cybernetics and Image and Vision Computing. He has been aGuest Editor in more than four special issues and co-organized more than five workshops and special sessions in top venues, such as CVPR/FG/ICCV/ECCV.

Maria Petrou (A’90–M’91–SM’05) received the degree in physics from Aristotle University of Thessaloniki, Thessaloniki, Greece,the degree in applied mathematics from University of Cambridge, Cambridge, U.K., the Ph.D. degree from the Institute of Astronomy,Cambridge, U.K., and the D.Sc. degree from Cambridge, in 2009.

She was the Chair of Signal Processing with the Department of Electrical and Electronic Engineering, Imperial College London,London, U.K., and was the Director of the Informatics and Telematics Institute, Centre of Research and Technology, Hellas, Greece.She has published over 350 scientific papers on astronomy, remote sensing, computer vision, machine learning, color analysis,industrial inspection, medical signal, and image processing. She has co-authored two books, Image Processing: The Fundamentals(Wiley, first edition 1999 and second edition 2010) and Image Processing: Dealing With Texture (Wiley, 2006). She has also co-editedthe book Next Generation Artificial Vision Systems: Reverse Engineering the Human Visual System. She has supervised to successfulcompletion of 43 Ph.D. theses.

Dr. Petrou was a fellow of the Royal Academy of Engineering, the City and Guilds Institute, the Institution of Engineeringand Technology, the International Association for Pattern Recognition, the Institute of Physics, and a Distinguished Fellow of the British Machine VisionAssociation. She served as a Trustee of the IET from 2006 to 2009, the IAPR Newsletter Editor from 1994 to 1998, and the IAPR Treasurer from 2002 to2006.