interfaces for human-centered production and use of computer … cannavò_presentation.pdf · prof....
TRANSCRIPT
Interfaces for human-centered production and use of computer graphics assets
Alberto Cannavò
XXXII cycle - Computer and Control Engineering
CommiteeProf. Bill Kapralos, Referee, University of Ontario Institute of TechnologyProf. Adalberto Simeone, Referee, Katholieke Universiteit LeuvenProf. Giuseppe Serra, Università di UdineProf. Riccardo Sisto, Politecnico di TorinoProf. Filippo Stanco, Università di Catania
SupervisorProf. Fabrizio Lamberti, Politecnico di Torino
INTRODUCTIONComputer Graphics
Core enabling technologies, e.g., for information visualization, Virtual Reality (VR) and Augmented Reality (AR)
INTRODUCTIONChallenges
Production & Utilization
Different user categories Expert users
First-time users
To improve the effectiveness of existing methods, tools and paradigms for the
production and use of computer
graphics content
By leveraging recent advances in the
fields of multi-modal interactionand intelligent computing
INTERFACES FOR HUMAN-CENTERED PRODUCTION AND USE OF COMPUTER GRAPHICS ASSETS
GOALResearch activities
AGENDAResearch activities
• Computer animation: How to bring a virtual character to life
• Is immersive virtual reality the ultimate interface for 3D animators?
• Interfaces and methods supporting the generation of graphics assets
• Leveraging graphics assets: Interactive applications development
• Augmented, mixed and virtual reality: A new perspective for human-machine interaction
COMPUTER ANIMATION: HOW TO BRING A VIRTUAL CHARACTER TO LIFE
• Labor-intensive task• Involvement of users
with significant expertise
• Sophisticated interfaces
• Motion capture• Reconfigurable tangible
user interface (TUI)• Natural language
processing (NLP)
[1] I., Satoru, et al. "Performance-based control interface for character animation", ACM Transactions on Graphics, 2009.
[2] A., Dennis & A. Whitehead "Wearable sensor networks for motion capture", Proc. Intelligent Technologies for Interactive Entertainment, 2015.
[3] N. Numaguchi, et al. "A puppet interface for retrieval of motion capture data", Proc. SIGGRAPH/Eurographics Symposium on Computer Animation, 2011.
[4] Y. Wataru, et al. "An actuated physical puppet as an input device for controlling a digital manikin", Proc. Conference on Human Factors in Computing Systems, 2011.
[5] J., Alec, et al. "Tangible and modular input device for character articulation", ACM Transactions on Graphics, 2015.
[6] O. Glauser, et al. "Rig animation with a tangible and modular input device", ACM Transactions on Graphics, 2016.
[7] M. Oshita. "Generating animation from natural language texts and semantic analysis for motion search and scheduling ", The Visual Computer, 2010.
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
Generically and Specially Shaped Tangible Devices
[4][3]
Reconfigurable Tangible Interfaces
[5] [6]
Related work
[1] [2]
Body and Hand Motion Tracking
Natural Language Processing
[7]
“Neo waves to Jack. At the same time,
Jack takes the red bottle”
Reconfigurable TUIs & Body tracking
Keyframing & Performance driven animation
Architecture designed for extensibility
F. Lamberti, G. Paravati, V. Gatteschi, A. Cannavò, P. Montuschi "Virtual character animation based on affordable motion capture and reconfigurable
tangible interfaces", IEEE Transactions on Visualization and Computer Graphic, 2017
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
Motion capture and reconfigurable tangible interfaces
F. Lamberti, G. Paravati, V. Gatteschi, A. Cannavò, P. Montuschi "Virtual character animation based on affordable motion capture and reconfigurable
tangible interfaces", IEEE Transactions on Visualization and Computer Graphic, 2017
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
Motion capture and reconfigurable tangible interfacesAutomatic mapping configurator:- Assignment problem
- Assembly to control all the DOFs
- Building step-by-step instructions
- Supporting forward and inverse kinematics
F. Lamberti, G. Paravati, V. Gatteschi, A. Cannavò, P. Montuschi "Virtual character animation based on affordable motion capture and reconfigurable
tangible interfaces", IEEE Transactions on Visualization and Computer Graphic, 2017
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
52 participants:- 26 Unskilled users (UNUs)- 26 Skilled users (SKUS)
Four animation tasks:- Lamp ref task- Crocodile ref task- Dyno ref task- Dyno free task
Motion capture and reconfigurable tangible interfaces
Animations systems evaluated:- Proposed System (Prop.)- Mouse & Keyboard (M&K)
Objective evaluation:- Completion time- Animation accuracy- Amount of work
Subjective evaluation:- Post-test questionnaire
Lamp ref task
Crocodile ref task
Dyno ref task
F. Lamberti, G. Paravati, V. Gatteschi, A. Cannavò, P. Montuschi "Virtual character animation based on affordable motion capture and reconfigurable
tangible interfaces", IEEE Transactions on Visualization and Computer Graphic, 2017
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
UNU SKU
Motion capture and reconfigurable tangible interfaces
Subjective evaluation
Objective evaluation
Required inputs:- screenplay- actions library
Modules:- Human-Computer Interaction (HCI)- Semantic Engine (SE)- Rendering Engine (RE)
F. Lamberti, V. Gatteschi, A. Sanna, A. Cannavò "A Multimodal Interface for Virtual Character Animation based on Live Performance and Natural Language Processing ",
International Journal of Human–Computer Interaction, 2019
Multimodal interfaces based on live performance and natural language processing
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
F. Lamberti, V. Gatteschi, A. Sanna, A. Cannavò "A Multimodal Interface for Virtual Character Animation based on Live Performance and Natural Language Processing ",
International Journal of Human–Computer Interaction, 2019
Multimodal interfaces based on live performance and natural language processing
50 participants:- 25 UNUs (Prop. & Blender)- 25 SKUS (Prop.)
wag: 2.6
walk: 1.1
P: The dog is in the garden (CAMERASIDE) and is wandering wagging its
tail
C: The dog exits from the garden (CAMERAFRONT) and walks looking
around (CAMERASIDE)
N: The dog looks at the balls on the walkway and smells the foods
P: The dog exits from the garden (CAMERAFRONT) and walks looking
around (CAMERASIDE)
C: The dog looks at the balls on the walkway and smells the foods
N: Once the dog arrives close to the river it lies down for a moment
on the shore
look to Actor_Ball_2
jump to Actor_Log_2
P: The dog gets up (CAMERABACK) and hops on the log that is floating
on the water
C: Finally the dog jumps on the bridge
N:
GardenWalkway
River
Objective evaluation:- Completion time
Subjective evaluation:- Usability (Nielsen) - Effectiveness (Dimension Star Model)
Animation 1 Animation 2
Animation 1 Animation 2
Post-test questionnaire
Post-test questionnaire
Animation 1 Animation 2
Skilled
Unskilled
CO
MPU
TER A
NIM
ATION
:
HO
W TO
BR
ING
A V
IRTU
AL C
HA
RA
CTER
TO LIFE
F. Lamberti, V. Gatteschi, A. Sanna, A. Cannavò "A Multimodal Interface for Virtual Character Animation based on Live Performance and Natural Language Processing ",
International Journal of Human–Computer Interaction, 2019
Multimodal interfaces based on live performance and natural language processing
NIELSEN DIMENSION STAR MODEL
0
1
2
3
4
5
−Concr
eteness
*
User c
ontributio
n
Cohesion*
Continuity
*
Stru
cture
−Cogniti
ve effo
rt*
Virtualit
y
Spat
iality*
Control
Inte
ract
ivity
*
Collabora
tion*
Imm
ersion*
Blender Proposed
Blender
Proposed
0
1
2
3
4
5
Lear
nabily
Effice
ncy
Mem
orabilit
y
Errors
Satis
fact
ion
Skilled Unskilled
0
1
2
3
4
5
−Concr
eteness
User c
ontributio
n
Cohesion
Continuity
Stru
cture
−Cogniti
ve effo
rt
Virtualit
y
Spat
iality
Control
Inte
ract
ivity
Collabora
tion
Imm
ersion
Skilled Unskilled
NIELSEN DIMENSION STAR MODEL
UNUs
SKUs vs UNUs
IS IMMERSIVE VIRTUAL REALITY THE ULTIMATE INTERFACE FOR 3D ANIMATORS?
Sense of presence3D interaction paradigm
VIRTUAL REALITY
SYSTEM OUTPUT
USER INPUT
[1] D. Vogel, et al. "AnimationVR – Interactive controller-based animating in virtual reality", Proc. IEEE Conference on Virtual Reality and 3D User Interfaces, 2018
[2] B. Berford, et al. "Building an animation pipeline for VR stories", Proc. ACM SIGGRAPH 2017 Talks, 2017.
[1] [2]
Research prototypes Commercial tools
Animations functionalities
Integration with animation suites
Extensive validations
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Related work
F. Lamberti, A. Cannavò, P. Montuschi "Is immersive virtual reality the ultimate interface for 3D animators?", COMPUTER, 2020.
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation
Buttons statusController
position/orientation
3D images of Blender’s viewport
VIRTUAL REALITY PLUGIN
BLENDER
HTC VIVE CONTROLLERS
HTC VIVE HEADSET
Transformations/properties changes
User’s commands
User & elements position/orientation
User’s head position/
orientation
VIRTUALREALITY
VIEWPORT
Animation functionalities accessible in VR
Tool for improving some of the stages of the animation pipeline
F. Lamberti, A. Cannavò, P. Montuschi "Is immersive virtual reality the ultimate interface for 3D animators?", COMPUTER, 2020.
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation27 participants:
- 15 Non-professional users (NPUs)- 12 Professional users (PRUs)
Five animation tasks:- Posing- Keyframing- Path animation- Performance anim.- Interpolation curve editing
Animations systems evaluated:
- VR-based interface (VRI)- Mouse & Keyboard (BNI)
Objective evaluation:- Completion time- Animation accuracy
Subjective evaluation:- System usability (SUS)- User’s satisfaction
Posingtask
Keyframing task
Performance animation task
Interpolation curve editing task
Path animation task
F. Lamberti, A. Cannavò, P. Montuschi "Is immersive virtual reality the ultimate interface for 3D animators?", COMPUTER, 2020.
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation
A. Cannavò, C.G. Demartini, L. Morra, F. Lamberti "Immersive virtual reality-based interfaces for character animation", IEEE ACCESS, 2019.
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation
New animation functionalities supporting Rigging, Skinning and Posing of virtual characters
BLENDER
Controller position/orientation
Buttons status
3D images of Blender’s viewport
VIRTUAL REALITY PLUGIN
HTC VIVE CONTROLLERS
HTC VIVE HEADSET
Transformations/properties changes
User’s commands
User & elements position/orientation
User’s head position/orientation
VIRTUALREALITY
VIEWPORT
INTERFACECONFIGURATOR
Configurations
Mapping
Customize VR functionalities
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation23 participants:
- 13 Non-professional users (NPUs)- 10 Professional users (PRUs)
Four animation tasks:- Rigging- Skinning- Posing- Complete anim. pipeline
Animations systems evaluated:
- VR-based interface (VRI)- Mouse & Keyboard (BNI)
A. Cannavò, C.G. Demartini, L. Morra, F. Lamberti "Immersive virtual reality-based interfaces for character animation", IEEE ACCESS, 2019.
Riggingtask
Skinning task
Posing task
Complete animation
pipeline task
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?Immersive virtual reality-based interfaces for character animation
A. Cannavò, C.G. Demartini, L. Morra, F. Lamberti "Immersive virtual reality-based interfaces for character animation", IEEE ACCESS, 2019.
PRUs
Subjective evaluation
Objective evaluation
A. Cannavò, F. Lamberti "A Virtual Character Posing System based on Reconfigurable Tangible User Interfaces and Immersive Virtual Reality",
Proc. Smart Tools and Applications in Graphics (STAG), 2018.
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?A virtual character animation system based on a reconfigurable TUI and immersive VR
Navigation of the scene
Selection Modalities:• Gaze selection• Proximity selection
VR Visual Feedbacks
20 participants:- 10 Unskilled users (UNUs)- 10 Skilled users (SKUs)
Animations systems evaluated:- Virtual Reality system (VRS)- Projected system (PS)- Mouse & Keyboard (M&K)
SKUs
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?
A. Cannavò, F. Lamberti "A Virtual Character Posing System based on Reconfigurable Tangible User Interfaces and Immersive Virtual Reality",
Proc. Smart Tools and Applications in Graphics (STAG), 2018.
A virtual character animation system based on a reconfigurable TUI and immersive VR
Animation task:- Posing task
Posing character through 3D sketching
Optimization problem
IS IMM
ERSIV
E VIR
TUA
L REA
LITY THE U
LTIMATE
INTER
FAC
E FOR
3D A
NIM
ATOR
S?
Rigged character
3D sketches
Characterpose
corr. matrix
input stroke
deformed vertices regularization term
constraints
ω: correspondence param.p: pose param.
y1
y2
yM
v1
v2
vK
3D sketch
Character skeleton
A. Cannavò, C. Zhang, W. Wang, F. Lamberti "Posing 3D Characters in Virtual Reality through In-the-Air Sketches",
Proc. 33rd International Conference on Computer Animation and Social Agents (CASA), 2020.
INTERFACES AND METHODS SUPPORTING THE GENERATION OF GRAPHICS ASSETS
3D computer graphics software suites
Steep learning curve
High flexibility
A. Cannavò, A. D'Alessandro, D. Maglione, G. Marullo, C. Zhang, F. Lamberti "Automatic Generation of Affective 3D Virtual Environments from 2D Images",
Proc. 15th International Conference on Computer Graphics Theory and Applications (GRAPP), 2020.
INTER
FAC
ES AN
D M
ETHO
DS SU
PPOR
TING
THE
GEN
ERATIO
N O
F GR
APH
ICS A
SSETSAutomatic generation of affective 3D virtualenvironments from 2D images
Source image Generated scene
3.08
2.75
3.58 3.17
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
4.25 4.08
4.25 4.42
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
Indoor Set. #1 Indoor Set. #2 Outdoor Set.#1 Outdoor Set. #2
Scene similarity Mood perception
A. Cannavò, F. Cermelli, V. Chiaramida, G. Ciccone, F. Lamberti, P. Montuschi, G. Paravati, "T4T: Tangible interface for tuning 3D object manipulation tools",
Proc. IEEE Symposium on 3D User Interfaces (3DUI), 2017.
G. Attanasio, A. Cannavò, F. Cibrario, F. Lamberti, P. Montuschi, G. Paravati, "HOT: Hold your own tools for AR-based constructive art",
Proc. IEEE Symposium on 3D User Interfaces (3DUI), 2017.
Interaction paradigm: - Cursor mode: virtual cursor- Tuning mode: tangible interface
INTER
FAC
ES AN
D M
ETHO
DS SU
PPOR
TING
THE
GEN
ERATIO
N O
F GR
APH
ICS A
SSETSAugmented reality for constructive art
Interaction paradigm:- Multi-touch capabilities- Printed markers
T4T: Tangible interface for tuning 3D object manipulation tools
HOT: Hold your own tools for AR-based constructive art
A participative system for tactics analysis in sport training based on immersive VR
A. Cannavò, M. Musto, F.G. Pratticò, F. Raho, F. Lamberti "A Participative System for Tactics Analysis in Sport Training based on Immersive Virtual
Reality", Proc. 4th Workshop on Everyday Virtual Reality (WEVR), 2018.
Pla
yer
app
licat
ion
Co
ach
ap
plic
atio
n
INTER
FAC
ES AN
D M
ETHO
DS SU
PPOR
TING
THE
GEN
ERATIO
N O
F GR
APH
ICS A
SSETSVirtual reality for sport training
A. Cannavò, D. Calandra, G. Basilicò, F. Lamberti "Automatic Recognition of Sport Events from Spatio-Temporal Data: an Application for Virtual
Reality-based Training in Basketball", Proc. 14th International Conference on Computer Graphics Theory and Applications (GRAPP), 2019
Automatic recognition of sport events from spatio-temporal data: An application for virtual reality-based training in basketball
VR Playbook
LEVERAGING GRAPHICS ASSETS: INTERACTIVE APPLICATIONS DEVELOPMENT
VR EXPERIENCE ALL NIPPON AIRWAYS (ANA)NATIONAL MUSEUM SINGAPORE,
SINGAPORE
MUSEO ARCHEOLOGICO
VIRTUALE (MAV), ITALY
SWAROVSKI VRSHOPPING EXPERIENCE
End-user development:- Visual language programming
Difficult for unskilled users
A. Cannavò, F. De Pace, F. Salaroglio, F. Lamberti "A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions",
International Journal of Human-Computer Studies, 2020
LEVER
AG
ING
GR
APH
ICS A
SSETS: INTER
AC
TIVE
APPLIC
ATION
S DEV
ELOPM
ENT.
A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions
GUI of Visual Scene Editor
A. Cannavò, F. De Pace, F. Salaroglio, F. Lamberti "A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions",
International Journal of Human-Computer Studies, 2020
LEVER
AG
ING
GR
APH
ICS A
SSETS: INTER
AC
TIVE
APPLIC
ATION
S DEV
ELOPM
ENT.
A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions20 participants:
- Three Expert users - 14 Unskilled users
Objective evaluation:- Completion time- Number of blocks- Number of errors
Tools evaluated:- Visual Scene Editor (VSE)- Leap Embedder (LE)- Blender Game Engine (BGE)
Subjective evaluation:
- System usability (SUS)- Task load (NASA-TLX)- Users’ preferences
Test application includes three scenes
A. Cannavò, F. De Pace, F. Salaroglio, F. Lamberti "A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions",
International Journal of Human-Computer Studies, 2020
LEVER
AG
ING
GR
APH
ICS A
SSETS: INTER
AC
TIVE
APPLIC
ATION
S DEV
ELOPM
ENT.
A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions
154
579
1,227
0
200
400
600
800
1000
1200
1400
VSE LE BGE
Co
mp
leti
on
tim
e [
s]
16
34
85
0
20
40
60
80
100
VSE LE BGE
Nu
mb
er
of
blo
cks
379
859
0
200
400
600
800
1000
1200
VSE LE
Co
mp
leti
on
tim
e [
s]
0.36
3.14
00.5
11.5
22.5
33.5
44.5
5
VSE LE
Nu
mb
er
of
err
ors
System SUS score
VSE 79.11
LE 53.57
System NASA-TLX Ind.
VSE 33.48
LE 59.79
Expert User
Unskilled users
AUGMENTED, MIXED AND VIRTUAL REALITY: A NEW PERSPECTIVE FOR HUMAN-MACHINE INTERACTION
High mental workloadSystems difficult to useEffectiveness of the system
CG assets as tools for providing effective feedback
D. Calandra, F.G. Pratticò, A. Cannavò, L. Micelli, F. Lamberti "Building reconfigurable passive haptic interfaces on demand using off-the-shelf
construction bricks", Proc. 6th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2019), 2019.
AU
GM
ENTED
, MIXED
AN
D V
IRTU
AL R
EALITY: A
NEW
PER
SPECTIV
E FOR
HU
MA
N-M
AC
HIN
E INTER
AC
TION
.Building reconfigurable passive haptic interfaces on demand using off-the-shelf construction bricks
Passive haptics built with
reconfigurable TUIs
Use case: Tasks of an escape room game
A. Cannavò, F.G. Pratticò, G. Ministeri, F. Lamberti ''A Movement Analysis System based on Immersive Virtual Reality and Wearable Technology for Sport
Training‘’, Proc. International Conference on Virtual Reality (ICVR 2018), 2018.
Real-time visual feedback
AU
GM
ENTED
, MIXED
AN
D V
IRTU
AL R
EALITY: A
NEW
PER
SPECTIV
E FOR
HU
MA
N-M
AC
HIN
E INTER
AC
TION
.A movement analysis system based on immersiveVR for sport training
17.3415.01
02468
101214161820
Eucl
idea
n d
ista
nce
[D
eg]
Projected Virtual
2255.09
1595.36
0300600900
1200150018002100240027003000
DTW
co
st
Projected Virtual
Ghost metaphor
Feedback on body
articulation
Overall evaluation
A. Cannavò, F. Lamberti, ''User interaction feedback in a hand controlled interface for robot team tele-operation using wearable augmented reality‘’,
Proc. Smart Tools and Applications in Graphics (STAG), 2017.
Hand tracking input method (Leap Motion)Wearable video-see-trough deviceAR contents include:
- interaction area of the leap motion controller- video stream - command issue- status of the robot
AU
GM
ENTED
, MIXED
AN
D V
IRTU
AL R
EALITY: A
NEW
PER
SPECTIV
E FOR
HU
MA
N-M
AC
HIN
E INTER
AC
TION
.Interaction feedback for robot tele-operation with wearable augmented reality
2.53
3.33
2.53
3.133.40 3.67
2.40
3.13
2.13
3.00
0
1
2
3
4
Sco
re
Learnability* Efficiency* Memorability Error recov.* Satisfaction*
WearableDesktop
Completion time [sec]
WearableDesktop
ARM
CONCLUSIONS
• Design and develop of interfaces to support users in the creation and use of computer graphics assets
• Improvements in both objective and subjective terms
• Allowing unskilled users to approach the Computer Graphics field
• Improvements are observed in different stages of the pipeline and for numerous application domains
THANKS FOR YOUR ATTENTIONQuestions & Answers