nevronmotion - lightwave 3dstatic.lightwave3d.com/downloads/installers/nevronmotion_1/... ·...

Manualv 1.0

NevronMotion™

© Copyright 2013 LightWave 3D Group, a division of NewTek, Inc. All rights reserved.

NevronMotion v1.0

Contents

IntroductionWelcome to NevronMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

InstallationInstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Kinect installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Editing the Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

RetargetingExample: Retargeting Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Example: Retargeting Motion Capture to a LightWave character . . . . . . . . . . . 12

KinectKinect Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Settings Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Example: Puppeteering the Alien Emissary’s head . . . . . . . . . . . . . . . . . . . . . . . . . 19

Example: Full-body Motion Capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Contents


NevronMotion v1.0

Introduction

Welcome to NevronMotion

Welcome to NevronMotion™ 1 .0!NevronMotion is a motion retargeting plugin for LightWave 3D 11 .6+ with support for the Microsoft Kinect camera . NevronMotion allows you to load common motion capture files in FBX or BVH format and easily retarget the motion onto your character directly in LightWave Layout . You can quickly adjust any of the character rig joints in areas like the shoulders or arms to compensate for any joint placement inaccuracies and bake out final animation directly to your character rig .

NevronMotion comes with rig retargeting presets for FBX and BVH files and allows you to fully customize your rig retargeting and save your own retargeting presets that you can load later to use with custom motion capture libraries . You set up the rig retargeting mapping once, save the preset and then you can reuse that over and over .

NevronMotion includes a rig preset that works directly with the Microsoft Kinect camera to allow you to puppet a character rig in real time and even record your real time Kinect motion capture performances to save to your library for later use .

NevronMotion greatly simplifies the motion capture workflow in LightWave 3D and allows you to get character motions or create your own motions with the Kinect camera to get your characters moving quickly and easily .


NevronMotion v1.0

Installation

NevronMotion will only work with LightWave 11.6 and above.

Installation

The first version of NevronMotion is supplied as a zip archive . Copy the zip’s NevronMotion folder to <username>\.NewTek\LightWave\11.6\Plugins folder on Windows or ~/Library/Application Support/NewTek/LightWave/11.6/ on OS X or your normal plugins folder if you have a custom setup .

If you have a custom setup, the Genoma folder in the NevronMotion directory in the zip contains two subdirectories - plugins and rigs . The CreateNevronMotionCapture.lsc file in the plugins directory should go in your LightWave \support\plugins\genoma directory . From the rigs folder Nevron_Genoma_Rig.lwo and its accompanying .txt file should go in the \support\genoma\rigs\Complex_Rigs\Bipeds folder .

Of the remaining files, Kinect_Skeleton_SKL0.lws and SKL1.lws are scene files for use with the Microsoft Kinect . They should be put with the rest of your LightWave content, and EULA.doc and ReadMe.rtf, are a licensing agreement and a ReadMe .

NevronMotion is a license-protected plug-in for LightWave 11 .6+ . NevronMotion licenses are added to LightWave the same way that LightWave licenses are added by dropping an LWK file downloaded from your account on LightWave3d.com onto your Layout window . If you already have an LWK (LightWave Key) file, you may drop it onto the Layout window .

If you are not a native English speaker and wish to change the localization for NevronMotion, opening the Hub and going to the Options section will reveal a language choice dropdown.

Kinect installation

For Kinect for Windows download KinectRuntime-v1 .7-Setup .exe from your account on LightWave3d.com and install it .


NevronMotion v1.0

To access the NevronMotion retargeting plugin, you can either access the Master Plugins panel with the keyboard shortcut Ctrl+q or through the Windows > Master Plugins... menu item . Alternatively you can create a button for NevronMotion retargeting in LightWave’s menus .

Editing the Menus

To place NevronMotion in your menus, follow these instructions:

1) Go to the Edit > Edit Menu Layout panel (Alt+F10) .

2) Click Search and type nevron, case is not important .

3) Once NevronMotion has been found (the second entry, not CreateNevronMotionCapture), select where you want it to be placed in your menus in the right list, Setup is a good place . Once done, hit the Add button for NevronMotion to be available there from now on .

Installation


NevronMotion v1.0

Retargeting

Retargeting

One of the main uses for NevronMotion is retargeting animation . The animation can be motion captured, but doesn’t have to be - you can in fact retarget any kind of animation, classic hand-animated character motion for instance, to the NevronMotion biped rig . A simple example can be presented using the NevronMotion rig as it stands .

Example: Retargeting Animation

1) In Modeler, go to the Setup tab and add a Nevron Genoma Rig from the Genoma Presets

window (in Complex Rigs/Bipeds) . Save it and send it to Layout . If you wish to add limbs or a tail, use the orange bones shown as the base connector .


NevronMotion v1.0

Retargeting

2) In Layout, switch the viewport display mode to Bounding Box to speed up rig creation and then from the Setup tab use Genoma > Create Rig .

3) Now add a Null (Ctrl+n) to the scene . We are going to set up a simple arm to show retargeting . Add three Bones to it (=) and position it to roughly duplicate the position and


NevronMotion v1.0

Retargeting

rotation of your Nevron rig arm .

4) Add some animation to your arm, perhaps rotating the arm to wave and bending at the elbow and wrist . You don’t need anything fancy .


NevronMotion v1.0

Retargeting

5) Open the NevronMotion retargeting window . In it click the LeftArm line, select the appropriate bone in the viewport and hit the GetSel button in the NevronMotion window . Do the same for the LeftForeArm and LeftHand with their respective bones . Untick the Retarget entries for the LeftArmRoll and LeftForeArmRoll . We want to continue to use the NevronMotion rig for the motions of these bones .

6) Now hit Retarget . You will be presented with a message saying the scene isn’t ready for retargeting (shown in the Step 5 picture), but this is because we have left all the other bones with Retarget active, but we don’t have the bones to retarget . You can happily say Yes to the requester, but if you don’t even want to see it in the first place, uncheck all of the bones other than the three you are retargeting (LeftArm, LeftForeArm and LeftHand) . As you scrub the timeline you should now see the Nevron rig mimicking the motion of the bone chain you animated .

Retargeting motion capture files to a character is explained on page 12


NevronMotion v1.0

Retargeting

Overview

The retargeting portion of NevronMotion is accessible through Setup > NevronMotion if you followed the Editing Menus tutorial on page 3 or within the Layout Master Plugins panel . This can be accessed with the Ctrl+q keyboard shortcut or through the Windows > Master Plugins... command .

Double click the entry you added to your Master Plugins window and the NevronMotion Retarget window looks like this:


NevronMotion v1.0

Retargeting

1) This list represents the bones in the target Rig (the Genoma NevronMotion rig) that will be driven by your source items .

2) These are the rig parts of the imported motion capture file . You want this list to match the names of the motion capture rig . You can double-click on these names to edit them . Any bones not present will be prefixed with an ! .

3) If you have a bone or a null from the motion capture skeleton selected in Layout, you can assign it to a bone in the Target rig by hitting Get Selected in this column .

4) Select here to retarget the specific bone in your rig to the target . If there is no equivalent bone in your rig, retargeting will not be performed for that item and the item will be prefixed with an exclamation mark .

5) Turn on or off IK solver for arms or legs . There are only certain bones in a target rig that support IK, notably hands and feet that can be directed with goals . The Off, On and Match Goal buttons activate and deactivate IK for the bones in the rig that support it .

• Off - Does not use IK but exactly matches to the original rig .

• On - This uses IK to control specific bones meaning that they won’t just blindly follow the animation of the original rig but will use LightWave’s IK solver to better position feet


NevronMotion v1.0

Retargeting

and hands . This setting keeps the goal object with the NevronMotion rig .

• Match Goal - As above but the IK Goal is positioned to exactly match the ankle of the motion capture file rig .

6) You can add a Prefix or Suffix to your rig’s names so they match the motion capture rig . You can also remove the prefix or suffix and replace text you’ve already added .

7) On this row there are four buttons:

• The Presets dropdown contains four options by default, more can be added .

• BVH - Standard naming for a BVH rig

• FBX - Standard naming for an FBX rig

• Kinect Player 0 - Rig names for the first Kinect character

• Kinect Player 1 - Rig names for a secondary Kinect character

• Load... - For loading a preset temporarily from elsewhere, perhaps one shared by another user . Use the Store... button to add presets permanently to the Preset list .

• The Store... button saves the current Rig as a Preset that will be added to the Presets dropdown . It is persistent between LightWave sessions .

• Remove is a dropdown that shows a list of user-added Presets . You can select from within this list to permanently remove added presets from the Presets dropdown .

8) Rig Bones - this button hides your NevronMotion rig’s bones so that you can concentrate on the rig you are retargeting . The NevronMotion rig’s controls will still be visible .

9) The Retarget and Untarget buttons globally activate and deactivate retargeting for the rig . If you see that your target character’s bones are not following the source target in a way you like, you can untarget, fix rotations so the source and target bones’ initial positions better matched, and retarget again to test the result .

10) Bake... - Bakes the keyframes from the Target Rig to the Source Rig .

11) The ? button gives license and version information for NevronMotion .

Added presets are stored in the <user>\.NewTek\LightWave\11.6\NevronMotionPresets folder on Windows and the ~/Library/Application Support/NewTek/LightWave/11.6/NevronMotionPresets for Mac. They can be copied from here to share with other users for use through the (Load...) button in the Presets dropdown.


NevronMotion v1.0

Retargeting

The NevronMotion rig. Hip position and scale are the most important things to match when retargeting animation.


NevronMotion v1.0

Retargeting

Example: Retargeting Motion Capture to a LightWave character

You have your character rigged with the Genoma Nevron rig and you’ve checked how it works . Now it’s time to apply some motion capture to it . For this example we will use one of many freely available FBX motion capture files - slip-down.fbx . You can find this file on many websites .

1) We’ve converted the Modeler Genoma Nevron rig into bones in Layout . Refer to the Genoma documentation for LightWave 11 .6 if you need a refresher (starting page 236) .


NevronMotion v1.0

Retargeting

2) Now we’ll use Load from Scene to bring in our motion capture file . The defaults should work okay, but there may be floor objects you don’t want to import .

3) The FBX file we have imported has lots of nulls that are getting in the way of our view, so we can hide them by opening the Scene Editor, selecting the object called Skeleton_v75:Character_Ctrl:Reference (2), clicking the RMB and choosing the Selection > Select Child Items (Recursive) option . This will select all the seemingly extraneous nulls . If you click in the OpenGL column now you can set them all to Hidden .


NevronMotion v1.0

Retargeting

4) Next we need to make sure our motion capture is more easily retargeted . To do so, select any one bone on the mocap skeleton and use Items > Select > All > Select all Bones in Current Object . Make sure you are in Rotate mode and enter 0, 0, 0 in the numeric panel at the bottom left of the Layout window - even if it’s already there . Your mocap skeleton will assume the traditional T-pose .

5) Now there’s some work to do . First select the root null for the mocap skeleton and move it to the same place as your character . Now, you need to be in Object mode and select the small green circles at each joint on your character to make them align better with the mocap skeleton . You don’t need to be ultra-precise with this, but as a minimum make sure the hips are on the same level and the arms match the mocap skeleton - you will probably need to rotate the hand to match and if your mocap has finger movement you will want to make sure the fingers match . The green controllers are symmetrically controlled, so you only need perform this matching for one side of your character .


NevronMotion v1.0

Retargeting

6) The next thing is to make sure the bone names in your rig match those in the mocap rig . We can see that the mocap rig has a prefix of Skeleton_v75: on all the bones, so now we’ll open the NevronMotion window and convert all our bone names . Type that name into the field in the NevronMotion window and choose Add Prefix from the dropdown . Once you’ve done that, just hit the Retarget button to match your rig to the mocap skeleton .

7) As you can see, we could certainly make our skeletal matching better, but we have a start . If you want to improve your matching, use Untarget in the NevronMotion window, adjust and then click Retarget again . Also, if you wish to add animation on top of the retargeting you can do so by animating the magenta controls .


NevronMotion v1.0

Kinect

Kinect Support

The Kinect For Windows device support is available via the Layout Device Manager panel . Please see the LightWave documentation regarding Virtual Studio .

Rename one of your Kinect devices to Kinect_A in this panel by double clicking on its entry for use with retargeting example content provided.

To use a Kinect device, you will also need to install the KinectRuntime driver (KinectRuntime-v1.7-Setup.exe) . This is available via Microsoft, but will be provided with this product eventually . With this driver, you must use a Kinect For Windows device and not the XBox360 Kinect device . (The Kinect For Windows SDK does allow use of the XBox360 Kinect device; but you must obtain that device on your own .)

The Kinect camera is not supported under Macintosh OSX

Settings Panel

The Settings panel for your Kinect device is where you can see what your Kinect device sees .


NevronMotion v1.0

Kinect

1) Tilt - This setting controls at what angle your Kinect camera is pointing .

2) Color Camera - This button is not available on an XBox Kinect and opens the Options panel shown below .

3) IR - This shows you what the Kinect’s infrared camera sees . Skeleton tracking is always performed using the IR image, but facial tracking works best with the color camera image and not the IR image .

4) Mirror - Reverses the way the shot is displayed on the screen in the X-axis .

5) Near Mode - This allows tracking for slightly closer subjects at the expense of losing some accuracy for distant subjects .

6) Audio - This enables the following data tracks: Audio_Peak, Audio_Source_Angle, Audio_Source_Confidence, Audio_Beam_Angle .

• Audio_Peak displays values between 0 .0 and 1 .0

• Audio_Source_Angle is the horizontal angle relative to straight ahead (relative to the sensor) that it thinks the sound is coming from .

• Audio_Source_Confidence is how sure the Kinect is that the source angle is correct . (0 .0 to 1 .0) .

• Audio_Beam_Angle is like the source angle except that it is where the mic-array is looking (listening) for the source sounds . It has 10° increments but is shown as radians in the track data .

7) Skeletons - Enables the tracking of skeleton-related data . Skeletons must be enabled before any facial recognition can occur . There can be up to six separately-tracked persons, but only two of them can provide full skeletons . The others will only have general position . Each person being tracked will show as a unique tint in the image . The skeletons are numbered 0 to 5 from closest to the sensor to furthest . When the person being tracked is known to be seated, the legs and spine portions of the skeleton are not trackable . The Seated suboption avoids looking for these lower extremities .

8) Smoothing Preset - This applies some preset settings for various skeleton tracking options . Typically, the skeleton tracking provides some very erratic results depending on the person’s shape and movements . Smoothing helps control this better . There are presets that result in no adjustment and presets the result in very smooth results . The more smoothing that is employed, the greater the lag of the results .

• Smoothing - Increasing the smoothing parameter value leads to more highly-smoothed skeleton position values being returned .

• Correction - Lower values are slower to correct towards the raw data and appear smoother, while higher values will correct toward the raw data more quickly .

• Prediction - The number of frames to predict into the future . Values must be greater than or equal to zero . Values greater than 0 .5 will likely lead to overshooting when moving quickly . This effect can be damped by using small values of Maximum Deviation Radius .


NevronMotion v1.0

Kinect

• Jitter Radius - The radius in meters for jitter reduction . Any jitter beyond this radius is clamped to the radius .

• Maximum Deviation Radius - The maximum radius in meters that filtered positions are allowed to deviate from raw data .

9) Faces - Allow facial tracking . This must be used in conjunction with Skeleton tracking . If a person is not currently being tracked as a skeleton, their face will not be tracked either . The Smoothing suboption averages out recent face tracking values for a more fluid movement . Reset Faces - Face tracking continually refines its knowledge of the faces it sees . If the initial face is not quite right, this refinement may just be too far off from reality . This button resets the face parameters and forces a new face to be tracked (hopefully a better one this time) .

The face tracking appears in the live images as a wireframe mask with a white box outlining the portions of the image that seem to contain the real facial features . When a face is tracked, there are several data tracks available for a status, pose, animation units, and shape units .

Kinect for Windows has this modal window when clicking on the Color Camera button (2) above . If you have an XBox Kinect, this button is not present .

In this window, adjustments can be made to what the color camera sees when it is not in IR (infrared) mode . The Auto White Balance toggle overrides the White Balance and Hue adjustments . The Auto Exposure toggle overrides the Exposure, Frame Interval, and Gain adjustments . Currently, the Power Line Frequency setting appears non-functional and the Backlight Compensation dropdown does not appear to have any effect . The Default button is used to reset the settings to their defaults .


NevronMotion v1.0

Kinect

Example: Puppeteering the Alien Emissary’s head

1) Load the Alien_Head.lws scene in the Nevron content .

2) Open the Virtual Studio Device Manager window .

3) Make sure you don’t skip this next step . You’ll only have to do it once but you need to


NevronMotion v1.0

Kinect

rename your Kinect to work with the examples . Double-clicking on the name of the device will present a window to rename it in . Simply select everything after Kinect_ and replace it with A . Kinect for Windows devices have 16 0 following the underscore, XBox360 Kinects have a 16-digit hexadecimal serial number . Clicking the Get Signature button will restore the original number .

4) Next, click on the Settings button next to your Kinect device in the Devices Manager window . This will open a new window containing a moving image of you and the settings covered in the Settings Panel section . If the facial overlay isn’t present to start with, make sure your skeleton is being seen . You may need to adjust your position with regard to the Kinect, moving towards it and away from it until it captures your skeleton first, then it will capture your face .


NevronMotion v1.0

Kinect

Example: Full-body Motion Capture

Using the same Bounce scene as with our retargeting example, nothing could be easier than retargeting Kinect to your character .

1) Load the Kinect_Skeleton_SKL0.lws scene . Use Load from Scene to bring in \The_Bounce\Scenes\Bounce_NevronMotion_Genoma_Rig.lws . Bring in the light from the Bounce scene because it is referred to by the CgFX shader .

2) Arrange your character’s skeleton to closely match that of the Kinect skeleton . You can scale the whole Kinect skeleton (using the Master_SKL_Global_0 item) or parts of it to


NevronMotion v1.0

Kinect

better match Bounce’s proportions . When adjusting the Genoma rig, only change the green controller objects at each joint, not the bones themselves . The most important thing to match is hip placement and orientation .

3) When the skeletons are close enough, open the NevronMotion window . Make sure you are in the Preset Kinect Player 0 and hit Retarget . After a pause you should see all the green controllers on the Bounce character turn into magenta boxes .

4) Open the Virtual Studio and Virtual Studio Devices Manager . In the Devices Manager make sure your Kinect is Enabled and that Virtual Studio has Live!, Allow Rec and Allow Play enabled .


NevronMotion v1.0

Kinect

Open the Kinect Settings window in the Devices Manager so you can see yourself and test a few motions .

5) The last step is to hit the Active button when you are ready . If you are not visibly being tracked in the Virtual Studio Device Manager Kinect settings panel, your Bounce will crumple . As soon as you step within range of the Kinect’s sensors Bounce will unfold himself and be subject to your every whim .

6) Now we need to be able to use our Kinect-controlled character in a scene . It’s great that we can move him around, but only useful if we can record what he’s doing .


NevronMotion v1.0

This is what Virtual Studio is for . There is full explanation of functionality in the main LightWave 11 .6 documentation but for now activate all these buttons in the Virtual Studio window: Active, Live!, Allow Rec, Allow Play, set the Punch In to 60 and click 1-shot . Then hit the Play button in Layout - you have two seconds (depending on your FPS setting in Layout) to get into position to start your performance .

Kinect

nevronmotion - lightwave 3dstatic.lightwave3d.com/downloads/installers/nevronmotion_1/... ·...

Documents