Ultra.
As the VTubing industry booms, virtual avatars are becoming more expressive and dynamic. Unfortunately, not everyone has access to a team of animators or a full-body motion capture suit. The struggle for solo VTubers (those who aren’t working for an agency), as well as hobbyists to keep up with their partnered counterparts is real. How can you make your avatar look more interesting and appealing without breaking the bank? The answer is hand tracking.
Hand and body tracking has been around for a while but hasn’t caught on with the VTuber community until recently. Even now, hand tracking isn’t a requirement, with many VTubers being perfectly satisfied from the expressiveness they can achieve through face tracking. For those who do more than just gaming on stream (singing, dancing, chatting, etc.), hand tracking can be a great way to bring your avatar to life.
Requirements for VTuber Hand Tracking
To be able to break out those jazz hands on stream, you’ll need a few things.
- A 3D or 2D avatar with arms and hands that have been rigged to move
- A webcam
- Software that supports hand tracking
Or
A physical device (Leap Motion or a VR setup)
If you’re planning to purchase or commission an avatar, you’ll want to check whether hand and finger rigging is available. While certain avatar features like blinking and mouth movement come standard, hand movement does not. If you’re making your own avatar, you’ll need to go through the extra steps of rigging the arms, hands, and fingers.
Types of Hand Tracking Available
Once you have a webcam and an avatar rigged for arm movement, all you need is either the right software or a physical device. Let’s talk about a few of the different options out there.
VTube Studio
The gold standard for 2D avatars, VTube Studio announced hand tracking capabilities just last year. We covered VTube Studio in our article about the best free software for VTubers. VTube Studio is incredibly easy to use and comes with a lot of free assets like backgrounds, test models, and accessories to stick on your avatar. For $15 USD, you can remove the watermark and support the development of VTube Studio at the same time.
To enable hand tracking, just go into your camera settings for VTube Studio, then change the tracking type from “Face tracking” to “Face and hand tracking.” Note: You will need to select the “Camera OFF” button when you update camera settings, otherwise they will remain grayed out. You can check VTube Studio’s wiki for more information on setting up hand tracking.
Kalidoface 3D
The first of its kind, Kalidoface 3D is a browser-based, free-to-use VTuber face and body tracking program (if you’re familiar with Talk Studio or Video Editor, you’ll know we’re big fans of browser-based tools!). Kalidoface 3D, as well as Kalidoface (a program for 2D avatars with just face tracking at the moment), were both created by a single developer. Rich is working hard to make body tracking for VTubers more accessible, so consider supporting him on Ko-fi if you give Kalidoface 3D a try.
As mentioned above, body tracking is only available for 3D avatars. There are several test models you can try or you can upload your own model as long as it’s a VRM file. You’ll want to give Kalidoface 3D a try before you do any streaming to ensure your computer can handle the load. While the tracking capabilities aren’t as smooth as Leap Motion or VR (both of which we’ll talk about shortly), Kalidoface is a great tool to dip your toes…er, fingers into the waters of VTuber hand tracking (not to mention, it’s free!).
Leap Motion
When most VTubers think of hand tracking, Leap Motion is likely the first thing to come to mind. The Leap Motion Controller is a small camera made specifically for hand tracking. It is designed to be placed on your desk (or clipped onto your shirt via an aftermarket accessory) and plugged into your computer. It retails for around $100 to $200 USD (depending on availability) and works beautifully with 3D avatars in VSeeFace or Luppet.
We go into detail about streaming with VSeeFace in our article on how to set up a 3D avatar. If you’re interested to see what an avatar created in VRoid Studio looks like when using the Leap Motion Controller, check out the gif at the top of this post, as well as the gif above. As you can see, hand tracking with the small-but-mighty Leap Motion Controller is smooth and sophisticated. Though not the cheapest option on our list, for certain VTubers, this is a must-have accessory and well worth the investment.
VR Systems
While any of the aforementioned methods should be sufficient for the everyday VTuber, some folks will require a more advanced setup. The final (and most expensive) option on our list is to use a VR system such as the Meta Quest 2, HTC Vive, or Oculus Quest 2 in conjunction with a software called VMC (Virtual Motion Capture). If you already own a VR system, great! You just need to download the VMC software, which is free (but a donation is recommended to support the developer, sh_akira).
If you’re interested in playing VR games on or off stream, a personal VR system could be a good choice for you and will run a few hundred dollars (vs professional systems, which start around $1000 USD and go up from there). Keep in mind that most VTubers only wear the VR goggles when they are playing a game and usually rest them on their head when interacting with the chat (which can be heavy, so do take care not to injure your neck). If you don’t have a lot of interest in VR games, stick with one of the more affordable options we mentioned, as purchasing a VR system specifically for hand tracking would be overkill.
As VTubing continues to experience exponential growth, it is likely that more software and avatar capabilities will become available over time. If you decide to experiment with hand tracking, definitely play with the free software options first to make sure they’re compatible with your model. You might discover that you rarely use your hands on stream anyway and therefore don’t need to shell out for any hand tracking hardware. We hope you enjoyed reading about all of the different hand tracking options out there.