We recently got our hands on an iPhone Pro, and one of the coolest things about this magnificent piece of engineering is the suite of incredibly sophisticated sensors packed into it.
We’ve been exploring one of the apps we’ve been working on: a hand-tracking system designed to let you interact with a website specifically built for it.
This is actually something we’ve been exploring for quite a while now. But, just for fun, we decided to take the experiment to a whole new level.
Hand tracking is nothing new. There are already plenty of really good models that can take a video stream from a webcam and provide fairly accurate hand tracking. We use the hand-tracking capabilities provided by iOS’s Vision framework as our starting point.
But we wanted to improve those coordinates for our particular use case.
First, the raw coordinates provided by the Vision framework are only 2D. We wanted a better understanding of the hand’s position and articulation in 3D space. Second, we wanted the system to work in complete darkness. An RGB video stream simply wouldn’t cut it.
So, we combined the hand-tracking data from Vision with the depth information from the iPhone’s TrueDepth sensor.
The process is fairly simple. Instead of blindly trusting the coordinates provided by Vision, we look around each detected point in the depth map to find the actual pixels belonging to the finger. We then apply an adaptive filter to smooth out tiny movements when the hand is still, while remaining responsive when it moves quickly. This gives us much more stable tracking without introducing noticeable lag.
For the network connection, the app can communicate through WebSockets, although we started connecting it via USB whenever possible to keep latency as low as possible.
This is mostly an experiment, an excuse to play with the sensors, try out a different way of interacting with the web, and see what we could make with it. Right now, we’re exploring how this kind of technology could be used to control interfaces and visualizations we’ve already built, and how these new interaction possibilities could change the way we think about them.
It might never become a finished product, and that’s not really the point. Sometimes the most interesting things come from simply experimenting with a new piece of technology, understanding what it can do, and pushing it a little further than its intended use.
For us, playing with these technologies is also a way of opening up new ideas. You start with a simple experiment, discover a new possibility, and suddenly you’re thinking about a completely different way of building something.
So this is just us experimenting, learning, and having fun with what’s possible.