Consider a device placed across the living room, designed with a display for output. How can one interact with it from a distance? Rejecting voice control, our team focused on air gestures, using a device-mounted camera for detection. The project aimed to create a prototype that enables this interaction method, solving the challenge of engaging with a device that's out of physical reach.
The "AirTap" feature, a novel interaction method, was designed for system navigation, allowing users to execute commands through air taps and gestures for dials and switches. This system also integrates face and gaze tracking to determine user engagement intentions. Developed in one month, the AirTap and knob gesture recognizer were enabled by custom CoreML models, trained on 8,000 and 10,000 examples, respectively, utilizing a unique data collection approach developed specifically for this project. Face tracking employed Apple's internal FaceKit framework, enhancing the system's understanding of user intention.