The Vision framework provides out-of-the-box machine learning tools for image analysis, including object detection, image classification, and tracking faces.
Vision processes hand poses by identifying 21 specific landmarks, including the wrist, palm, and four points on each finger and thumb.
Hand pose recognition can fail due to common real-world issues like partial occlusion, hands near screen edges, wearing gloves, or confusing hands with feet.
The sample application is built around three main components: a CameraView for display, a ViewController for control logic, and a HandGestureProcessor for analyzing gestures.
A live demonstration shows how to use the tips of the thumb and index finger to create a pinch gesture that draws lines on the iPhone screen.
The implementation relies on key classes like CameraViewController, VNdetectHumanHandPoseRequest for analysis, and UIBezierPath for drawing the visual feedback.
The VNImageRequestHandler processes the camera buffer and returns observations, from which you can extract the coordinates of specific finger joints like the thumb tip.
A custom processor manages gesture states like 'pinched' or 'apart' by calculating the distance between finger landmarks and using a counter for stability.
The same principles used for hand pose can be applied to full-body pose detection, which tracks major body joints like shoulders, eyes, and ears.
Pose detection technology can be used to build applications that understand sign language, analyze human interaction in images, or create new forms of user input.
36:34Milan Todorovic
48:21Nermin Sehic
28:54MIlan Todorović
27:23Nico Martin
48:38Flo Pachinger
42:50Milica Aleksic
28:48Mario Petricevic
34:34Daniel Madalitso Phiri
Jobs that call for the skills explored in this talk.
Verity AG
ANavS GmbH
München, Germany