Adapting Face Trackers

The DFX Extraction Library does not provide any built-in facial landmark tracking. Face detection and facial landmark tracking is available as a commodity with implementations like Dlib, Visage etc.

MPEG-4 Facial Data Points

Whatever the output of the face tracking engine, it will need to be mapped into standard MPEG-4 Facial Data Points before being inserted into a DFX Frame as a DFX Face structure.

MPEG-4 Facial Data Points

Please see the _dlib2mpeg4 structure in dlib_tracker.py in the dfxdemo as an example of such a mapping for the Dlib face tracker. NuraLogix will be happy to assist you in this process and we already have similar mappings for several common face tracking engines.

Required Landmarks

The DFX Collector has a method called getRequiredPosePointIDs which will return a list of MPEG-4 Facial Data Point names that need to be added to the Face structure for the blood-flow extraction to work correctly. This list will change base on the study configuration data used to initialize the DFX Factory.

However, there are currently 15 core facial landmarks used by the majority of DFX Studies. These landmarks and other details are discussed thoroughly in the next chapter.

DFX Face

A DFX Face structure consists of:

  • id - unique to one person in a frame. The person's name, a GUID or even a counter will suffice
  • faceRect - the bounding rectangle of the face in the frame
  • poseValid - true if the posePoints are valid
  • detected - true if the face was detected in this frame, false if the face-tracking information is cached from an older frame and was used to estimate landmark points on this frame.
  • posePoints - a map of MPEG-4 point names and DFX PosePoints
  • attributes - a map of additional face attributes like yaw, pitch etc.

A DFX PosePoint structure consists of:

  • x - the X location on the frame
  • y - the Y location on the frame
  • z - the Z location on the frame (reserved for future use)
  • valid - true if this point was valid in this frame
  • estimated - false if this point was returned by the face tracker, true if it was estimated by interpolating from points that were returned by the face tracker
  • quality - the tracking quality (or probability) of this point (between 0 and 1)

Face-tracking strategies

If the face tracker that you have selected cannot keep pace with the frame rate of the image source, then you can run the face tracker on a best-effort basis on a background task and use its most recently produced results (within reason.) Please be sure to set the detected and estimated fields correctly when using cached data.

Currently, the DFX solution can better tolerate some inaccuracies in PosePoint locations better than dropped frames. (This may change in the future and a different face tracker strategy may be more appropriate.)