Constraints

The DeepAffex Cloud will occasionally fail to return a result successfully. This usually occurs when the Signal-to-Noise Ratio (SNR) of the extracted blood-flow signal isn't high enough or when there are other issues with it.

While the user cannot do much about this for blood-flow extracted from videos, if a measurement is being made live using a camera, your application can provide actionable feedback to the user based on the Constraints system in the Extraction Library which may increase the chances of a successful result. You can also cancel a failing measurement early without waiting for a round trip over the internet.

Constraints demo using dfxdemo

dfxdemo, introduced earlier, has the ability to make measurements using your computer's webcam and provide actionable feedback using the Constraints system while you do so. Assuming you have completed all the prerequisite steps needed to make measurements (i.e. registered your license, logged in as a user and selected a study,) you can run the following command:

dfxdemo measure make_camera

Note: If your computer has multiple webcams and you want to select a different one, you can use the --camera <index> argument. You can pass --help to get a full list of options.

Once the demo starts, you may see multiple feedback messages in red, asking you, for example, to look straight at the camera or to hold still. Once all the constraints are met, you can press s to start the measurement. You may also see warnings during the measurement and if you violate constraints too much during the measurement, it may fail.

The flowchart below shows the steps needed to make a measurement from a webcam with constraints enabled.

Checking constraints

Programmatically, you need to call checkConstraints on every Frame - typically, before starting a collection.

dfx_frame = collector.createFrame(dfx_video_frame)
for dfx_face in dfx_faces:
    dfx_frame.addFace(face)
result, details = collector.checkConstraints(dfx_frame)

The result return value is the aggregate result of the constraint check and can be either GOOD, WARN or ERROR.

• GOOD means that the constraints are all met and collection can proceed. If it hasn't started yet you could choose to automatically start it or allow the user to start it using an input. (Recall that a measurement is started by calling startCollection)

• WARN means that constraints are currently violated but the user has still has a chance to change conditions and change things back to GOOD and perhaps still have a successful measurement. Feedback should be provided to the user.

• ERROR means that the collection has failed because of too many constraint violations. Obviously, ERROR can only be returned after the collection has started. On an ERROR, the application is expected to call cancelCollection to terminate the ongoing measurement and resetCollection to reset the internal collector state.

On a result of WARN and ERROR the details return value contains a map of reason strings and result values. The reason strings correspond to each possible constraint that is violated with a WARN OR ERROR.

The possible reason strings and their meanings are listed below:

Clearly, feedback to the user can and should be customized for your specific application. The Image and LowFps reasons in particular can be dealt with using application specific camera settings and controls. In addition to the above, you may get ImageEmpty (an empty image was passed) and CameraMovement (capture device was moving during capture) which will not be discussed in this guide.

Configuring constraints

The constraints that will be checked by the DFX Collector are configurable by the developer. You can query the currently set constraints by calling collector.getConstraintsConfig("json") and modify the values by calling collector.setConstraintsConfig("json", json_string) - the return value and the modified values are expected to be JSON strings which will contain a simple JSON dictionary. Typically, you would get the existing constraints, modify what you need to and set them again.

constraints_cfg_str = collector.getConstraintsConfig("json")
# Parse the json, modify the values as needed
modified_constraints_cfg_str = parse_and_modify(constraints_cfg_str)
collector.setConstraintsConfig("json", modified_constraints_cfg_str)

The following constraints can be adjusted:

Minimum frame rate

The capture frame rate in frames/second has to exceed minimumFps (default: 27). This check can be disabled using checkMinFps.

Face position and presence

The face has to be present within a target rectangle within the frame. (The rectangle is specified as a percent of the frame size using boxCenterX_pct, boxCenterY_pct, boxWidth_pct and boxHeight_pct) (default: 50, 50, 50, 99). These defaults are optimized for the landscape orientation. This check can be disabled using checkCentered.

Face distance from camera

The face has to be close enough to the camera. Minimum inter-pupillary distance in pixels (minInterPupilDist_px) is used a proxy for distance (default: 60.0). This check can be disable using checkDistance. (This check assumes that you haven't disabled the face rotation constraint described below.)

Image brightness and backlighting

The rectangle containing the face has to be brighter than threshDark (default: 48) and darker than threshBright. (default: 225). This check can be disabled using checkLighting.

If the region around the face, created by growing the face rectangle by backLightSearchMult (default: 2.0) and cropping at the bottom of the face rectangle, has more than backLightMaxPixels_pct percentage (default: 30%) of its pixels brighter than backLightThresh (default: 240) then the backlight constraint is violated. This check can be disabled using checkBackLight.

Face rotation

The user has to look at the camera. The maximum allowed left-right face rotation is maxFaceRotLR_deg degrees (default: 12.5) and max allowed up-down face rotation is maxFaceRotUD_deg degrees. (default: 25). This check can be disabled using checkFaceDirection.

Face movement

The nose position is tracked in time (for the last movementWindow_ms milliseconds) (default: 2000) and if it exceeds maxMovement_mm millimeters (default: 10), then a warning is issued. This check can be disabled using checkMovement.

Eyebrow movement

The perpendicular distance from the eyebrow midpoint to the line joining the two lateral canthuses is calculated in mm for both eyebrows and tracked. If the range of the larger tracked value exceeds maxEyebrowMovement_mm millimeters (default: 3), then a warning is issued. This check can be disabled using checkEyebrowMovement

NOTE: The face movement and eyebrow movement constraints are only checked if a measurement is active.

We recommend using your judgement to modify and/or disable constraints to get the best results for your particular use case. Testing with a wide variety of users and in a wide range of environments is very helpful!