Getting started

This chapter walks through dfxdemo - a simple Python-based demo that demonstrates how to use the DeepAffex™ Library and DeepAffex™ Cloud API.

dfxdemo can extract facial blood-flow from a video file or webcam, send it to the DeepAffex™ Cloud for processing and display the results. It can also be used to display historical results. In this chapter, we will focus on video files.

To begin, please clone the demo from its GitHub repo and follow the instructions in the README.

Understanding dfxdemo

dfxdemo has top-level commands that roughly correspond to the way the DeepAffex™ Cloud API is organized. It uses dfx-apiv2-client-py to communicate with the Cloud.

Device tokens, user tokens and one-time refresh tokens

Authentication on the DeepAffex™ Cloud API uses JSON Web Tokens. Each token contains information pertaining to the current access request. On the Cloud, a token is mapped against an internal policy manager that specifies what access levels it has. The token policies control access to the various parts of the API. There are three types of tokens:

A Device Token represents a single device. It provides access to endpoints that allow for device-specific features e.g., making measurements. To obtain a Device Token for the device in use, register your organization license on the DeepAffex™ Cloud. A pair of tokens (a Device Token and a One-Time Refresh Token) is returned.

dfxdemo org register <your_license_key>

A User Token provides access to most non-measurement related endpoints. To obtain a User Token, login as a user. A pair of tokens (a User Token and a One-Time Refresh Token) is returned.

dfxdemo user login <email> <password>

A One-Time Refresh Token (OTRT) is a single-use token that is used to refresh User and Device tokens. Refreshing tokens at appropriate times lets devices can "roam" properly and improves token security.

The flowchart below shows the steps needed to obtain and renew a token.

By default, dfxdemo stores tokens in a file called config.json. In a production application, you will need to manage all tokens securely.

Note: All the commands below, use the tokens obtained above as illustrated in the demo code.

Points and Studies

The DeepAffex™ Cloud organizes around the concept of Points and Studies.

A DeepAffex Point is a biosignal of interest that can be extracted, computed or estimated from facial blood-flow data and that is assigned a unique name on the DeepAffex™ Cloud. (e.g., HR_BPM is the Point for pulse rate.)

A DeepAffex Study is a collection of DFX Points that are computed in one measurement. A Study is based on a Study Template which determines which Points it contains. You can list the Studies available to you, retrieve a Study's details and select one for use in measurements.

dfxdemo studies list
dfxdemo study get <study_id>
dfxdemo study select <study_id>

Making a measurement

The process of extracting facial blood-flow from a sequence of images and sending it to the DeepAffex™ Cloud for processing is called making a measurement.

dfxdemo uses OpenCV to read individual frames from a video, MediaPipe Face Mesh to track facial landmark features in each frame and libdfxpython (DFX Extraction Library's Python bindings) to extract facial blood-flow. To make a measurement from a video using the selected study:

dfxdemo measure make /path/to/video_file

The facial blood-flow data from a video is sent to the DeepAffex™ Cloud in fixed duration chunks (5 seconds by default) over a WebSocket. As the measurement progresses, accumulated results are returned over the same WebSocket and displayed. When the last chunk is received by the DeepAffex™ Cloud, the overall results are computed and returned.

The flowchart below shows the steps needed to make a measurement from a video.

You can also make a measurement using a webcam. This will enable the measurement constraints which guide the user to better position themselves.

Measurement results and history

Historical measurement results associated with a user and their details can also be retrieved using dfxdemo if your legal agreement with Nuralogix™ allows saving results:

dfxdemo measurements list
dfxdemo measure get <measurement_id>

These results are also available on the DeepAffex™ Dashboard which includes a sophisticated graphical display.

Next steps

In the next chapter, we will discuss the DeepAffex™ Cloud API in more detail.