Measurements

The process of extracting facial blood-flow from a sequence of images and sending it to the DeepAffex Cloud for processing is called making a measurement. The results of this process is called Measurement.

Making a measurement

Making a measurement involves three (or two) different endpoints - Measurement.Create, Measurement.AddData and, if real-time results are desired, Measurements.Subscribe.

Creating a measurement

Measurement.Create is the first endpoint that you must call - it will prepare the DeepAffex Cloud for receiving data and return a Measurement ID.

Subscribing to results

In order to receive real-time results, you will need to touch the Measurements.Subscribe endpoint. This is WebSocket-only endpoint. You must pass a unique Request ID to it - the results will be returned with the same Request ID on the WebSocket as they become available on the Cloud.

Measurement Questionnaire

Measurement questionnaires are used to enhance the accuracy of the measurements. To learn more about this, please refer to Measurement Questionnaire section.

Adding data

Once you have subscribed, you can send data to the Cloud using the Measurements.AddData endpoint. The bytearray containing the facial blood-flow data that is sent to the Cloud (a "chunk"), is produced by the DeepAffex Extraction Library. For both the REST and the WebSocket transports, this array needs to be Base64 encoded. In the legacy system, WebSockets connections used the DeepAffex Protobufs for this encoding.

Each chunk also needs an action associated with it which tells the Cloud how to process it. The actions are summarized below:

ActionDescription
FIRST::PROCESSStart the measurement. Process this chunk and return intermediate results.
FIRST::IGNOREStart the measurement. Process this chunk but don’t return intermediate results.
CHUNK::PROCESSProcess this chunk and return intermediate results.
CHUNK::IGNOREProcess this chunk but don’t return intermediate results.
LAST::PROCESSFinish the measurement. Process this chunk and return cumulative final results.
LAST::IGNOREFinish the measurement. Process this chunk but don’t return cumulative final results. (This would not normally be used.)

Each chunk also has a StartTime and EndTime in seconds. The StartTime of a chunk cannot be less than EndTime of the previous chunk. The difference between StartTime and EndTime cannot be more than 100 seconds nor less than 5 seconds. Usually, it's best to use the values of StartTime, EndTime etc. that are returned by the DeepAffex Extraction Library.

Note

This endpoint is subject to request throttling. You should not submit more data than can be obtained in real time. i.e. do not send more than five seconds of chunk data over the course of five seconds of real time.

Decoding real-time results

Real-time results that are returned via the Measurements.Subscribe endpoint are in JSON format. If you are using the legacy method of Protobufs then it can be deserialized using the DeepAffex Protobufs or using the (deprecated) decodeMeasurementResult method in the DeepAffex Extraction Library.

Notes in real-time results

Real-time results may also contain Notes for each Point. They provide additional information about the computation of the signal on the Cloud. If present for a specific DFX Point, notes will be as a value to the "Notes" key in that point. (e.g., for ABSI they will be at "result"/"Channels"/"ABSI"/"Notes").

"Notes" is an enum; multiple notes may be present against a single Point. They are described in the table below:

NoteDescription
NOTE_USED_PRED_DEMOGUser profile data was predicted for computations since user-entered data was missing
NOTE_SNR_BELOW_THRESHOLDSNR was below 0.5 db; no additional signals were computed
NOTE_FT_LIVENSSS_FAILEDFirst liveness test failed
NOTE_MODEL_LIVENSSS_FAILEDSecond liveness test failed
NOTE_MISSING_MEDICAL_INFOMedical history questionnaire information was not sent
NOTE_DEGRADED_ACCURACYSignal computation suffered from degraded accuracy

A measurement may sometimes return an error. To know more about different error codes, please refer to the error codes table.

Results on DeepAffex Cloud

A history of the measurements results made by a device, user or profile will be available on the DeepAffex Cloud if your legal agreement with Nuralogix allows saving results. (Please see the next section for more information if this is not the case.)

A historical list of measurements can be retrieved by using one of the following endpoints:

  • Measurements.List- Behavior of this endpoint depends upon the type of token used in the header.

    • Device Token - Will return a list of measurements made by the same Device Token as used in the header of this call.

    • User Token - Will return a list of measurements made by the user whose credentials were used to generate the User Token used in the header of this call.

  • Organizations.ListMeasurements- Using this endpoint, you can retrieve a list of measurements made across the organization, but it requires a token generated by a user with Admin, Lead, or Researcher roles.

The lists can be filtered by Date, EndDate, UserProfileID, UserProfileName, PartnerID, StudyID, StatusID or any combination thereof. The retrieved lists are paginated with a limited number of measurements in the response. There is an Offset parameter which can be used to specify the start of the list, and can be adjusted to get all the required measurements.

A measurement can have one of the following statuses:

StatusDescription
COMPLETEThis measurement has been successfully processed and all the expected results have been computed.
PARTIALThis measurement has been fully processed and one (or possibly more) of the expected results could not be accurately computed.
IN_PROGRESSThis measurement has been created and the DeepAffex Cloud is waiting for payloads from the client.
PENDINGThe DeepAffex Cloud has received all the payloads and the results are being computed.
INACTIVEThis measurement has been cancelled because a new measurement has been created with the same DeviceID.
ERRORAn error occurred in the DeepAffex Cloud while processing this measurement and none of the expected results could be computed.

Detailed measurement results can then be retrieved by using one of the following endpoints:

  • Measurements.Retrieve- Behavior of this endpoint depends upon the type of token used in the header.

    • Device Token - Will return the results for the measurements made by the same Device Token as used in the header of this call.

    • User Token - Will return the results for the measurements made by the user whose credentials were used to generate the User Token used in the header of this call.

  • Organizations.RetrieveMeasurement- Using this endpoint, you can retrieve any measurement across the organization, but it requires a token generated by a user with Admin, Lead, or Researcher roles.

Notes, if present, will also be part of the detailed results for each point.

Results not saved on DeepAffex Cloud

If your measurement results are not being saved on the DeepAffex Cloud, then:

  • The Measurements.Retrieve and Organizations.RetrieveMeasurement endpoints will return the MeasurementID and the status only.

  • On the DeepAffex Dashboard, a list of the measurements will be visible but the individual measurements will only show MeasurementID, creation date, Study name and status.

  • Your app must use the WebSocket-only Measurements.Subscribe endpoint during a measurement and save the results on the device or on your own backend.

Errors

A measurement may have one of the following errors received from the DeepAffex Cloud:

Error codeMeasurement statusDescription
WORKER_ERRORERRORThe DeepAffex Cloud encountered an unrecoverable internal error during the computation of results and could not continue.
LIVENESS_ERRORPARTIALThe Signal to Noise Ratio (SNR) of the measurement was too low to meet the minimum threshold. The results will contain only SNR = -100dB.
ANALYSIS_ERRORPARTIALThe DeepAffex Cloud encountered an error while computing a Point. The JSON results will have Point IDs as keys for Points with errors with values indicating failure reason.

The Errors field in the JSON will have the following format:

"Errors": {
  "Code": "<Error code>",
  "Errors": {
    "<signal_id>": {
      "msgs": [
        "<Error description>"
      ]
    }
  }
}

Next steps

In the next chapter, we will discuss the DeepAffex Extraction Library in more detail.