Detailed Integration¶
Overview¶
The Knomi Face SDK is designed to be simple to integrate against. This chapter will show an example integration using samples from the Knomi-S-Face-App demo application that has been provided in the SDK package.
The Knomi S Face Demo Program¶
Introduction¶
The Knomi-S-Face-App demo program is a generic Android application employing basic design principles. As of Knomi Face SDK version 2.3.1 an optional mode for using a more accurate but larger face model has been provided. Applications looking to utilize the higher accuracy face model must build with the additional AAR provided within the SDK.
MainAppActivity¶
The MainAppActivity is the starting point for application development.
The basic application methods of permission and fragment management.
A basic layout (FrameLayout) in which the Knomi Face SDK will reside.
As it relates to the Knomi Face SDK it implements
ActionButtonFragment.ActionButtonListener and
FaceLiveness.LivenessActivityPresenter
GetServerVersion.GetServerVersionListener
ActionButtonFragment.ActionButtonListener
ErrorReporterCallback
Some key functions are the following: onWorkflowSelected, onCaptureStart, onCaptureEnd
In addition, the MainAppActivity shows how to create the InitializeBackgroundTask to set the selected model with the setStaticProperty and have it loaded in a thread. Using the InitializeBackgroundTask will allow the main thread to continue initialization and load the necessary model data into memory.
public class InitializeBackgroundTask extends AsyncTask<String, Void, Void> {
private boolean mCouldNotOpenModel = false;
private String mModelName = "";
private ModelInitializationListener mInitializationListener;
InitializeBackgroundTask(ModelInitializationListener listener) {
mInitializationListener = listener;
mCouldNotOpenModel = false;
mModelName = "";
}
@Override
protected void onPreExecute() {
super.onPreExecute();
}
@Override
protected Void doInBackground(final String[] params) {
mModelName = params[0];
try {
FaceLiveness.setStaticProperty(FaceLiveness.StaticPropertyTag.FACE_MODEL, params[0]);
mLivenessApi = new FaceLiveness( getApplicationContext() );
} catch (Exception e) {
e.printStackTrace();
}
try {
// initialize the library, wait for callback...
mLivenessApi.initializeLivenessLibrary(MainAppActivity.this);
}
catch (Exception e) {
mCouldNotOpenModel = true;
return null;
}
synchronized (InitializeBackgroundTask.this) {
while (!mInitComplete && !isCancelled()) {
try {
Log.i(LIVENESS_SAMPLE_TAG, "Launch starting WAIT");
InitializeBackgroundTask.this.wait(SPLASH_TIME);
Log.i(LIVENESS_SAMPLE_TAG, "Launch completing WAIT");
} catch (InterruptedException e) {
Log.e(LIVENESS_SAMPLE_TAG, "Launch INTERRUPTED: " + e.getMessage());
}
}
}
return null;
}
@Override
protected void onPostExecute(Void o) {
super.onPostExecute(o);
if (isCancelled()) {
finish(); // ends the Launch Activity
}
HideHourglass();
mInitializationListener.onInitializationComplete(!mCouldNotOpenModel, mModelName);
}
}
In MainAppActivity onWorkflowSelected sets up the LivenessComponentAPI and shows how the PropertyTag values need to be set before selectWorkflow is called.
@Override
public void onWorkflowSelected(String workflowName, String nexaMode, String id, String overrideJson) {
boolean isRunning = false;
SharedPreferences sharedPref = PreferenceManager.getDefaultSharedPreferences(this);
mUsername = sharedPref.getString("pref_user_name", mUsername);
mCaptureTimeout = sharedPref.getString("pref_capture_timeout", mCaptureTimeout);
mImageCaptureProperty = sharedPref.getBoolean("pref_image_capture_selection", false);
mCaptureOnDevice = sharedPref.getBoolean("pref_capture_on_device_selection", false);
if (mCaptureTimeout.equals(""))
mCaptureTimeout = "0";
// Note the following calls must be in the correct order.
// Must set properties before calling selectWorkflow.
try {
mLivenessApi.setProperty(FaceLiveness.PropertyTag.USERNAME, mUsername);
mLivenessApi.setProperty(FaceLiveness.PropertyTag.CONSTRUCT_IMAGE, mImageCaptureProperty);
mLivenessApi.setProperty(FaceLiveness.PropertyTag.TIMEOUT, Double.parseDouble(mCaptureTimeout));
mLivenessApi.setProperty(FaceLiveness.PropertyTag.CAPTURE_ON_DEVICE, mCaptureOnDevice);
} catch (Exception e) {
runOnUiThread(new Runnable() {
@Override
public void run() {
String message = "Invalid property setting!!!";
Toast t = Toast.makeText(getApplicationContext(), message, Toast.LENGTH_LONG);
t.setGravity(Gravity.CENTER, 0, 0);
t.show();
}
});
e.printStackTrace();
}
// select a workflow
try {
mLivenessApi.selectWorkflow(this, workflowName, overrideJson);
} catch (Exception e) {
runOnUiThread(new Runnable() {
@Override
public void run() {
String message = "Invalid workflow setting!!!";
Toast t = Toast.makeText(getApplicationContext(), message, Toast.LENGTH_LONG);
t.setGravity(Gravity.CENTER, 0, 0);
t.show();
}
});
e.printStackTrace();
return;
}
mNexaMode = nexaMode;
mID = id;
checkPermissionsAndSelectItem(WORKFLOW_EXECUTE);
}
In MainAppActivity onErrorReporter is the callback used when the library encounters an error from which it cannot recover. This callback provides the application integrator a procedure that can be used create a graceful exit from the application. After performing the necessary operations and presenting a message to the user the application must be terminated.
@Override
public void onErrorReporter(int errorCode, final String info, final Context context) {
final String message = info;
Toast t = Toast.makeText(context, message, Toast.LENGTH_LONG);
t.setGravity(Gravity.BOTTOM, 0, 0);
t.show();
AlertDialog dialog;
AlertDialog.Builder builder = new AlertDialog.Builder(context);
builder.setMessage("Application was stopped...")
.setPositiveButton("The LivenessAPI encountered a problem.", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int id) {
}
})
.setNegativeButton("Exit", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int id) {
// Not worked!
dialog.dismiss();
System.exit(0);
android.os.Process.killProcess(android.os.Process.myPid());
}
});
dialog = builder.create();
if(!dialog.isShowing())
dialog.show();
}
MainAppActivity also implements methods such as onCaptureStart, onCaptureEnd and onCaptureAbort to process workflow events.
The LivenessFragment runs the workflow.
LivenessFeedbackView is used to implement the display and receive data from the callbacks. This data includes information related to the feedback of the image captures and the device positioning information. The main display has been changed to look like a race track surrounding the face being captured. The exterior of the race track has the opacity set such that it is darkened to focus the attention of the user on the task.
Below is a more detailed description of the updates to the Knomi Face SDK with examples from the Knomi S Face-APP demo program.
MainAppActivity¶
As it related to the Knomi Face SDK the MainAppActivity implements the FaceLiveness.LivenessActivityPresenter. The implementation of these methods provides a listener for the completion of initialization of the FaceLiveness.
Upon completion of the workflow the onCaptureEnd method (Listing 14) is called. At this point the application can request the server package from the Knomi Face SDK. This is the data that will be sent to the server for processing. The implementation for Knomi-S-Face-App is the following:
public void processResults() {
try {
String serverPackage = mLivenessApi.getServerPackage();
RestClientTask postTask = new RestClientTask(this);
if(mCaptureOnDevice)
mDisplayOnDeviceImage = true;
postTask.executeLiveness(serverPackage, mMatchThresholdValue);
} catch (FaceLivenessException e) {
//getServerPackage can throw an exception on an error
e.printStackTrace();
}
}
public void onCaptureEnd() {
runOnUiThread(new Runnable() {
@Override
public void run() {
popUpToBaseFragment();
ShowHourglassDirect();
}
});
processResults();
}
LivenessFragment¶
The LivenessFragment implementation is used to run the workflow. It creates the user interface used to display the device positioning information and it performs initialization of the Knomi Face SDK. For example, the onCreateView method (Listing 15) performs the following:
@Nullable
@Override
public View onCreateView(LayoutInflater inflater,
@Nullable ViewGroup container, @Nullable Bundle savedInstanceState) {
mLivenessApi = mWorkflowActiveListener.getFaceLiveness().get();
View view = inflater.inflate(R.layout.fragment_liveness_view, container, false);
// Create the LivenessFeedbackView
LivenessFeedbackView feedbackView = new LivenessFeedbackView(getActivity(), this);
mFrameLayout = (FrameLayout) view.findViewById(R.id.liveness_component_imp);
// LivenessSDK - Allocate
mLivenessApi.AllocateLivenessComponentView(getContext());
// LivenessSDK – Set some callbacks
mLivenessApi.setWorkflowStateCallback(this);
mLivenessApi.setFeedbackCallback(feedbackView);
mLivenessApi.setDevicePositionCallback(feedbackView);
mLivenessApi.setErrorReporter(mUI);
try {
mLivenessApi.bindLivenessApi(mFrameLayout, getActivity());
mLivenessApi.onCreateView(getActivity(), mFrameLayout);
} catch (Exception invalidWorkflow) {
Log.e(WORKFLOW_FRAGMENT_TAG, "Workflow Error: " + invalidWorkflow.getMessage());
}
// The following is for the device orientation indicator.
// It is used during the face capture
int index = 0;
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator0);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator1);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator2);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator3);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator4);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator5);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator6);
mPositionViews[index++] = (ImageView) view.findViewById(R.id.positionIndicator7);
mPositionViews[index] = (ImageView) view.findViewById(R.id.positionIndicator8);
for(int i=0;i<9;i++) {
mPositionColor[index] = false;
}
mFeedback = (TextView) view.findViewById(R.id.tv_feedback);
return view;
LivenessFragment – User Interface and Workflow¶
The workflowStateCallback is used to store the workflow state, display the necessary messages on the user interface or take the necessary action depending on the received state. For example on receiving the WorkflowState.Complete state it will call the onCaptureEnd of the workflow action listener. The second parameter representing the event is only valid when the WorkflowState is EVENT. The utility procedure getAutoCaptureString is provided to get a string from the feedback enumeration value. These strings can be localized as they are extracted from the string.xml file.
As an example, the Knomi-S-Face-App workflowStateCallback (Listing 16) looks like the following:
@Override
public void workflowStateCallback(WorkflowState workflowState, String event) {
String message = "";
if (workflowState == WorkflowState.COMPLETE) {
mUI.onCaptureEnd();
Log.d(WORKFLOW_FRAGMENT_TAG, "Complete");
}
else if ( (workflowState == WorkflowState.TIMEDOUT) ) {
mUI.onCaptureTimedout();
Log.d(WORKFLOW_FRAGMENT_TAG, "Timed Out");
} else if (workflowState == WorkflowState.ABORT) {
mUI.onCaptureAbort();
Log.d(WORKFLOW_FRAGMENT_TAG, "Abort");
} else if (workflowState == WorkflowState.DEVICE_IN_POSITION) {
Log.d(WORKFLOW_FRAGMENT_TAG, "Device In position");
} else if (workflowState == WorkflowState.PREPARING) {
displayMessage("Position Device Vertically");
Log.d(WORKFLOW_FRAGMENT_TAG, "Preparing");
} else if (workflowState == WorkflowState.EVENT) {
mEvent = event;
if(mEvent.equals("NONE"))
displayMessage("HOLD");
else {
displayMessage(mEvent);
}
Log.d(WORKFLOW_FRAGMENT_TAG, "Event = " + event);
} else if (workflowState == WorkflowState.HOLD_STEADY) {
displayMessage("HOLD");
Log.d(WORKFLOW_FRAGMENT_TAG, "Hold");
} else if (workflowState == WorkflowState.SENSOR_ERROR) {
displayDeviceError(event);
}
mWorkflowState = workflowState;
}
LivenessFragment – Lifecycle procedures¶
Note that the Android lifecycle procedures (Listing 18) implemented for the LivenessFragment need to perform any user application specific tasks as well as call the corresponding procedure in the SDK. For the SDK to function correctly the LivenessFragment must be removed from the fragment stack after each capture attempt. Removing the LivenessFragment from the stack causes the Android system to call the lifecycle procedures. In the following example code popUpToBaseFragment demonstrates the case where multiple fragments are on the stack and removeLivenessFragment demonstrates the case where there is only one fragment on the stack.
private void popUpToBaseFragment() {
FragmentManager fm = getSupportFragmentManager();
if(fm.isStateSaved()) {
mPopWait = true;
return;
}
int count = fm.getBackStackEntryCount();
if (count > 1) {
FragmentManager.BackStackEntry backStackEntry = fm.getBackStackEntryAt(count - 2);
int position = findStackPosition(backStackEntry.getName());
fm.popBackStack(mActionPanelTitles[position], 0);
updatedPositionTitle(position);
}
}
private void removeLivenessFragment() {
FragmentManager fm = getSupportFragmentManager();
if(fm.isStateSaved())
{
mPopWait = true;
return;
}
int count = fm.getBackStackEntryCount();
if (count > 0)
{
FragmentTransaction trans = fm.beginTransaction();
trans.remove(mFragment);
trans.commit();
fm.popBackStack();
}
}
The lifecycle procedures for the LivenessFragment need a minimum implementation as shown below. They need to call the corresponding procedure in the SDK.
@Override
public void onStart() {
super.onStart();
mLivenessApi.onStart();
}
@Override
public void onResume() {
super.onResume();
try {
mLivenessApi.onResume();
} catch (Exception e) {
Log.e(WORKFLOW_FRAGMENT_TAG, "No Liveness API: " + e.getMessage());
}
}
@Override
public void onPause() {
super.onPause();
try {
mLivenessApi.onPause();
} catch (Exception e) {
Log.e(WORKFLOW_FRAGMENT_TAG, "No Liveness API: " + e.getMessage());
}
}
@Override
public void onDestroy() {
super.onDestroy();
mLivenessApi.onDestroy();
}
@Override
public void onDetach() {
super.onDetach();
mWorkflowActiveListener = null;
mLivenessApi.onDetach();
}
LivenessFeedbackView¶
Contains the implementation of the other callbacks and procedures to provide user interface updates based on the received data from the LivenessSDK. The implementation of the callbacks (Listing 19) is very simple in that data is stored and then the UI or some variables are updated from the input.
@Override
public void onCanvasUpdateCallback(Canvas canvas, RectF areaOfInterest, float displayWidthScale) {
mCanvas = canvas;
mAreadOfInterest = areaOfInterest;
mDisplayWidthScale = displayWidthScale;
}
@Override
public void onDevicePositionCallback(float positionState) {
//!!TODO: fill in device indicator
Log.v(CUSTOM_VIEW_TAG, "onDevicePositionCallback positionState: " + positionState);
mLastPositionState = positionState;
reportValues(mLastPositionState);
}
Additionally, in this file are procedure for updating the user interface. There are procedures to update the “race track” display and the image, and update the text directions. There are some auxiliary procedures in this file for drawing the user interface in different ways.
RestClientTask¶
This file contains methods that send and receive data to and from the server. There is a ClientTask class derived from the AsyncTask to perform the server interaction. When the result is returned there are procedure to parse the resulting data: parseCaptureOnlyServerResults, parseRegisterServerResults, parseVerifyServerResults.