Integrating ML model in React js. Hey folks! 👋 | by Pranav | Mar, 2025

Hey of us! 👋

Earlier than diving in, it’s essential to grasp Atlys’ position as a visa firm working at scale. Correct passport picture seize is crucial for guaranteeing legitimate submissions, whether or not by means of a direct add or an automatic seize technique. Under, we’ll discover the automated passport seize course of intimately.

On this submit, I’ll present you find out how to create a React/Subsequent.js app that may work together with a TensorFlow Lite (TFLite) mannequin to auto-capture passport pictures instantly within the browser. By the top, you’ll have a working app that makes safe, environment friendly doc processing a breeze.

However first, let’s get a fast really feel for what we’re constructing.

1. Select a Nation: Choose any nation of your alternative. This may take you to that nation’s touchdown web page.
2. Begin the Utility: On the touchdown web page, find and click on the “Begin Utility” button.
3. Seize a Selfie: Observe the directions to take a selfie as a part of the appliance course of.
4. Proceed to the Passport Seize Step: After finishing the selfie step, you’ll be directed to the passport seize display screen. Right here, you may work together with the passport picture seize function and discover its performance.

What We’ll Cowl:

1. What’s TensorFlow Lite?
2. Why Run ML Fashions within the Browser?
3. Establishing and Loading a TFLite Mannequin
4. Enabling the Digital camera for Passport Scanning
5. Run modal by passing body from the video
6. Drawing a Stay Border for Auto-Seize
7. Finest practices for efficiency

TensorFlow Lite is a streamlined model of TensorFlow, particularly designed for cell and embedded gadgets. It permits on-device machine studying, which gives a number of benefits:

• Diminished Latency: All processing occurs on the gadget, without having for a server.
• Offline Performance: Works with out web ones the modal is loaded.
• Enhanced Privateness: Person information by no means leaves the gadget.

Why Run ML Fashions within the Browser?

Operating machine studying fashions within the browser is turning into more and more widespread resulting from advantages like:

• Higher Person Expertise: On the spot responses with out server delays.
• Diminished Prices: No want for server infrastructure.
• Privateness: Delicate information stays on the person’s gadget.
• Offline Capabilities: Full performance with out web entry.

Discover ways to create a mannequin by exploring this detailed information: Automating Passport Detection and Quality Analysis Using Deep Learning.

• Mannequin Conversion: Convert an current TensorFlow mannequin to TFLite.
• Net Optimization: Tremendous-tune the mannequin for net deployment.

On this information, I’ll stroll you thru the core setup. For an entire code reference, try this gist

To get began, we’ll create a customized hook to load our TFLite mannequin in React.

Create a file named useLoadModel.tsx and use it to handle mannequin loading and availability.

To allow the digital camera, we’ll create one other hook, useCamera, which initializes and manages the video feed from the person’s gadget.

Right here’s find out how to arrange the mannequin to course of frames from the video feed in real-time.

import { useEffect, useRef } from "react";
import useCamera from "./useCamera";
import { useLoadModal } from "./useLoadModal";
import { TFLiteModel } from "@tensorflow/tfjs-tflite";
import * as tf from "@tensorflow/tfjs";
import { resizeCanvas, transpose } from "./passport.helpers";

export const useModal = () => {
const captureRef = useRef(null);
const canvasBoxRef = useRef(null);
const isModalRunning = useRef(false);
const animationFrameId = useRef(null);
const { error, videoRef } = useCamera();
const { mannequin, standing } = useLoadModal();

const processFrame = async (
video: HTMLVideoElement,
canvas: HTMLCanvasElement,
context: CanvasRenderingContext2D
) => {
if (video.readyState === video.HAVE_ENOUGH_DATA) {
canvas.width = video.videoWidth;
canvas.top = video.videoHeight;
context.drawImage(video, 0, 0, canvas.width, canvas.top);
const processimage = resizeCanvas(canvas);
const tensor = (await tf.browser
.fromPixels(processimage)
.expandDims(0)
.toFloat()
.div(255)) as tf.Tensor;
const outputTensor = await mannequin.predict(tensor);
const packing containers = (await outputTensor.array()) as quantity[][][];
const rework = (await transpose(packing containers[0])) as quantity[][];
// now you will have an inventory of lessons based mostly on the mannequin
// you may draw the packing containers on the canvas
// after which name the perform once more
}
};
useEffect(() => {
const video = videoRef.present;
const canvas = captureRef.present;
if (!video || !canvas) return;
const context = canvas.getContext("second");
if (
standing === "loading" ||
standing === "error" ||
isModalRunning.present ||
!context
)
return;
animationFrameId.present = requestAnimationFrame(() =>
processFrame(video, canvas, context)
);
return () => {
if (animationFrameId.present) {
cancelAnimationFrame(animationFrameId.present);
}
};
}, []);

return { captureRef, canvasBoxRef, error, videoRef };
};Drawing a dwell border across the passport on the display screen

After enabling the digital camera, seize frames periodically, move them to the mannequin, and mechanically draw a field on the display screen based mostly on the mannequin’s output. The TFLite mannequin returns an array response, the place every component represents a category. Use this information to determine and draw the bounding field on a component, making the seize course of hands-free.

Under, I’m itemizing all of the helper strategies wanted to run the mannequin and seize the passport picture:

Canvas Sizing: A way to arrange and measurement the canvas.

Bounding Box Drawing: A way to attract a bounding field across the detected passport.

Image Capture and Cropping: A way to seize a picture from the video feed and crop it to the passport space.

For the reason that digital camera streams a number of frames per second, we have to keep away from processing each body to stop lagging:

• Body Processing: Renders just one body at a time for mannequin inference, then shows the output to keep away from efficiency degradation. Utilizing this strategy, we forestall system lag and guarantee a easy person expertise.
• Environment friendly Animation: Use the requestAnimationFrame() API, which calls the perform each second. Learn more about it on MDN.
• While you’re completed, you’ll want to cancel the animation with cancelAnimationFrame() to unlock assets. Name this perform whenever you unmount the body element.