Extracting machine-readable text from scanned physical documents, receipts, identification cards, and photographs is a foundational capability in document engineering. Historically, optical character recognition (OCR) was performed exclusively by native desktop utilities or cloud vision APIs requiring raw document uploads over the network.
Modern browser execution environments, powered by WebAssembly (WASM), Web Workers, and SIMD hardware acceleration, can execute complex computer vision and deep learning pipelines locally on the client device.
However, running OCR in a web browser involves more than compiling an inference model. High-accuracy text extraction requires in-browser image preprocessing, adaptive Otsu binarization, skew detection algorithms, geometric layout analysis, and the construction of a hierarchical Document Abstract Syntax Tree (hAST) that maps character glyphs directly to canvas coordinate space.
1. In-Browser Image Preprocessing: Canvas Math and Computer Vision
The accuracy of any downstream neural recognition model depends heavily on the quality of input imagery. Scanned documents and mobile camera photos frequently suffer from uneven lighting, sensor noise, optical perspective distortions, and skewed rotation angles.
Before feeding pixel buffers to an OCR engine, the browser must execute a series of computer vision transformations directly on raw ImageData structures via HTML5 Canvas.
+-----------------------------------------------------------------------+
| In-Browser Computer Vision Preprocessing |
+-----------------------------------------------------------------------+
| Raw Image (JPEG, PNG, WebP) |
| | |
| v |
| [ HTML5 OffscreenCanvas: 32-bit RGBA Byte Buffer ] |
| | |
| v |
| [ Grayscale Luminance Conversion: Y = 0.299R + 0.587G + 0.114B ] |
| | |
| v |
| [ Otsu's Adaptive Thresholding: Maximizing Inter-Class Variance ] |
| | |
| v |
| [ Skew Angle Calculation: Radon / Hough Transform Line Accumulation] |
| | |
| v |
| [ 2D Matrix Rotation Interpolation (Bilinear Filtering) ] |
| | |
| v |
| Clean, Deskewed, Binarized 1-Bit Buffer Passed to Inference Engine |
+-----------------------------------------------------------------------+Grayscale Conversion Using Perceptual Luminance
An uncompressed color image is represented in canvas memory as a flat Uint8ClampedArray formatted as interleaved 32-bit [R, G, B, A] bytes.
Converting this color space into a single 8-bit channel requires applying ITU-R BT.601 perceptual luma weighting, which accounts for the human visual system's heightened sensitivity to green wavelengths:
Executing this calculation in JavaScript over a 12-megapixel photograph involves 12 million loop iterations. By structuring the array access using 32-bit unsigned integer views (Uint32Array), bitwise shift operations replace expensive property accessors:
// High-performance grayscale conversion over raw Canvas buffer
export function convertToGrayscale(imageData: ImageData): Uint8Array {
const { width, height, data } = imageData;
const pixelCount = width * height;
const grayscaleBuffer = new Uint8Array(pixelCount);
// Read data buffer through 32-bit words for memory efficiency
const uint32View = new Uint32Array(data.buffer);
for (let i = 0; i < pixelCount; i++) {
const pixel = uint32View[i];
// Extract channels from 0xAABBGGRR little-endian integer representation
const r = pixel & 0xff;
const g = (pixel >> 8) & 0xff;
const b = (pixel >> 16) & 0xff;
grayscaleBuffer[i] = (r * 77 + g * 150 + b * 29) >> 8; // Integer approximation
}
return grayscaleBuffer;
}Otsu's Global Binarization Algorithm
Neural text recognition models perform best on clean, binarized images where text ink is represented by 1 (black) and paper background is represented by 0 (white). Fixed thresholding fails when shadows, uneven flash lighting, or yellowed paper are present.
Otsu's algorithm calculates the optimal global threshold that separates pixels into two statistical classes: background (intensities ) and foreground (intensities ). It achieves this by finding the threshold that maximizes the between-class variance :
Where:
- is the cumulative probability of class .
- is the cumulative probability of class .
- is the mean intensity of class .
- is the mean intensity of class .
- is the normalized frequency histogram of pixel intensity .
// Fast Otsu threshold calculator running in browser RAM
export function calculateOtsuThreshold(gray: Uint8Array): number {
const histogram = new Int32Array(256);
const totalPixels = gray.length;
for (let i = 0; i < totalPixels; i++) {
histogram[gray[i]]++;
}
let sumAll = 0;
for (let t = 0; t < 256; t++) {
sumAll += t * histogram[t];
}
let sumBack = 0;
let weightBack = 0;
let maximumVariance = 0;
let optimalThreshold = 128;
for (let t = 0; t < 256; t++) {
weightBack += histogram[t];
if (weightBack === 0) continue;
const weightFore = totalPixels - weightBack;
if (weightFore === 0) break;
sumBack += t * histogram[t];
const meanBack = sumBack / weightBack;
const meanFore = (sumAll - sumBack) / weightFore;
const varianceBetween = weightBack * weightFore * Math.pow(meanBack - meanFore, 2);
if (varianceBetween > maximumVariance) {
maximumVariance = varianceBetween;
optimalThreshold = t;
}
}
return optimalThreshold;
}Skew Detection via the Radon and Hough Transforms
When paper is scanned at an angle, text baselines tilt, complicating horizontal line segmentation. Using the Hough Transform or discrete Radon projections, the preprocessor evaluates line projections across angles from to at intervals.
The projection angle that produces the highest intensity variance along its projection profile corresponds to the baseline angle of the text. Once calculated, the canvas is rotated by using bilinear interpolation to produce level horizontal text lines.
2. Neural Network Execution in WebAssembly: Tesseract vs Modern Architectures
Once an image is cleaned, cropped, and binarized, the pixel buffer is handed to a WebAssembly neural inference pipeline.
+-----------------------------------------------------------------------+
| Tesseract WASM Inference Pipeline |
+-----------------------------------------------------------------------+
| Binarized Image Buffer (Web Worker) |
| | |
| v |
| [ Line Finding & Baseline Fitting Engine ] |
| | |
| v |
| [ Text Line Normalization: 32px Vertical Image Slices ] |
| | |
| v |
| [ Bidirectional LSTM Network (WASM + SIMD128) ] |
| - Input: Sequence of vertical pixel column feature vectors |
| - Forward LSTM Pass + Backward LSTM Pass |
| - Softmax Output Matrix: Probability distribution over character set|
| | |
| v |
| [ CTC (Connectionist Temporal Classification) Decoder ] |
| - Collapse repeated character predictions |
| - Strip blank non-emission tokens |
| | |
| v |
| Recognized Character Sequence + Confidence Probabilities |
+-----------------------------------------------------------------------+The Tesseract LSTM Model Architecture
Modern Tesseract (v4 and v5) replaced legacy topological character classifiers with a deep recurrent neural network using Long Short-Term Memory (LSTM) cells.
The text recognition pipeline operates across distinct stages:
- Line Normalization: Segmented text lines are scaled to a fixed height of 32 or 48 pixels while preserving natural aspect ratios.
- Feature Extraction: The normalized image slice is scanned from left to right as an ordered sequence of narrow vertical feature windows.
- Bidirectional LSTM Propagation: Two recurrent layers (one scanning left-to-right, one right-to-left) maintain internal cell states that capture character sequence context, improving accuracy for ambiguous letterforms.
- CTC Decoding: Because character widths vary significantly (for example, the letter "m" spans many more pixel columns than "l"), the network outputs a sequence of probability vectors across time slices using Connectionist Temporal Classification (CTC). The decoder collapses repeated emissions and removes blank tokens to produce final character strings.
Hardware Acceleration via WebAssembly SIMD
Evaluating thousands of matrix-vector multiplications for every character in pure JavaScript would introduce significant processing lag. Modern browsers support WebAssembly 128-bit SIMD (Single Instruction, Multiple Data).
Compiling Tesseract with -msimd128 instructs Clang to vectorize inner loops across 128-bit registers (such as v128), evaluating four 32-bit floating-point multiplications or sixteen 8-bit integer dot products in a single CPU cycle:
// Vectorized SIMD dot-product loop in WebAssembly C++ core
v128_t acc = wasm_f32x4_splat(0.0f);
for (int i = 0; i < vectorLength; i += 4) {
v128_t w = wasm_v128_load(&weights[i]);
v128_t in = wasm_v128_load(&inputs[i]);
acc = wasm_f32x4_add(acc, wasm_f32x4_mul(w, in));
}SIMD vectorization reduces total neural inference time by 60% to 75% compared to baseline WebAssembly builds, enabling near-instantaneous page transcription.
Neural Weight Caching via IndexedDB
Pre-trained language models for OCR (eng.traineddata, deu.traineddata) range from 4 MB to 25 MB in size. Re-downloading these model binaries on every page visit creates unnecessary network overhead.
The browser pipeline uses an IndexedDB caching layer that stores compiled model buffers locally. On subsequent runs, model weights load directly from persistent browser storage within milliseconds.
3. Document Layout Analysis and Geometric Segmentation
An unstructured string of text stripped of spatial context provides limited utility for downstream processing. Modern document engineering requires physical layout analysis to identify document hierarchy, reading order, and table boundaries.
+-----------------------------------------------------------------------+
| Hierarchical Physical Document Segmentation |
+-----------------------------------------------------------------------+
| Input Binarized Page (1-Bit Bitmap) |
| | |
| v |
| [ Connected Component Analysis (CCA): 8-Way Pixel Neighborhoods ] |
| | |
| v |
| [ Bounding Box Generation for Discrete Ink Blobs ] |
| | |
| v |
| [ Anisotropic Morphological Dilation: Rectangular Structuring Element]|
| - Horizontal Kernel: Bridges inter-character gaps into words |
| - Extended Horizontal: Bridges word gaps into complete text lines |
| - Vertical Kernel: Detects column breaks and paragraph gutters |
| | |
| v |
| [ Geometric Classification ] |
| - Aspect Ratio & Density -> Paragraphs, Tabular Grids, Images |
+-----------------------------------------------------------------------+Connected Component Analysis (CCA)
Physical segmentation begins by identifying connected components (islands of adjacent black pixels). Using 8-way connectivity, an algorithm scans the binary bitmap and labels every discrete glyph:
// Two-pass connected component labeling with disjoint-set union
export interface BoundingBox {
x0: number;
y0: number;
x1: number;
y1: number;
}
export function extractConnectedComponents(binary: Uint8Array, width: number, height: number): BoundingBox[] {
// Label matrix tracking component membership
const labels = new Int32Array(width * height);
const parent = [0];
let currentLabel = 1;
function find(i: number): number {
let root = i;
while (root !== parent[root]) root = parent[root];
let curr = i;
while (curr !== root) {
const next = parent[curr];
parent[curr] = root;
curr = next;
}
return root;
}
function union(i: number, j: number): void {
const rootI = find(i);
const rootJ = find(j);
if (rootI !== rootJ) parent[rootI] = rootJ;
}
// First pass: Assign initial labels and record equivalences
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const idx = y * width + x;
if (binary[idx] === 0) continue; // Background pixel
const left = x > 0 ? labels[idx - 1] : 0;
const top = y > 0 ? labels[idx - width] : 0;
if (left === 0 && top === 0) {
labels[idx] = currentLabel;
parent[currentLabel] = currentLabel;
currentLabel++;
} else if (left !== 0 && top === 0) {
labels[idx] = left;
} else if (left === 0 && top !== 0) {
labels[idx] = top;
} else {
labels[idx] = left;
if (left !== top) union(left, top);
}
}
}
// Second pass: Consolidate equivalent labels into bounding boxes
const boxes = new Map<number, BoundingBox>();
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const idx = y * width + x;
const rawLabel = labels[idx];
if (rawLabel === 0) continue;
const rootLabel = find(rawLabel);
let box = boxes.get(rootLabel);
if (!box) {
box = { x0: x, y0: y, x1: x, y1: y };
boxes.set(rootLabel, box);
} else {
if (x < box.x0) box.x0 = x;
if (y < box.y0) box.y0 = y;
if (x > box.x1) box.x1 = x;
if (y > box.y1) box.y1 = y;
}
}
}
return Array.from(boxes.values());
}Morphological Dilation and Text Block Grouping
Once individual character bounding boxes are established, the engine clusters them into reading blocks using mathematical morphology. Applying an anisotropic rectangular structuring element (stretching wider horizontally than vertically) dilates character boxes into solid line strips.
Analyzing white-space projection profiles (horizontal and vertical projection cutting) isolates multi-column text layouts, table cells, and isolated header elements.
4. Constructing the Hierarchical Document Abstract Syntax Tree (hAST)
To make OCR results programmatically actionable, the system transforms flat recognition output into a hierarchical Document Abstract Syntax Tree (hAST).
+-----------------------------------------------------------------------+
| Document hAST Object Hierarchy |
+-----------------------------------------------------------------------+
| DocumentNode (Root) |
| | |
| +-- PageNode [pageNumber: 1, width: 2480, height: 3508] |
| | |
| +-- BlockNode [type: 'paragraph', bbox: { x0, y0, x1, y1 }] |
| | |
| +-- LineNode [baseline: { y0, y1 }, bbox: { ... }] |
| | |
| +-- WordNode [text: "PetexSpace", confidence: 0.985] |
| | |
| +-- SymbolNode [char: 'P', bbox: { ... }] |
| +-- SymbolNode [char: 'e', bbox: { ... }] |
| +-- SymbolNode [char: 't', bbox: { ... }] |
| ... |
+-----------------------------------------------------------------------+Schema Definition of the hAST
Every node in the tree tracks its semantic category, bounding box coordinates normalized against the original image dimensions, and recognition confidence scores:
// TypeScript schema for hierarchical OCR Abstract Syntax Tree
export interface DocumentNode {
type: 'document';
version: '1.0';
pages: PageNode[];
}
export interface PageNode {
type: 'page';
pageNumber: number;
width: number;
height: number;
blocks: BlockNode[];
}
export interface BlockNode {
type: 'paragraph' | 'heading' | 'table' | 'figure';
bbox: BoundingBox;
confidence: number;
lines: LineNode[];
}
export interface LineNode {
type: 'line';
bbox: BoundingBox;
baseline: { slope: number; intercept: number };
words: WordNode[];
}
export interface WordNode {
type: 'word';
text: string;
confidence: number;
bbox: BoundingBox;
symbols: SymbolNode[];
}
export interface SymbolNode {
type: 'symbol';
char: string;
confidence: number;
bbox: BoundingBox;
}Unicode Normalization and Ligature Separation
Raw neural predictions often contain typographic ligatures (such as "fi", "fl", "ffi", or diphthongs) or decomposed diacritic sequences.
The AST builder applies Unicode Normalization Form C (NFC) via String.prototype.normalize('NFC') to ensure consistent searchability. It also resolves end-of-line hyphenation breaks by checking if the trailing character is a hyphen and the succeeding word forms a valid dictionary entry.
5. The PetexSpace OCR Engine: Interactive Canvas Overlay and Privacy
The PetexSpace extract text with in-browser OCR tool demonstrates how WebAssembly, computer vision math, and hierarchical AST generation combine to deliver privacy-preserving text recognition. You can also preprocess image documents locally to optimize scans before recognition.
+-----------------------------------------------------------------------+
| PetexSpace OCR Interactive Architecture |
+-----------------------------------------------------------------------+
| User Image Drag-and-Drop |
| | |
| v |
| [ Dedicated Dedicated Worker: In-Browser Preprocessing & WASM ] |
| | |
| +--> Emit hAST Payload + Normalized Coordinates |
| |
| Main UI Viewport: Multi-Layer Canvas Stacking |
| +-----------------------------------------------------------------+ |
| | Layer 3: Invisible, Selectable Transparent Text DOM Spans | |
| | Layer 2: Canvas Bounding Box Highlight Overlay (Active Cursor) | |
| | Layer 1: Hardware-Accelerated High-DPI Source Image Render | |
| +-----------------------------------------------------------------+ |
+-----------------------------------------------------------------------+The Invisible Text Selection Layer
A common limitation of browser-based OCR tools is providing only a disconnected textarea containing raw extracted text. PetexSpace solves this by rendering an interactive multi-layer interface directly over the document preview:
- Base Layer: Renders the original document photograph on a high-DPI HTML5 canvas.
- Overlay Layer: Draws active bounding boxes around words as the user hovers over the document.
- Selection Layer: Positions transparent HTML
<span>elements directly above the corresponding words on the canvas using CSS transforms:
// Mounting transparent DOM spans mapped to hAST coordinates
export function mountSelectableTextLayer(container: HTMLElement, page: PageNode, scale: number): void {
container.innerHTML = '';
container.style.position = 'relative';
for (const block of page.blocks) {
for (const line of block.lines) {
for (const word of line.words) {
const span = document.createElement('span');
span.textContent = word.text;
span.className = 'ocr-selectable-word';
// Exact pixel positioning matching canvas scale
span.style.position = 'absolute';
span.style.left = `${word.bbox.x0 * scale}px`;
span.style.top = `${word.bbox.y0 * scale}px`;
span.style.width = `${(word.bbox.x1 - word.bbox.x0) * scale}px`;
span.style.height = `${(word.bbox.y1 - word.bbox.y0) * scale}px`;
// Render completely invisible while preserving browser text selection
span.style.color = 'transparent';
span.style.fontSize = `${(word.bbox.y1 - word.bbox.y0) * scale * 0.85}px`;
span.style.userSelect = 'text';
container.appendChild(span);
}
}
}
}This architecture allows users to click, highlight, copy, and search text directly within the original visual document context.
Complete Data Privacy via Local Execution
Scanned documents often contain sensitive personal data: passports, government identification cards, personal tax filings, medical reports, and legal agreements. Submitting these images to remote cloud APIs exposes users to data retention hazards, security breaches, and regulatory non-compliance.
PetexSpace enforces total data isolation. Every preprocessing filter, neural inference pass, and AST construction step runs exclusively within local browser memory. No pixel buffers, network packets, or document tokens are ever transmitted across external networks.
By executing OCR entirely inside the browser runtime, sensitive records remain private and secure on the user's device.