High-Performance Web Architecture

A four megapixel image is sixteen million bytes. Any operation that touches every byte more than a couple of times will be felt by the user. Performance work in image processing is mostly about reducing the number of passes and keeping each pass friendly to the memory hierarchy.

Low-Level Pixel Manipulation

The base representation in the browser is a Uint8ClampedArray inside an ImageData. Clamping is free, which removes the need for manual bounds checks on assignment. Reading and writing through the same view is fine. Creating a new array per operation is not.

// One allocation, reused across the pipeline.
const scratch = new Float32Array(width * height * 4);

// Reading a channel through a DataView costs more than it saves.
// Prefer direct indexed access on a typed array.
const i = (y * width + x) * 4;
const r = data[i];
const g = data[i + 1];
const b = data[i + 2];

Memory access order

Iterate rows in the outer loop and columns in the inner loop. The reverse order touches a new cache line on every step, and on a large image that turns a linear scan into something closer to random access. A separable vertical blur pass is the classic offender, and transposing between passes is often faster than the strided read it replaces.

Hardware Acceleration

Image operations are embarrassingly parallel, which is exactly what a GPU is for. The catch is transfer cost. Uploading a texture and reading it back can dominate the runtime of a cheap filter, so the GPU wins on long chains and loses on single passes.

Fragment shaders

In WebGL the image becomes a texture, a full-screen quad becomes the geometry, and the filter becomes a fragment shader executed once per output pixel. Neighbourhood access is a texture sample at an offset, and bilinear interpolation between samples is free in hardware.

precision highp float;
uniform sampler2D uImage;
uniform vec2 uTexel;      // 1.0 / resolution
varying vec2 vUv;

void main() {
  vec4 sum = vec4(0.0);
  for (int j = -1; j <= 1; j++) {
    for (int i = -1; i <= 1; i++) {
      vec2 offset = vec2(float(i), float(j)) * uTexel;
      sum += texture2D(uImage, vUv + offset);
    }
  }
  gl_FragColor = sum / 9.0;
}

Figure 8.1

Image prompt

Minimal architecture diagram on a transparent background, three horizontal lanes stacked vertically, each drawn as a thin rounded rectangle outline: main thread, worker thread, GPU. Small labelled-by-shape blocks sit inside each lane: a decode block and a display block on the main lane, two processing blocks on the worker lane, three chained shader blocks on the GPU lane. Thin arrows cross between lanes at four points, with the two crossings that represent expensive transfers drawn in cyan and thicker. No text, flat vector, generous spacing, 16:8.

Copy

Compute shaders and WebGPU

WebGPU replaces the fragment-shader trick with a real compute model. Storage buffers hold the data, workgroups map naturally onto tiles, and shared memory inside a workgroup makes separable filters genuinely efficient. Where support exists it is the better target.

Parallelism and WebAssembly

Not everything belongs on the GPU. Sorting, entropy coding and pointer-heavy algorithms run better on CPU cores, and the browser gives you those through workers.

1. 01

   Move decoding off the main thread

   createImageBitmap in a worker keeps the interface responsive during the most expensive single step in most pipelines.

2. 02

   Render into an OffscreenCanvas

   The canvas can be transferred to a worker, which means the whole processing chain runs without touching the main thread at all.

3. 03

   Transfer, do not copy

   Pass ArrayBuffers as transferables. A structured clone of a sixteen megabyte buffer is a visible pause.

4. 04

   Split by tile, not by channel

   Tiles keep each worker on a contiguous memory region. Splitting by channel makes every worker stride across the whole buffer.

Compiling OpenCV or libvips to WebAssembly brings decades of tuned code into the browser. The cost is a multi-megabyte binary and a copy across the JavaScript boundary in each direction. For one filter it is a poor trade. For a full editing pipeline it is often the right one.

Module 09 covers the case where no kernel is written at all, because the operation is learned from data instead.