Image Processing Handbook
    Module 0917 min read

    Neural Methods

    The operations that are learned rather than written. What they do well, what they invent, and where a hand-written kernel is still the better answer.

    A convolutional network is a stack of the same operation from Module 03, with the kernel weights found by optimisation instead of by design. That continuity matters. Neural methods did not replace the classical toolkit, they extended the part of it that was already dominant.

    Classical Filters against Deep Learning

    The dividing line is whether the task has a correct answer that can be written down. Rotating an image by twelve degrees has one. Deciding which pixels belong to a cat does not.

    PropertyClassical operatorLearned model
    DeterminismIdentical output every runIdentical only with fixed weights and seed.
    ExplainabilityThe kernel is the explanationBehaviour inferred from evaluation.
    CostMicroseconds to millisecondsMilliseconds to seconds, often on a GPU.
    Failure modePredictable artefactsPlausible but incorrect content.
    Best atWell-specified transformsTasks defined by examples.

    Super-Resolution

    Classical upscaling interpolates between known samples and is therefore limited to what a smooth function can express. Learned upscalers hallucinate texture that matches the statistics of real images at the target scale, which is why they look sharp where bicubic looks soft.

    GAN based

    ESRGAN and successors

    An adversarial loss pushes output toward realistic texture. Very sharp results, occasional invented detail that was never in the source.

    Transformer based

    SwinIR and relatives

    Attention over shifted windows captures longer-range structure. More faithful than GAN approaches, and heavier to run.

    Diffusion based

    Latent upscalers

    Highest perceptual quality on difficult inputs, at the cost of many sampling steps and weak guarantees of fidelity.

    Classical

    Lanczos and friends

    Still the right answer when the output must be an interpolation of the input and nothing else. Instant, and honest about what it does not know.

    Learned 4xLearned 4x
    Lanczos 4xLanczos 4x
    The same low resolution source upscaled four times by interpolation and by a learned model.
    Image prompt

    Two versions of the same upscaled image for a before and after slider. Subject: a close crop of a weathered wooden shutter with peeling paint and a small ceramic tile, originally low resolution. Version A: interpolated upscale, soft edges, smeared paint texture, no fine grain. Version B: identical framing from a learned upscale, crisp paint flake edges, invented but plausible wood grain, slightly waxy surface texture. Photographic, identical composition and colour between the two, no text, 16:9.

    Segmentation and Background Removal

    Segmentation assigns a class or an instance to every pixel. U-Net established the encoder and decoder shape with skip connections that carry spatial detail past the bottleneck, and most architectures since are variations on it.

    Promptable models such as Segment Anything changed the interface rather than the output. A click, a box, or a coarse mask becomes the query, and the model returns a boundary. For background removal that turns a per-image training problem into a per-image interaction.

    Figure 9.1
    A U-shaped network diagram with four descending blocks, a bottleneck, four ascending blocks, and dashed skip connections across the middle
    The encoder and decoder structure, with skip connections carrying detail across the bottleneck.
    Image prompt

    Minimal symmetric network diagram on a transparent background shaped like a U. Left descending arm: four rectangles decreasing in height, connected by downward arrows. Bottom: one small rectangle. Right ascending arm: four rectangles increasing in height, connected by upward arrows. Four horizontal dashed cyan arrows connect each left rectangle to its counterpart on the right. All shapes drawn as thin 1.5px grey outlines with no fill, no text, flat vector, generous negative space, 16:8.

    Inpainting

    Inpainting fills a masked region with content consistent with its surroundings. Classical methods propagate structure inward from the boundary and work well on small scratches and dust. They fail as soon as the hole is large enough to require an object rather than a texture.

    Diffusion based inpainting generates the region conditioned on the rest of the image, which handles object removal convincingly. The mask edge is where quality is decided: feather it, and dilate slightly beyond the object to include its shadow and reflection.

    Deployment notes

    RuntimeWhere it runsTrade-off
    ONNX Runtime WebBrowser, Wasm or WebGL backendWide model support, large bundle.
    TensorFlow.jsBrowser, WebGL or WebGPUMature tooling, uneven operator coverage.
    Transformers.jsBrowser, Wasm and WebGPUSimple API for common vision models.
    Server inferenceRemote GPUNo client cost, but latency and privacy questions.

    Whichever runtime is chosen, the pre and post processing stages remain classical image processing. Resizing to the model input size, normalising channels, converting colour spaces, and cleaning the output mask are all operations from the earlier modules. The network is one stage in a pipeline, not a replacement for it.