How Computer Vision Detects Objects

Three related tasks

People often use “computer vision” for several different jobs, and picking the right one saves effort.

• Image classification assigns a single label to a whole picture: “this image contains a dog”.
• Object detection finds each object and says where it is, usually by drawing a rectangle (a bounding box) around it and attaching a label and a confidence score.
• Segmentation goes further and marks which individual pixels belong to each object or category.

This article focuses on detection, the middle option, because it is the one most builders reach for when a project needs to know what is in front of a camera and roughly where.

What a detector outputs

For each image, a detector typically returns a list of candidate objects. Each entry contains a box (position and size), a class label such as “person” or “bottle”, and a confidence score, normally a number between zero and one.

That score is not a probability you can take at face value. It reflects how strongly the model’s internal calculation favoured that label, and a model can be confidently wrong. Treat it as a ranking signal: higher is more likely, but not guaranteed.

How the model learns to see

Early image-processing methods relied on hand-designed rules, such as looking for edges and corners and matching them to templates. Modern detectors instead learn their own features from data, using neural networks, most often convolutional ones or newer attention-based designs.

In rough terms, early layers of the network respond to simple patterns like edges and colour changes. Later layers combine those into textures, parts and shapes, and the final layers decide which combinations suggest an object of a particular class. Nobody programs those features; they emerge from training on many labelled images, as described in how models learn from datasets and parameters.

The labelled examples for detection are more laborious than for classification: each object in each training image needs a box drawn around it and a label assigned.

From pixels to boxes

Detectors differ in design, but most do some version of the following:

1. Pass the image through a network that produces feature maps summarising what is where.
2. Propose many candidate regions, or predict boxes at a grid of positions and sizes.
3. For each candidate, predict a class and a confidence score, and refine the box position.
4. Discard weak candidates using a confidence threshold.
5. Remove duplicates: when several boxes overlap the same object, keep the best one. This step is commonly called non-maximum suppression.

Some designs do everything in a single pass and are generally faster, which suits live video. Others use two stages for potentially higher accuracy at greater computing cost. Which approach fits depends on your hardware; the factors are discussed under inference hardware.

Thresholds are a trade-off

The confidence threshold you set directly shapes behaviour. A low threshold catches more real objects but also produces false alarms, boxes around things that are not there. A high threshold cuts false alarms but misses real objects.

No threshold removes both problems. Choose it based on the cost of each mistake in your project. A feeder that counts visiting birds can tolerate an occasional miss. A system that decides whether a machine guard is clear should not rely on a single camera model at all, and any safety function needs proper engineering and applicable standards, not a hobby detector.

Why detectors get it wrong

Errors usually come from a mismatch between the conditions the model learned from and the conditions it faces. Common causes:

• Lighting: dim scenes, glare, backlighting and strong shadows change how objects look. A model trained mostly on daylight pictures may struggle at night.
• Camera quality and angle: motion blur, low resolution, lens distortion and unusual viewpoints reduce reliability. Objects seen from above may look very different from the side views in training photos.
• Object size: small or distant objects occupy few pixels and are easily missed. Many detectors resize the input image, which can shrink small objects further.
• Occlusion and clutter: partly hidden objects and busy backgrounds make boxes less accurate.
• Look-alikes: similar shapes, patterns or printed pictures of objects can be mistaken for the real thing.
• Unusual examples: objects of types, colours or designs rare in the training set are recognised less reliably.
• Label set limits: a detector can only name classes it was trained on. Show it something unfamiliar and it will tend to assign the nearest known label rather than say “unknown”.

The last two points tie to training data quality and bias. If the training images came mostly from one country, season, camera type or group of people, performance can be uneven elsewhere, and the unevenness is often invisible until you test.

Practical habits for camera projects

• Test with your own images taken by your camera, in your lighting, at your distances. Published accuracy figures come from particular datasets and may not transfer.
• Keep a log of failures and look for patterns, such as every miss happening at dusk.
• Fix the physical setup first. Better lighting, a steadier mount and a closer camera often help more than switching models.
• Consider fine-tuning a pretrained detector on a small set of your own labelled photos if the standard classes do not match your needs, understanding that results depend on the quality and variety of those photos.
• Smooth over time. In video, requiring an object to appear in several consecutive frames before acting reduces flicker and one-frame false alarms.
• Plan for modest hardware. Smaller, compressed detectors can run on single-board computers and some low-power chips, with reduced accuracy.

Privacy and responsibility

A camera that identifies people or records a home raises real questions for the people in view. Think about where images are stored, whether they leave the device, and whether others have agreed to be recorded. These issues are covered in the notes on camera privacy, and if frames are sent to an online service, cloud AI privacy applies as well. Local laws on filming people differ, so check the rules where you live.

When detection is the wrong tool

If you only need to know whether something is present, a simple sensor such as a break-beam or one of the distance sensors used for obstacle avoidance may be cheaper, faster and more dependable than a camera and a neural network. Vision is most valuable when the thing you care about cannot be captured by a simpler measurement. It is worth weighing that honestly, as discussed in judging whether an AI feature is worth adding.