Start with the problem, not the technology
It is easy to begin with “I want to add AI” and then hunt for a place to put it. The sounder order is the reverse: describe what the project needs to do, what currently stops it doing that, and only then ask whether a model is the best fix. Many good ideas turn out to need nothing more than a threshold on a sensor value or a lookup table, and those are cheaper, faster and easier to debug.
A useful test is to write the feature as one sentence in the form “When X happens, the device should Y, and I will know it worked because Z.” If you cannot state Z, you cannot evaluate any solution, AI or otherwise.
A five-part way to weigh a feature
A structured AI feature evaluation does not need to be elaborate. Five headings cover most of it.
1. Benefit
Who gains, and by how much? Be honest about whether the improvement is something users would notice or merely something interesting to build. Ask what happens if the feature is simply absent. Compare against a non-AI baseline: a fixed rule, a manual setting, a simple formula. If the baseline gets you most of the way, the extra gain from a model may not justify the added complexity.
2. Cost
Costs come in several kinds, and the first is rarely the largest.
- Build effort: collecting data, choosing or training a model, integrating it.
- Running cost: compute for each use. A local model may need hardware, which is where inference hardware choices come in; a hosted service typically charges per use, and prices change.
- Data cost: labelled examples take time, and good ones are harder to get than expected.
- Opportunity cost: time spent here is time not spent on the rest of the project.
3. Failure modes
Every model is wrong some of the time. The question is what a wrong answer does. Think through the different ways it can fail:
- A false alarm (acting when nothing is there).
- A miss (not acting when something is there).
- A confident but incorrect output, which is a particular risk with language models, as covered under model hallucinations.
- Gradual decline when real-world conditions drift away from what the model was built on.
- Unfair or uneven behaviour across different users or conditions, which links to training data and bias.
Then ask who bears the consequence. An annoying wrong song suggestion is a different matter from a wrong decision about a heater, a lock or a moving machine. For anything involving safety, do not let a model be the only safeguard; use conventional protections and follow relevant standards and manufacturer guidance.
4. Maintenance
AI features are not finished when they ship. Consider:
- Will the model need retraining as conditions change?
- Who will notice if quality quietly drops, and how?
- If you depend on a third-party service, what happens if it changes behaviour, pricing or terms, or disappears?
- If you use pretrained models or libraries, do their software licenses permit your use, and will updates break your code?
- Can you reproduce the result later? Keeping track of the model file, the settings and the data version in version control helps.
5. Data and privacy
Does the feature need personal or sensitive data? Where would it be processed? Sending content to an online service creates obligations and risks, set out in cloud AI privacy, while keeping it on-device may limit which models you can run. Cameras and microphones deserve particular care.
Try the simple alternative first
Before building anything, try the most basic approach that could work. Examples:
- A fixed threshold or moving average on a sensor reading.
- A small state machine that moves between modes using clear conditions.
- Keyword rules or templates instead of a language model for a limited set of commands.
- A manual setting the user can change.
If the simple version works well enough, you have saved effort and gained something more predictable. If it fails in a specific, describable way, you now know exactly what a model needs to improve, which makes the case for AI far stronger and gives you something to measure against.
Decide how you will measure success
Agree on a test before building. Collect a modest set of realistic cases, including awkward ones, write down the correct outcome for each, and score the simple baseline against the AI candidate. Look at the types of errors, not only the total. A system that is right ninety-odd times out of a hundred may still be unacceptable if the remaining errors are the costly kind.
Remember that a demo on friendly examples tells you little. The behaviour that matters shows up in odd lighting, noisy audio, unusual phrasing or unexpected inputs.
Plan for graceful failure
If you do add a model, design around its imperfection:
- Provide a manual override or a clear way to undo an action.
- Show confidence or ask for confirmation when the model is unsure.
- Have a safe default when the model is unavailable, whether the network is down or the service is slow.
- Log inputs and outputs where appropriate and legal, so you can investigate mistakes. Mind the privacy implications of what you store.
A quick decision guide
An AI feature is more likely to be worth it when:
- The task is hard to express as rules but easy to show by example.
- Errors are cheap, reversible, or reviewed by a person.
- You have, or can responsibly gather, representative data.
- A pretrained model already does close to what you need.
- You are prepared to maintain and re-check it.
It is less likely to be worth it when:
- A simple rule solves most of the problem.
- Errors could cause injury, damage or serious unfairness.
- The data is sensitive and the processing location is unclear.
- You cannot say how you would know it works.
Neither list is a verdict. They are prompts for an honest conversation with yourself about whether the feature earns its place, and sometimes the answer is to build the plain version now and revisit AI later, when the project and your understanding of how models are trained and used have both grown.