Local AI means that inference and sensitive data processing happen on infrastructure controlled by the organisation. This can be an office server, an edge device or a dedicated environment in a managed private deployment. The important point is not a slogan about locality; it is the ability to define where data travels and who can inspect the system.
Architecture first
Start with the task, input data, target decision and exception path. Then decide whether the job needs generation, retrieval, classification or an external tool. This often reveals that a compact model plus a verified data source is more useful than a large general-purpose endpoint.
Practical uses
Local systems are well suited to document triage, knowledge retrieval, structured extraction, internal assistance and device-side analysis. Each use case needs a quality target and a human escalation path. A system should be able to say that the available evidence is insufficient.
Teams of specialists
A small classifier, a retrieval component and a narrow language model can form a more controllable system than one model handling every step. Components can be benchmarked independently and changed without reworking the whole application.
Conclusion
Success is measurable: lower handling time, fewer errors, clearer evidence and predictable operating cost. Model size is only one variable in that result.