What does running a model locally mean?
A local model runs on a computer or server configured for the task. Employees may use an application on their own workstation or an interface shared over the company network. The device displaying the interface does not have to be the device running the model.
Check the whole workflow. An application with a local model may still connect to external search or other services. Ollama’s documentation distinguishes local and cloud features and explains how to disable cloud features. Installing an application does not prove that every part works offline.
If working without internet access is a requirement, state this explicitly and include it in your tests. Distinguish between downloading the software and model during preparation and performing the task later.
When should you consider a local environment?
Reasons may include the need to work with internal materials, infrastructure control, or the requirement to operate on a specific network. These are grounds for analysis, not automatic proof that the local option will be the best.
Define the task and users. A single person attempting to summarise short notes is different from a shared service searching documents across multiple departments. The latter also includes access to materials, the task queue, and application maintenance.
If you only draft text occasionally, compare a local setup with a hosted service. If company requirements call for a dedicated environment, still check whether the chosen model meets your quality expectations.
First, try it on a real task
Prepare documents and questions that reflect the company’s work. Use material approved for testing and follow the relevant access rules. Describe what a correct result should look like for each task.
When summarising, check for the correctness of terms, names, and numbers. For responses from documents, assess consistency with the source and the response to missing information. Add similar documents from different periods to ensure the results don't confuse outdated findings with current ones.
For example, use a fictional complaints-handling manual. Ask the model to identify the required documents and cite the relevant section. If the manual does not cover a case, it should say the answer is missing. This is a testable objective; asking whether a model “communicates well” is not enough to assess its suitability for a business.
What determines the hardware you need?
Hardware requirements depend on the model, input length, runtime and workload. Ollama’s documentation explains that larger context windows and more parallel requests increase memory requirements. A single-question test cannot represent several people working with long documents.
Determine acceptable wait times and the number of concurrent users. Your sample should meet these requirements. If your team works on longer content in the morning, consider testing this scenario.
I don't offer a single, universal configuration: it could be too weak or unnecessarily expensive. The purchase should be based on the task at hand and the test results.
Corporate documents require more than just a model
The model doesn't automatically become the company's current knowledge base. In an application that works with documents, it's necessary to determine where the materials are sourced, how the fragments are selected for response, and who is responsible for ensuring the sources are up to date.
Permissions are equally important. A user shouldn't be granted access to material simply because it's been added to a shared collection. The design must consider who can see the document and the response based on it.
A link to the source helps verify the result, but it's important to verify that the fragment you're referencing actually supports it. If the program also needs to perform actions, the distinction described in the guide on what an AI agent is will help.
Running locally still requires administration
A local environment needs authentication, network access controls and an update process. Agree where documents, conversation history and results are stored. Having a device in the office does not resolve these questions.
Who will restore the configuration after a failure? Who will check the application after a model change? Who will add a new employee and revoke access from someone who leaves? These are practical elements of everyday system use.
Keep the test set from the pilot. After a significant change, you can reassess quality using the same examples. The decision to update should be based on the results of the work, not solely on the availability of a newer version.
How do you compare the total cost?
Compare the full costs: hardware, setup, electricity, administration, application maintenance and changes over time. For hosted services, include fees, restrictions and data-use terms. Compare options that can perform the same task to an acceptable standard.
No per-query fee does not mean there are no costs. A local environment may also meet business requirements that a simple price comparison misses.
Before we talk, prepare a description of the task, the types of material, number of users, current hardware and network requirements. I can help define a trial to inform your implementation decision.
Questions and Answers
Does the local model always work without internet?
This depends on the overall configuration. Computing can occur locally, but additional tools or selected services may use the network. Offline functionality must be verified separately.
Can I start on my current computer?
That can be assessed once the task and hardware specifications are clear. A successful test by one person does not prove that the setup can support a whole team at once.
Sources and further reading
From knowledge to action
See how I can help with your project.
