Using the CPU servers we have now<\/h1>\n\n\n\n
Until 2023, graphical processing units (GPUs) were only of interest to video gamers, animators, and mechanical designers. There is now an imperative for GPU resources on most new servers going forward, for local inference and retrieval augmented generation (RAG). However we will need to devise an interim approach to use the CPU-centric servers we have, even for some AI inference tasks, until the capex cycles have refreshed in 3-4 years from now. On a CPU-only system, the system response time for a query can range from 2-5 seconds to 30-40 seconds. This level of performance may be acceptable for some use cases, including scripted tasks for which a 40 second delay is not material. Deploying this solution on a system with even a modest Nvidia GPU will result in dramatic increases in performance.<\/p>\n\n\n\n
Why host an LLM locally<\/h1>\n\n\n\n\n- To learn how LLMs are built<\/li>\n\n\n\n
- To achieve data sovereignty by operating on a private system<\/li>\n\n\n\n
- To save expense by avoiding the need for external LLM vendors<\/li>\n<\/ul>\n\n\n\n
Preparing a computer for deployment<\/h1>\n\n\n\n
This procedure was tested on Ubuntu Server 24.04. Baremetal is better than a virtual machine for this use case, allowing the software to access all of the resources of the host system. In terms of resources, you will need a relatively powerful CPU, like an i7, and 16-32GB of RAM.<\/p>\n\n\n\n
Note: the version of Python required by Open-WebUI is Python 3.12, which is supported by default in Ubuntu Server 24.04 LTS. If you are on an older version of the operating, you can install a newer version of Python using a PPA<\/a>.<\/p>\n\n\n\nDo you need a GPU?<\/h2>\n\n\n\n
No, but a GPU will make your inference noticeably faster. If you have an Nvidia GPU, ensure that you have Nvidia CUDA drivers enabled. If you have an AMD GPU, ensure that you have AMD ROCM drivers. There is some talk of support for Intel GPUs but none of it is yet practical.<\/p>\n\n\n\n
Note: if you have an Nvidia GPU, you may want to consider vLLM<\/a>.<\/p>\n\n\n\nOllama is able to work on a CPU-only system<\/h2>\n\n\n\n
Ollama is able to work on a CPU-only system, and that is what we will implement in this post. Ollama is able to achieve performance that may be acceptable for certain kinds of operations. For example, large batch operations that run overnight, that can accept a 30-60 second delay versus 2-10 seconds for a GPU-driven solution. For some questions, like \u201cwhy is the sky blue?\u2019 an answer will start immediately. For more complex questions, there may be a 5-10 second delay before answering, and the text will arrive slowly enough to remind you of 300 baud modems (for those of you who get that reference). The wonder of a dancing bear is not in how well it dances, but that it dances at all. This level of performance may be acceptable for some use cases, in particular batched operations and programmatic access via a custom function invoking commands sent to the API server.<\/p>\n\n\n\n
Escalating to root using sudo<\/h2>\n\n\n\n
From a shell, enter the following command:<\/p>\n\n\n
\nsudo su\n<\/pre><\/div>\n\n\n(enter the password when requested)<\/p>\n\n\n\n
Opening ports in the UFW firewall<\/h2>\n\n\n\n
You may need to open ports on the UFW firewall to enable the chat client.<\/p>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\nufw allow 11434\/tcp\nufw allow 8080\/tcp\n<\/pre><\/div>\n\n\nEnsuring that the system is up to date<\/h2>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\napt clean\napt update\napt upgrade\napt install curl python3-venv python3-pip ffmpeg\n<\/pre><\/div>\n\n\nA note re RHEL and variants like Fedora and AlmaLinux<\/h2>\n\n\n\n
Although this procedure has not been tested on RHEL and variants like Fedora and AlmaLinux, I looked at the installation script and those platforms are supported. In theory, you could configure an RHEL-type system by using equivalent firewall-cmd and dnf commands.<\/p>\n\n\n\n
Installing Ollama using the installation script<\/h1>\n\n\n\n
Ollama provides an installation script that automates the installation. From a shell as root, enter the following command:<\/p>\n\n\n
\ncurl -fsSL https:\/\/ollama.com\/install.sh | sh\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-21.png\")
<\/figure>\n\n\n\nPulling the mistral image<\/h2>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nollama pull mistral\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-7.png\")
<\/figure>\n\n\n\nListing the images available<\/h2>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nollama list\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-8.png\")
<\/figure>\n\n\n\nTesting Ollama and the LLM using the command line<\/h2>\n\n\n\n
Enter the following command. Test the chat interface on the command line in the shell:<\/p>\n\n\n
\nollama run mistral\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-9.png\")
<\/figure>\n\n\n\nTesting the API server using curl<\/h2>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\nsystemctl restart ollama\nsystemctl status ollama\nsystemctl enable ollama\ncurl http:\/\/localhost:11434\/api\/generate -d '{\n"model": "mistral",\n"prompt":"Why is the sky blue?"\n}'\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-22.png\")
<\/figure>\n\n\n\nEnter the following command:<\/p>\n\n\n
\ncurl http:\/\/localhost:11434\/api\/chat -d '{\n"model": "mistral",\n"messages": [\n{ "role": "user", "content": "why is the sky blue?" }\n]\n}'\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-11.png\")
<\/figure>\n\n\n\nPreparing the system for Open-WebUI<\/h1>\n\n\n\n
To prepare the system for Open-WebUI, we must create a working directory, and create a venv (virtual python environment).<\/p>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\ncd ~\npwd\nmkdir ollamatmp\ncd ollamatmp\npython3 -m venv ollama_env\nsource ollama_env\/bin\/activate\npip install open-webui\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-24.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-13.png\")
<\/figure>\n\n\n\nStarting the open-webui serve process<\/h2>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nopen-webui serve\n<\/pre><\/div>\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-25.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-16.png\")
<\/figure>\n\n\n\nVisiting the Open-WebUI web page interface<\/h2>\n\n\n\n
Using a web browser, visit this address:<\/p>\n\n\n\n
http:\/\/127.0.0.1:8080<\/a><\/p>\n\n\n\nYou will be prompted to create an admin account:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-17.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-18-1024x606.png\")
<\/figure>\n\n\n\nUsing the Open-WebUI chat interface<\/h2>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-19-1024x623.png\")
<\/figure>\n\n\n\n
This window took 30 seconds to begin showing its answer, then another 20 seconds to complete generating the answer:<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.gordonbuchan.com\/blog\/wp-content\/uploads\/2025\/01\/image-20-1024x606.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
Using nginx as a proxy to expose the API port to the local network<\/h1>\n\n\n\n
By default, the Ollama API server answers on port 11434 but only on the local address 127.0.0.1. You can use nginx as a proxy to expose the API to the local network. Enter the following commands:<\/p>\n\n\n
\nufw allow 8085\/tcp\napt install nginx\ncd \/etc\/nginx\/sites-enabled\nnano default\n<\/pre><\/div>\n\n\nUse the nano editor to add the following text:<\/p>\n\n\n
\nserver {\n listen 8085;\n\n location \/ {\n proxy_pass http:\/\/127.0.0.1:11434; # Replace with your Ollama API port\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection 'upgrade';\n proxy_set_header Host $host;\n proxy_cache_bypass $http_upgrade;\n\n # Optional: Add timeout settings for long-running API calls\n proxy_connect_timeout 60s;\n proxy_read_timeout 60s;\n proxy_send_timeout 60s;\n }\n}\n<\/pre><\/div>\n\n\n<\/p>\n\n\n\n
Save and exit the file.<\/p>\n\n\n\n
Enter this command:<\/p>\n\n\n
\nsystemctl restart nginx\n<\/pre><\/div>\n\n\nTesting the exposed API port from another computer<\/h2>\n\n\n\n
From another computer, enter the command (where xxx.xxx.xxx.xxx is the IP address of the computer hosting the Ollama API server):<\/p>\n\n\n
\ncurl http:\/\/xxx.xxx.xxx.xxx:8085\/api\/chat -d '{\n"model": "mistral",\n"messages": [\n{ "role": "user", "content": "why is the sky blue?" }\n]\n}'\n<\/pre><\/div>\n\n\nCreating a systemd service to start the Open-WebUI chat interface automatically<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nnano \/etc\/systemd\/system\/open-webui.service\n<\/pre><\/div>\n\n\nUse the nano editor to add the following text:<\/p>\n\n\n
\n[Unit]\nDescription=Open-WebUI Service\nAfter=network.target\n\n[Service]\nUser=root\nWorkingDirectory=\/root\/ollamatmp\/ollama_env\nExecStart=\/usr\/bin\/bash -c "source \/root\/ollamatmp\/ollama_env\/bin\/activate && open-webui serve"\nRestart=always\nEnvironment=PYTHONUNBUFFERED=1\nStandardOutput=journal\nStandardError=journal\n\n[Install]\nWantedBy=multi-user.target\n\n<\/pre><\/div>\n\n\nSave and exit the file.<\/p>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\nsystemctl daemon-reload\nsystemctl start open-webui\nsystemctl enable open-webui\nsystemctl status open-webui\n<\/pre><\/div>\n\n\nConclusion<\/h1>\n\n\n\n
You now have an LLM API server, and a web chat for interactive access.<\/p>\n\n\n\n
References<\/h1>\n\n\n\n
https:\/\/ollama.com<\/a><\/p>\n\n\n\nhttps:\/\/github.com\/ollama\/ollama<\/a><\/p>\n\n\n\nA related post<\/h1>\n\n\n\n
Preparing a computer for deployment<\/h1>\n\n\n\n
This procedure was tested on Ubuntu Server 24.04. Baremetal is better than a virtual machine for this use case, allowing the software to access all of the resources of the host system. In terms of resources, you will need a relatively powerful CPU, like an i7, and 16-32GB of RAM.<\/p>\n\n\n\n
Note: the version of Python required by Open-WebUI is Python 3.12, which is supported by default in Ubuntu Server 24.04 LTS. If you are on an older version of the operating, you can install a newer version of Python using a PPA<\/a>.<\/p>\n\n\n\n No, but a GPU will make your inference noticeably faster. If you have an Nvidia GPU, ensure that you have Nvidia CUDA drivers enabled. If you have an AMD GPU, ensure that you have AMD ROCM drivers. There is some talk of support for Intel GPUs but none of it is yet practical.<\/p>\n\n\n\n Note: if you have an Nvidia GPU, you may want to consider vLLM<\/a>.<\/p>\n\n\n\n Ollama is able to work on a CPU-only system, and that is what we will implement in this post. Ollama is able to achieve performance that may be acceptable for certain kinds of operations. For example, large batch operations that run overnight, that can accept a 30-60 second delay versus 2-10 seconds for a GPU-driven solution. For some questions, like \u201cwhy is the sky blue?\u2019 an answer will start immediately. For more complex questions, there may be a 5-10 second delay before answering, and the text will arrive slowly enough to remind you of 300 baud modems (for those of you who get that reference). The wonder of a dancing bear is not in how well it dances, but that it dances at all. This level of performance may be acceptable for some use cases, in particular batched operations and programmatic access via a custom function invoking commands sent to the API server.<\/p>\n\n\n\n From a shell, enter the following command:<\/p>\n\n\n (enter the password when requested)<\/p>\n\n\n\n You may need to open ports on the UFW firewall to enable the chat client.<\/p>\n\n\n\n Enter the following commands:<\/p>\n\n\n Enter the following commands:<\/p>\n\n\n Although this procedure has not been tested on RHEL and variants like Fedora and AlmaLinux, I looked at the installation script and those platforms are supported. In theory, you could configure an RHEL-type system by using equivalent firewall-cmd and dnf commands.<\/p>\n\n\n\n Ollama provides an installation script that automates the installation. From a shell as root, enter the following command:<\/p>\n\n\n Enter the following command:<\/p>\n\n\n Enter the following command:<\/p>\n\n\n Enter the following command. Test the chat interface on the command line in the shell:<\/p>\n\n\n Enter the following commands:<\/p>\n\n\n Enter the following command:<\/p>\n\n\n To prepare the system for Open-WebUI, we must create a working directory, and create a venv (virtual python environment).<\/p>\n\n\n\n Enter the following commands:<\/p>\n\n\n Enter the following command:<\/p>\n\n\n Using a web browser, visit this address:<\/p>\n\n\n\n http:\/\/127.0.0.1:8080<\/a><\/p>\n\n\n\n You will be prompted to create an admin account:<\/p>\n\n\n\n <\/p>\n\n\n\n By default, the Ollama API server answers on port 11434 but only on the local address 127.0.0.1. You can use nginx as a proxy to expose the API to the local network. Enter the following commands:<\/p>\n\n\n Use the nano editor to add the following text:<\/p>\n\n\n <\/p>\n\n\n\n Save and exit the file.<\/p>\n\n\n\n Enter this command:<\/p>\n\n\n From another computer, enter the command (where xxx.xxx.xxx.xxx is the IP address of the computer hosting the Ollama API server):<\/p>\n\n\n Enter the following command:<\/p>\n\n\n Use the nano editor to add the following text:<\/p>\n\n\n Save and exit the file.<\/p>\n\n\n\n Enter the following commands:<\/p>\n\n\n You now have an LLM API server, and a web chat for interactive access.<\/p>\n\n\n\n https:\/\/ollama.com<\/a><\/p>\n\n\n\n https:\/\/github.com\/ollama\/ollama<\/a><\/p>\n\n\n\nDo you need a GPU?<\/h2>\n\n\n\n
Ollama is able to work on a CPU-only system<\/h2>\n\n\n\n
Escalating to root using sudo<\/h2>\n\n\n\n
\nsudo su\n<\/pre><\/div>\n\n\n
Opening ports in the UFW firewall<\/h2>\n\n\n\n
\nufw allow 11434\/tcp\nufw allow 8080\/tcp\n<\/pre><\/div>\n\n\n
Ensuring that the system is up to date<\/h2>\n\n\n\n
\napt clean\napt update\napt upgrade\napt install curl python3-venv python3-pip ffmpeg\n<\/pre><\/div>\n\n\n
A note re RHEL and variants like Fedora and AlmaLinux<\/h2>\n\n\n\n
Installing Ollama using the installation script<\/h1>\n\n\n\n
\ncurl -fsSL https:\/\/ollama.com\/install.sh | sh\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\nPulling the mistral image<\/h2>\n\n\n\n
\nollama pull mistral\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\nListing the images available<\/h2>\n\n\n\n
\nollama list\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\nTesting Ollama and the LLM using the command line<\/h2>\n\n\n\n
\nollama run mistral\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\nTesting the API server using curl<\/h2>\n\n\n\n
\nsystemctl restart ollama\nsystemctl status ollama\nsystemctl enable ollama\ncurl http:\/\/localhost:11434\/api\/generate -d '{\n"model": "mistral",\n"prompt":"Why is the sky blue?"\n}'\n<\/pre><\/div>\n\n\n<\/figure>\n\n\n\n\ncurl http:\/\/localhost:11434\/api\/chat -d '{\n"model": "mistral",\n"messages": [\n{ "role": "user", "content": "why is the sky blue?" }\n]\n}'\n<\/pre><\/div>\n\n\n<\/figure>\n\n\n\nPreparing the system for Open-WebUI<\/h1>\n\n\n\n
\ncd ~\npwd\nmkdir ollamatmp\ncd ollamatmp\npython3 -m venv ollama_env\nsource ollama_env\/bin\/activate\npip install open-webui\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\n<\/figure>\n\n\n\nStarting the open-webui serve process<\/h2>\n\n\n\n
\nopen-webui serve\n<\/pre><\/div>\n\n\n
<\/figure>\n\n\n\n<\/figure>\n\n\n\nVisiting the Open-WebUI web page interface<\/h2>\n\n\n\n
<\/figure>\n\n\n\n<\/figure>\n\n\n\nUsing the Open-WebUI chat interface<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
This window took 30 seconds to begin showing its answer, then another 20 seconds to complete generating the answer:<\/p>\n\n\n\n<\/figure>\n\n\n\nUsing nginx as a proxy to expose the API port to the local network<\/h1>\n\n\n\n
\nufw allow 8085\/tcp\napt install nginx\ncd \/etc\/nginx\/sites-enabled\nnano default\n<\/pre><\/div>\n\n\n
\nserver {\n listen 8085;\n\n location \/ {\n proxy_pass http:\/\/127.0.0.1:11434; # Replace with your Ollama API port\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection 'upgrade';\n proxy_set_header Host $host;\n proxy_cache_bypass $http_upgrade;\n\n # Optional: Add timeout settings for long-running API calls\n proxy_connect_timeout 60s;\n proxy_read_timeout 60s;\n proxy_send_timeout 60s;\n }\n}\n<\/pre><\/div>\n\n\n\nsystemctl restart nginx\n<\/pre><\/div>\n\n\n
Testing the exposed API port from another computer<\/h2>\n\n\n\n
\ncurl http:\/\/xxx.xxx.xxx.xxx:8085\/api\/chat -d '{\n"model": "mistral",\n"messages": [\n{ "role": "user", "content": "why is the sky blue?" }\n]\n}'\n<\/pre><\/div>\n\n\nCreating a systemd service to start the Open-WebUI chat interface automatically<\/h1>\n\n\n\n
\nnano \/etc\/systemd\/system\/open-webui.service\n<\/pre><\/div>\n\n\n
\n[Unit]\nDescription=Open-WebUI Service\nAfter=network.target\n\n[Service]\nUser=root\nWorkingDirectory=\/root\/ollamatmp\/ollama_env\nExecStart=\/usr\/bin\/bash -c "source \/root\/ollamatmp\/ollama_env\/bin\/activate && open-webui serve"\nRestart=always\nEnvironment=PYTHONUNBUFFERED=1\nStandardOutput=journal\nStandardError=journal\n\n[Install]\nWantedBy=multi-user.target\n\n<\/pre><\/div>\n\n\n
\nsystemctl daemon-reload\nsystemctl start open-webui\nsystemctl enable open-webui\nsystemctl status open-webui\n<\/pre><\/div>\n\n\n
Conclusion<\/h1>\n\n\n\n
References<\/h1>\n\n\n\n
A related post<\/h1>\n\n\n\n