In this blog post, we create a proxy server that enables the Open-WebUI chat server to connect to an OpenAI-compatible API. Although it is an interesting technical exercise to self-host, as a business case it does not make sense for long-term production. Certain kinds of LLM inference workloads can be handled by a CPU-only system, using a tool like Ollama, but the performance is not sufficient for real-time interaction. Dedicating GPU-enabled hardware is a significant expense, whether it be the acquisition of dedicated GPU hardware such as an A30, H100, RTX 4090, or RTX 5090 card. Renting or leasing this hardware is even more expensive. We seem to be heading into an era in which LLM inference itself is software as a service (SaaS), unless there are specific reasons why inference data cannot be shared with a public cloud, such as a legal or medical application.<\/p>\n\n\n\n
There are many chat user interfaces available, but Open-WebUI has been easier to deploy and for the moment is my preference. The need for proxy servers to translate between LLM API servers that have slightly different protocols will likely be with us for some time, until LLM APIs have matured and become more compatible.<\/p>\n\n\n\n
At this time in 2025, most LLM inference APIs emulate the OpenAI protocol, with support for the chat completions API and the use of a message array. In this exercise, we will be connecting the Open-WebUI chat server to an OpenAI compatible LLM API. In the future, we may see more abstracted toolchains, for example, a retrieval augmented generation (RAG) server offering an API that encapsulates the local RAG functionality and enhanced by the remote LLM inference API, to which a chat server will connect. In this case, the chat server will be Open-WebUI, but in other applications it could be a web chat user interface embedded in a website.<\/p>\n\n\n\n
\nsudo su\n<\/pre><\/div>\n\n\nCreating a virtual environment and installing dependencies<\/h1>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\ncd ~\nmkdir proxy_workdir\ncd proxy_workdir\npython3 -m venv proxy_env\nsource proxy_env\/bin\/activate\npip install fastapi uvicorn httpx python-dotenv\n<\/pre><\/div>\n\n\nCreating the proxy.py file<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nnano proxy.py\n<\/pre><\/div>\n\n\nUse the nano editor to add the following text:<\/p>\n\n\n
\n# MIT license Gordon Buchan 2025\n# see https:\/\/opensource.org\/license\/mit\n# Some of this code was generated with the assistance of AI tools.\n\nfrom fastapi import FastAPI, Request\nimport httpx\nimport logging\nimport json\nimport time\n\napp = FastAPI()\n\n# Enable logging for debugging\nlogging.basicConfig(level=logging.DEBUG)\n\n# LemonFox API details\nLEMONFOX_API_URL = "https:\/\/api.lemonfox.ai\/v1\/chat\/completions"\nAPI_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"\n\n@app.get("\/api\/openai\/v1\/models")\nasync def get_models():\n return {\n "object": "list",\n "data": [\n {\n "id": "mixtral-chat",\n "object": "model",\n "owned_by": "lemonfox"\n }\n ]\n }\n\nasync def make_request_with_retry(payload):\n """Send request to LemonFox API with one retry in case of failure."""\n headers = {\n "Authorization": f"Bearer {API_KEY}",\n "Content-Type": "application\/json",\n }\n \n for attempt in range(2): # Try twice before failing\n async with httpx.AsyncClient() as client:\n try:\n response = await client.post(LEMONFOX_API_URL, json=payload, headers=headers)\n response_json = response.json()\n \n # If response is valid, return it\n if "choices" in response_json and response_json["choices"]:\n return response_json\n \n logging.warning(f"\u274c Empty response from LemonFox on attempt {attempt + 1}: {response_json}")\n\n except httpx.HTTPStatusError as e:\n logging.error(f"\u274c LemonFox API HTTP error: {e}")\n except json.JSONDecodeError:\n logging.error(f"\u274c LemonFox returned an invalid JSON response: {response.text}")\n \n # Wait 1 second before retrying\n time.sleep(1)\n\n # If we get here, both attempts failed\u2014return a default response\n logging.error("\u274c LemonFox API failed twice. Returning a fallback response.")\n return {\n "id": "fallback-response",\n "object": "chat.completion",\n "created": int(time.time()),\n "model": "unknown",\n "choices": [\n {\n "index": 0,\n "message": {\n "role": "assistant",\n "content": "I'm sorry, but I couldn't generate a response. Try again."\n },\n "finish_reason": "stop"\n }\n ],\n "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}\n }\n\n@app.post("\/api\/openai\/v1\/chat\/completions")\nasync def proxy_chat_completion(request: Request):\n """Ensure Open WebUI's request is converted and always return a valid response."""\n try:\n payload = await request.json()\n logging.debug("\ud83d\udfe2 Open WebUI Request: %s", json.dumps(payload, indent=2))\n\n # Convert `prompt` into OpenAI's `messages[]` format\n if "prompt" in payload:\n payload["messages"] = [{"role": "user", "content": payload["prompt"]}]\n del payload["prompt"]\n elif "messages" not in payload or not isinstance(payload["messages"], list):\n logging.error("\u274c Open WebUI sent an invalid request!")\n return {"error": "Invalid request format. Expected `messages[]` or `prompt`."}\n\n # Force disable streaming\n payload["stream"] = False\n\n # Set max tokens to a high value to avoid truncation\n payload.setdefault("max_tokens", 4096)\n\n # Call LemonFox with retry logic\n response_json = await make_request_with_retry(payload)\n\n # Ensure response follows OpenAI format\n if "choices" not in response_json or not response_json["choices"]:\n logging.error("\u274c LemonFox returned an empty `choices[]` array after retry!")\n response_json["choices"] = [\n {\n "index": 0,\n "message": {\n "role": "assistant",\n "content": "I'm sorry, but I didn't receive a valid response."\n },\n "finish_reason": "stop"\n }\n ]\n\n logging.debug("\ud83d\udfe2 Final Response Sent to Open WebUI: %s", json.dumps(response_json, indent=2))\n return response_json\n\n except Exception as e:\n logging.error("\u274c Unexpected Error in Proxy: %s", str(e))\n return {"error": str(e)}\n\n<\/pre><\/div>\n\n\nSave and exit the file.<\/p>\n\n\n\n
Running the proxy server manually<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nuvicorn proxy:app --host 0.0.0.0 --port 8000\n<\/pre><\/div>\n\n\nConfiguring Open-WebUI<\/h1>\n\n\n\n
Go to Open-WebUI Settings | Connections<\/p>\n\n\n\n
Set API Base URL to:<\/p>\n\n\n
\nhttp:\/\/localhost:8000\/api\/openai\/v1\n<\/pre><\/div>\n\n\nEnsure that model name matches:<\/p>\n\n\n\n
mixtral-chat<\/p>\n\n\n\n
Testing Open-WebUI with a simple message<\/h2>\n\n\n\n
Enter some text in the chat window and see if you get a response from the LLM.<\/p>\n\n\n\n
Creating the systemd service<\/h1>\n\n\n\n
Enter the following command: <\/p>\n\n\n
\nnano \/etc\/systemd\/system\/open-webui-proxy.service\n<\/pre><\/div>\n\n\nUse the nano editor to add the following text:<\/p>\n\n\n
\n[Unit]\nDescription=open-webui Proxy for Open WebUI and LLM API\nAfter=network.target\n\n[Service]\nType=simple\nWorkingDirectory=\/root\/proxy_workdir # Change to your script's location\nExecStart=\/usr\/bin\/env bash -c "source \/root\/proxy_workdir\/proxy_env\/bin\/activate && uvicorn proxy:app --host 0.0.0.0 --port 8000"\nRestart=always\nRestartSec=5\nUser=root\nGroup=root\nStandardOutput=journal\nStandardError=journal\n\n[Install]\nWantedBy=multi-user.target\n<\/pre><\/div>\n\n\nSave and exit the file.<\/p>\n\n\n\n
Enabling and starting the systemd service<\/h1>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\nsystemctl daemon-reload\nsystemctl enable open-webui-proxy.service\nsystemctl start open-webui-proxy.service\n<\/pre><\/div>\n\n\nChecking the status of the systemd service<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nsystemctl status open-webui-proxy.service\n<\/pre><\/div>\n\n\nCreating the systemd timer<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nnano open-webui-proxy.timer\n<\/pre><\/div>\n\n\nUse the nano editor to add the following text:<\/p>\n\n\n
\n[Unit]\nDescription=Periodic restart of open-webui Proxy\n\n[Timer]\nOnBootSec=1min\nOnUnitActiveSec=1h\nUnit=open-webui-proxy.service\n\n[Install]\nWantedBy=timers.target\n<\/pre><\/div>\n\n\nSave and exit the file.<\/p>\n\n\n\n
Enabling and starting the systemd timer<\/h1>\n\n\n\n
Enter the following commands:<\/p>\n\n\n
\nsystemctl daemon-reload\nsystemctl enable open-webui-proxy.timer\nsystemctl start open-webui-proxy.timer\n<\/pre><\/div>\n\n\nChecking the systemd timer<\/h1>\n\n\n\n
Enter the following command:<\/p>\n\n\n
\nsystemctl list-timers --all | grep open-webui-proxy\n<\/pre><\/div>\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Open-WebUI, a web chat server for LLMs, is not compatible with some LLM APIs that support the chat completions API, and use a message array. Although there are other tools available, I wanted to use Open-WebUI. I resolved this by creating a proxy server that acts as a translation layer between the Open-WebUI chat server … <\/p>\n
Continue reading “Creating a proxy server to allow Open-WebUI to connect to a large language model (LLM) application programming interface (API) with chat completions and a message array”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5339","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/5339","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=5339"}],"version-history":[{"count":36,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/5339\/revisions"}],"predecessor-version":[{"id":5381,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/5339\/revisions\/5381"}],"wp:attachment":[{"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=5339"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=5339"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.gordonbuchan.com\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=5339"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}