tech-envision commited on
Commit
e9772d2
·
1 Parent(s): c56348f

Improve README readability

Browse files
Files changed (1) hide show
  1. README.md +34 -92
README.md CHANGED
@@ -1,103 +1,53 @@
1
  # llm-backend
2
 
3
- This project provides a simple async interface to interact with an Ollama model
4
- and demonstrates basic tool usage. Chat histories are stored in a local SQLite
5
- database using Peewee. Histories are persisted per user and session so
6
- conversations can be resumed with context. One example tool is included:
7
-
8
- * **execute_terminal** – Executes a shell command inside a persistent Linux VM
9
- with network access. Use it to read uploaded documents under ``/data``, fetch
10
- web content via tools like ``curl`` or run any other commands. The assistant
11
- must invoke this tool to search online when unsure about a response. Output
12
- from ``stdout`` and ``stderr`` is captured when each command finishes.
13
- The output string is capped at the last 10,000 characters so very long
14
- results are truncated. A short notice is prepended whenever data is hidden.
15
- Execution happens asynchronously so the assistant can continue responding
16
- while the command runs.
17
- The VM is created when a chat session starts and reused for all subsequent
18
- tool calls. When ``PERSIST_VMS`` is enabled (default), each user keeps the
19
- same container across multiple chat sessions and across application restarts,
20
- so any installed packages and filesystem changes remain available. The
21
- environment includes Python and ``pip`` so complex tasks can be scripted using
22
- Python directly inside the terminal.
23
-
24
- Sessions share state through an in-memory registry so that only one generation
25
- can run at a time. Messages sent while a response is being produced are
26
- ignored unless the assistant is waiting for a tool result—in that case the
27
- pending response is cancelled and replaced with the new request.
28
-
29
- The application injects a robust system prompt on each request. The prompt
30
- guides the model to plan tool usage, execute commands sequentially and
31
- verify results before replying. When the assistant is uncertain, it is directed
32
- to search the internet with ``execute_terminal`` before giving a final answer.
33
- The prompt is **not** stored in the chat history but is provided at runtime so
34
- the assistant can orchestrate tool calls in sequence to fulfil the user's
35
- request reliably. It also directs the assistant to avoid technical jargon so
36
- responses are easy for anyone to understand. If a user message ends with
37
- ``/think`` it simply selects an internal reasoning mode and should be stripped
38
- from the prompt before processing.
39
-
40
- ## Usage
41
 
42
  ```bash
43
  python run.py
44
  ```
45
 
46
- The script will instruct the model to run a simple shell command and print the result. Conversations are automatically persisted to `chat.db` and are now associated with a user and session.
47
 
48
- Uploaded files are stored under the `uploads` directory and mounted inside the VM at `/data`. Call ``upload_document`` on the chat session to make a file available to the model:
49
 
50
  ```python
51
  async with ChatSession() as chat:
52
- path_in_vm = chat.upload_document("path/to/file.pdf")
53
- async for part in chat.chat_stream(f"Summarize {path_in_vm}"):
54
  print(part)
55
  ```
56
 
57
- When using the Discord bot, attach one or more text files to a message to
58
- upload them automatically. The bot responds with the location of each document
59
- inside the VM so they can be referenced in subsequent prompts.
60
-
61
  ## Discord Bot
62
 
63
- Create a `.env` file with your Discord token:
64
 
65
- ```bash
66
- DISCORD_TOKEN="your-token"
67
- ```
 
68
 
69
- Then start the bot:
 
 
70
 
71
- ```bash
72
- python -m bot
73
- ```
74
-
75
- Any attachments sent to the bot are uploaded to the VM and the bot replies with
76
- their paths so they can be used in later messages.
77
 
78
  ## VM Configuration
79
 
80
- The Linux VM used for tool execution runs inside a Docker container. By default
81
- it pulls the image defined by the ``VM_IMAGE`` environment variable, falling
82
- back to ``python:3.11-slim``. This base image includes Python and ``pip`` so
83
- packages can be installed immediately. The container has network access enabled
84
- which allows fetching additional dependencies as needed.
85
-
86
- When ``PERSIST_VMS`` is ``1`` (default), containers are kept around and reused
87
- across application restarts. Each user is assigned a stable container name, so
88
- packages installed or files created inside the VM remain available the next
89
- time the application starts. Set ``VM_STATE_DIR`` to specify the host directory
90
- used for per-user persistent storage mounted inside the VM at ``/state``.
91
- Set ``PERSIST_VMS=0`` to revert to the previous behaviour where containers are
92
- stopped once no sessions are using them.
93
 
94
- To use a fully featured Ubuntu environment, build a custom Docker image and set
95
- ``VM_IMAGE`` to that image. An example ``docker/Dockerfile.vm`` is provided:
96
 
97
  ```Dockerfile
98
  FROM ubuntu:22.04
99
-
100
- # Install core utilities and Python
101
  RUN apt-get update && \
102
  apt-get install -y --no-install-recommends \
103
  python3 \
@@ -107,7 +57,6 @@ RUN apt-get update && \
107
  git \
108
  build-essential \
109
  && rm -rf /var/lib/apt/lists/*
110
-
111
  CMD ["sleep", "infinity"]
112
  ```
113
 
@@ -119,12 +68,9 @@ export VM_IMAGE=llm-vm
119
  python run.py
120
  ```
121
 
122
- The custom VM includes typical utilities like ``sudo`` and ``curl`` so it behaves
123
- more like a standard Ubuntu installation.
124
-
125
  ## REST API
126
 
127
- Start the API server as a module or with ``uvicorn``:
128
 
129
  ```bash
130
  python -m api_app
@@ -134,9 +80,9 @@ uvicorn api_app:app --host 0.0.0.0 --port 8000
134
 
135
  ### Endpoints
136
 
137
- - ``POST /chat/stream``Stream the assistant's response as plain text.
138
- - ``POST /upload``Upload a document so it can be referenced in chats.
139
- - ``GET /sessions/{user}``List available session names for ``user``.
140
 
141
  Example request:
142
 
@@ -146,26 +92,22 @@ curl -N -X POST http://localhost:8000/chat/stream \
146
  -d '{"user":"demo","session":"default","prompt":"Hello"}'
147
  ```
148
 
149
- ## CLI
150
 
151
- An interactive command line interface is provided for Windows and other
152
- platforms. Install the dependencies and run:
153
 
154
  ```bash
155
  python -m src.cli --user yourname
156
  ```
157
 
158
- The tool lists your existing chat sessions and lets you select one or create a
159
- new session. Type messages and the assistant's streamed replies will appear
160
- immediately. Enter ``exit`` or press ``Ctrl+D`` to quit.
161
 
162
- ### Windows executable
163
 
164
- For a standalone application that does not require Python, use the code in
165
- `cli_app`. After installing ``pyinstaller`` run:
166
 
167
  ```bash
168
  pyinstaller --onefile -n llm-chat cli_app/main.py
169
  ```
170
 
171
- The resulting ``llm-chat.exe`` can be used on Windows 10/11.
 
1
  # llm-backend
2
 
3
+ `llm-backend` provides an asynchronous chat interface built around Ollama models. It supports running shell commands in an isolated Linux VM and persists conversations in SQLite.
4
+
5
+ ## Features
6
+
7
+ - **Persistent chat history** – conversations are stored in `chat.db` per user and session so they can be resumed later.
8
+ - **Tool execution** – a built-in `execute_terminal` tool runs commands inside a Docker-based VM. Network access is enabled and both stdout and stderr are captured (up to 10,000 characters). The VM is reused across chats when `PERSIST_VMS=1` so installed packages remain available.
9
+ - **System prompts** every request includes a system prompt that guides the assistant to plan tool usage, verify results and avoid unnecessary jargon.
10
+
11
+ ## Quick Start
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  ```bash
14
  python run.py
15
  ```
16
 
17
+ The script issues a sample command to the model and prints the streamed response. Uploaded files go to `uploads` and are mounted in the VM at `/data`.
18
 
19
+ ### Uploading Documents
20
 
21
  ```python
22
  async with ChatSession() as chat:
23
+ path = chat.upload_document("path/to/file.pdf")
24
+ async for part in chat.chat_stream(f"Summarize {path}"):
25
  print(part)
26
  ```
27
 
 
 
 
 
28
  ## Discord Bot
29
 
30
+ 1. Create a `.env` file with your bot token:
31
 
32
+ ```bash
33
+ DISCORD_TOKEN="your-token"
34
+ ```
35
+ 2. Start the bot:
36
 
37
+ ```bash
38
+ python -m bot
39
+ ```
40
 
41
+ Attachments sent to the bot are uploaded automatically and the VM path is returned so they can be referenced in later messages.
 
 
 
 
 
42
 
43
  ## VM Configuration
44
 
45
+ The shell commands run inside a Docker container. By default the image defined by `VM_IMAGE` is used (falling back to `python:3.11-slim`). When `PERSIST_VMS=1` (default) each user keeps the same container across sessions. Set `VM_STATE_DIR` to choose where per-user data is stored on the host.
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
+ To build a more complete environment you can create your own image, for example using `docker/Dockerfile.vm`:
 
48
 
49
  ```Dockerfile
50
  FROM ubuntu:22.04
 
 
51
  RUN apt-get update && \
52
  apt-get install -y --no-install-recommends \
53
  python3 \
 
57
  git \
58
  build-essential \
59
  && rm -rf /var/lib/apt/lists/*
 
60
  CMD ["sleep", "infinity"]
61
  ```
62
 
 
68
  python run.py
69
  ```
70
 
 
 
 
71
  ## REST API
72
 
73
+ Start the API server either as a module or via `uvicorn`:
74
 
75
  ```bash
76
  python -m api_app
 
80
 
81
  ### Endpoints
82
 
83
+ - `POST /chat/stream`stream the assistant's response as plain text.
84
+ - `POST /upload`upload a document that can be referenced in chats.
85
+ - `GET /sessions/{user}`list available session names for a user.
86
 
87
  Example request:
88
 
 
92
  -d '{"user":"demo","session":"default","prompt":"Hello"}'
93
  ```
94
 
95
+ ## Command Line Interface
96
 
97
+ Run the interactive CLI on any platform:
 
98
 
99
  ```bash
100
  python -m src.cli --user yourname
101
  ```
102
 
103
+ Existing sessions are listed and you can create new ones. Type messages to see streamed replies. Use `exit` or `Ctrl+D` to quit.
 
 
104
 
105
+ ### Windows Executable
106
 
107
+ For a standalone Windows build install `pyinstaller` and run:
 
108
 
109
  ```bash
110
  pyinstaller --onefile -n llm-chat cli_app/main.py
111
  ```
112
 
113
+ The resulting `llm-chat.exe` works on Windows 10/11.