docs: README /metrics + MAX_TOKENS, smoke-test health/metrics/timing checks

feat(metrics): prometheus /metrics + generation timing in usage
- /metrics endpoint: request/token/latency counters, tok/s gauge, build info (std-only, no deps) - usage.duration_ms + usage.tokens_per_second in non-streaming and streaming responses - /health now reports uptime_s, n_ctx, version - MAX_TOKENS env config (hard cap, default 2048; 0 = unlimited) - metrics unit tests (counters, prometheus shape, uptime monotonic)
2026-08-03 11:44:27 +07:00 · 2026-08-03 11:44:21 +07:00 · 2026-08-03 11:03:26 +07:00 · 2026-08-03 08:58:13 +07:00 · 2026-08-02 16:44:43 +07:00 · 2026-08-02 16:18:17 +07:00
20 changed files with 2026 additions and 433 deletions
@@ -0,0 +1,25 @@
+name: Notify Parent Repo
+
+on:
+  push:
+    branches:
+      - master
+  workflow_dispatch:
+
+jobs:
+  dispatch:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Trigger root monorepo build
+        uses: peter-evans/repository-dispatch@v3
+        with:
+          token: ${{ secrets.DISPATCH_TOKEN }}
+          repository: asepharyana/asepharyana-hub
+          event-type: submodule-updated
+          client-payload: |
+            {
+              "service": "llm-api",
+              "ref": "${{ github.ref }}",
+              "sha": "${{ github.sha }}",
+              "actor": "${{ github.actor }}"
+            }
@@ -568,6 +568,7 @@ dependencies = [
 "chrono",
 "futures",
 "llama-cpp-2",
+ "minijinja",
 "serde",
 "serde_json",
 "thiserror 1.0.69",
@@ -615,12 +616,28 @@ version = "2.8.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"

+[[package]]
+name = "memo-map"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "38d1115007560874e373613744c6fba374c17688327a71c1476d1a5954cc857b"
+
 [[package]]
 name = "mime"
 version = "0.3.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"

+[[package]]
+name = "minijinja"
+version = "2.21.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cb3d648e68cea56d9858d535ee28f9538404e2dd8cb08ed0bd05dca379477f39"
+dependencies = [
+ "memo-map",
+ "serde",
+]
+
 [[package]]
 name = "minimal-lexical"
 version = "0.2.1"
@@ -6,6 +6,7 @@ edition = "2021"
 [dependencies]
 # LLM inference
 llama-cpp-2 = "0.1"
+minijinja = "2"

 # HTTP server
 axum = { version = "0.8", features = ["json"] }
@@ -2,7 +2,7 @@

 OpenAI-compatible LLM inference server using `llama-cpp-2` (Rust).

-**Model:** MiniCPM-V-4.6 Q4_K_M (505 MB)  
+**Model:** MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking (Q8_0)  
 **Engine:** llama.cpp via `llama-cpp-2` crate  
 **Domain:** [ai.asepharyana.my.id](https://ai.asepharyana.my.id)

@@ -10,7 +10,23 @@ OpenAI-compatible LLM inference server using `llama-cpp-2` (Rust).

 ### `GET /health`
 ```json
-{"status": "ok", "model": "minicpm-v-4.6-q4_k_m"}
+{"status": "ok", "model": "minicpm5-1b-fable5-v2-thinking", "uptime_s": 1234, "n_ctx": 8192, "version": "0.1.0"}
+```
+
+### `GET /metrics`
+Prometheus text exposition (no auth) — request counters, token usage, generation
+latency/throughput, process uptime:
+
+```
+llm_api_requests_total            # total /v1/chat/completions
+llm_api_errors_total              # errored requests
+llm_api_streaming_requests_total  # stream: true requests
+llm_api_aborted_requests_total    # aborted generations (client disconnect)
+llm_api_prompt_tokens_total       # prompt tokens accepted
+llm_api_completion_tokens_total   # tokens generated
+llm_api_generation_ms_total       # generation time (ms)
+llm_api_tokens_per_second         # lifetime throughput gauge
+llm_api_build_info{version,model} # identity
 ```

 ### `GET /v1/models`
@@ -19,11 +35,13 @@ OpenAI-compatible model listing.
 ### `POST /v1/chat/completions`
 OpenAI-compatible chat completions.

+The server serves a single model and rejects unknown model ids with `400`:
+
 ```bash
 curl https://ai.asepharyana.my.id/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
-    "model": "minicpm-v-4.6",
+    "model": "minicpm5-1b-fable5-v2-thinking",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'
@@ -42,14 +60,38 @@ MODEL_PATH=/path/to/model.gguf ./target/release/llm-api
 ./target/release/llm-api
 ```

-## Docker
+### Environment variables
+
+| Var | Default | Description |
+|-----|---------|-------------|
+| `MODEL_PATH` | `/models/MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-Q8_0.gguf` | GGUF model file |
+| `API_KEY` | *(empty = auth off)* | Bearer token required on `/v1/chat/completions` |
+| `SERVER_PORT` | `4010` | Listen port |
+| `RUST_LOG` | `info` | Log level |
+| `N_CTX` / `N_BATCH` / `N_THREADS` | `8192` / `512` / `4` | llama.cpp context/batch/threads |
+| `MAX_TOKENS` | `2048` | Hard cap untuk `max_tokens` request (0 = unlimited) |
+
+### Smoke test (setelah deploy)

 ```bash
-docker compose -f docker-compose.yml up -d
+./scripts/smoke-test.sh http://127.0.0.1:4010        # tanpa auth
+./scripts/smoke-test.sh https://ai.asepharyana.my.id "$API_KEY"
 ```

+## Deploy (Nix + systemd)
+
+```bash
+nix build .#default --impure --option sandbox false
+# GitHub Actions: nix copy ssh://vps → systemctl restart llm-api
+```
+
+> **Legacy (2026-08-02):** Docker compose dihapus dari produksi. Deploy sekarang Nix+systemd.
+
 ## Benchmark

+> *Historic* (MiniCPM-V-4.6). Kept for reference; numbers predate the current
+> MiniCPM5-1B Thinking model.
+
 | Framework | Model Size | tok/s | vs PyTorch |
 |-----------|-----------|-------|------------|
 | PyTorch BF16 | 2.48 GB | 0.97 | 1.0x |
@@ -57,6 +99,5 @@ docker compose -f docker-compose.yml up -d

 ## Infrastructure

- Traefik router: `ai.asepharyana.my.id` → `llm-api:8080`
- Network: `app-shared-net`
- Docker Compose: see `llm-api.yml`
+- Caddy reverse proxy: `ai.asepharyana.my.id` → `127.0.0.1:4010`
+- systemd unit `llm-api`, deploy Nix via GitHub Actions
@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+# Smoke test untuk llm-api — jalankan di VPS setelah deploy.
+#
+# Memverifikasi alur inti:
+#   /health, /v1/models, chat non-streaming, chat streaming (SSE + [DONE]),
+#   penolakan model tidak dikenal (400), dan request dengan tools.
+#
+# Usage:
+#   ./scripts/smoke-test.sh [BASE_URL] [API_KEY]
+#   BASE_URL default: http://127.0.0.1:4010
+set -euo pipefail
+
+BASE_URL="${1:-http://127.0.0.1:4010}"
+API_KEY="${2:-}"
+MODEL_ID="minicpm5-1b-fable5-v2-thinking"
+
+AUTH=()
+if [[ -n "$API_KEY" ]]; then
+  AUTH=(-H "Authorization: Bearer $API_KEY")
+fi
+
+pass=0
+fail=0
+check() {
+  local name="$1" ok="$2"
+  if [[ "$ok" == "0" ]]; then
+    echo "  PASS: $name"
+    pass=$((pass + 1))
+  else
+    echo "  FAIL: $name"
+    fail=$((fail + 1))
+  fi
+}
+
+echo "== 1. Health ==="
+health=$(curl -sf "$BASE_URL/health") || { echo "FAIL: /health unreachable"; exit 1; }
+echo "$health" | jq -e ".status == \"ok\" and .model == \"$MODEL_ID\"" >/dev/null
+check "health melaporkan $MODEL_ID" $?
+echo "$health" | jq -e '.uptime_s >= 0 and (.version | type == "string")' >/dev/null
+check "health punya uptime + version" $?
+
+echo "== 2. Models =="
+models=$(curl -sf "$BASE_URL/v1/models")
+echo "$models" | jq -e ".data[0].id == \"$MODEL_ID\"" >/dev/null
+check "models mencantumkan $MODEL_ID" $?
+
+echo "== 2b. Metrics =="
+metrics=$(curl -sf "$BASE_URL/metrics") || { echo "FAIL: /metrics unreachable"; exit 1; }
+echo "$metrics" | grep -q "llm_api_requests_total"
+check "metrics punya llm_api_requests_total" $?
+echo "$metrics" | grep -q "llm_api_build_info{"
+check "metrics punya llm_api_build_info" $?
+
+echo "== 3. Chat non-streaming =="
+resp=$(curl -sf "${AUTH[@]}" -H "Content-Type: application/json" \
+  -d "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hi\"}],\"max_tokens\":64}" \
+  "$BASE_URL/v1/chat/completions")
+echo "$resp" | jq -e '.choices[0].message.content | type == "string"' >/dev/null
+check "non-streaming mengembalikan content" $?
+echo "$resp" | jq -e '.usage.total_tokens > 0' >/dev/null
+check "non-streaming berisi usage" $?
+echo "$resp" | jq -e '.usage.duration_ms > 0 and .usage.tokens_per_second > 0' >/dev/null
+check "non-streaming berisi timing (duration_ms + tok/s)" $?
+
+echo "== 4. Chat streaming =="
+stream=$(curl -sfN "${AUTH[@]}" -H "Content-Type: application/json" \
+  -d "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"Count from 1 to 5\"}],\"stream\":true,\"max_tokens\":64}" \
+  "$BASE_URL/v1/chat/completions")
+
+echo "$stream" | grep -q '\[DONE\]'
+check "stream diakhiri [DONE]" $?
+
+echo "$stream" | grep -q '"finish_reason":"stop"\|"finish_reason":"length"'
+check "stream punya finish_reason" $?
+
+# Gabungkan semua delta content untuk memastikan output tidak kosong
+# dan tidak bocor markup (mis. <tool_call> / <|im_end|>).
+joined=$(echo "$stream" | grep '^data: ' | sed 's/^data: //' \
+  | grep -v '\[DONE\]' \
+  | jq -r 'select((.choices? // []) | length > 0) | .choices[0].delta.content // empty' 2>/dev/null \
+  | tr -d '\n')
+if [[ -z "$joined" ]]; then
+  # Sebagian output mungkin semua ber-label reasoning_content; cek keduanya.
+  joined=$(echo "$stream" | grep '^data: ' | sed 's/^data: //' \
+    | grep -v '\[DONE\]' \
+    | jq -r 'select((.choices? // []) | length > 0) | (.choices[0].delta.content // .choices[0].delta.reasoning_content) // empty' 2>/dev/null \
+    | tr -d '\n')
+fi
+[[ -n "$joined" ]]
+check "stream menghasilkan teks" $?
+if [[ -n "$joined" ]]; then
+  if [[ "$joined" == *"<"* ]]; then
+    check "tidak ada markup bocor di stream" 1
+  else
+    check "tidak ada markup bocor di stream" 0
+  fi
+fi
+
+echo "== 5. Model tidak dikenal ditolak =="
+status=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" \
+  -d '{"model":"minicpm-v-4.6","messages":[{"role":"user","content":"hi"}]}' \
+  "$BASE_URL/v1/chat/completions")
+[[ "$status" == "400" ]]
+check "unknown model -> 400" $?
+
+echo "== 6. Request dengan tools diterima =="
+status2=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" \
+  -d "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"What's the weather in Jakarta?\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}}}}}],\"max_tokens\":128}" \
+  "$BASE_URL/v1/chat/completions")
+[[ "$status2" == "200" ]]
+check "tools request sukses (200)" $?
+
+echo
+echo "Result: $pass passed, $fail failed"
+[[ $fail -eq 0 ]]
@@ -1,3 +1,6 @@
 pub mod use_cases;

-pub use use_cases::{build_prompt, build_sampler, clean_text, parse_tool_calls, SamplerParams};
+pub use use_cases::{
+    build_prompt, build_sampler, clean_text, parse_tool_calls, split_stream_chunk, strip_markup,
+    validate_model, SamplerParams,
+};
@@ -0,0 +1,44 @@
+{{- bos_token }}
+{%- if tools %}
+    {%- set tool_definitions %}
+        {{- "# Tools\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+        {%- for tool in tools %}
+            {{- "\n" }}
+            {{- tool | tojson }}
+        {%- endfor %}
+        {{- '\n</tools>\n\nTool usage guidelines:\n- You may call zero or more functions. If no function calls are needed, just answer normally.\n- When calling a function, wrap each call in <tool_call> tags:\n<tool_call>\n<function=name>\n<parameter=key>value</parameter>\n</function>\n</tool_call>' }}
+    {%- endset %}
+
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' + tool_definitions }}
+    {%- else %}
+        {{- tool_definitions }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = message.content %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {#- Tool-call XML from history is pre-embedded in content by build_prompt. #}
+        {{- '<|im_start|>assistant\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {{- '<|im_start|>user\n<tool_response>\n' + content + '\n</tool_response><|im_end|>\n' }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined %}
+        {%- if enable_thinking is false %}
+            {{- '<think>\n\n</think>\n\n' }}
+        {%- elif enable_thinking is true %}
+            {{- '<think>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}
@@ -1,97 +1,119 @@
 //! Chat completion use cases.
 //!
-//! Orchestrates prompt building, sampler construction, and output parsing.
-//! These are pure functions with no framework dependencies.
+//! Orchestrates prompt building using the model's baked-in Jinja template
+//! via the `minijinja` crate, sampler construction, and output parsing.
+
+use std::collections::HashMap;

 use llama_cpp_2::sampling::LlamaSampler;
+use minijinja::{Environment, Value};

-use crate::domain::entity::{ChatMessage, ChatRequest, ToolCall, ToolCallFunction, ToolDef};
+use crate::domain::entity::{ChatMessage, ChatRequest, ToolCall, ToolCallFunction};

-/// Build a prompt string from conversation messages and optional tool definitions.
+/// Build a prompt from messages using the project's Jinja chat template.
 ///
-/// Uses ChatML format with `<|im_start|>` / `<|im_end|>` delimiters. Tool
-/// definitions are injected into the first system message.
-pub fn build_prompt(messages: &[ChatMessage], tools: &Option<Vec<ToolDef>>) -> String {
-    let mut prompt = String::new();
+/// Renders the template (see `templates/chat_template.jinja`) via minijinja,
+/// passing the message history, optional tool definitions, and the
+/// generation-prompt switches. The template owns the `<s>` BOS token, so
+/// tokenization must NOT add another one (`AddBos::Never`).
+pub fn build_prompt(
+    messages: &[ChatMessage],
+    tools: &Option<Vec<crate::domain::entity::ToolDef>>,
+) -> Result<String, String> {
+    let template_str = include_str!("templates/chat_template.jinja");

-    for (i, msg) in messages.iter().enumerate() {
-        match msg.role.as_str() {
-            "system" => {
-                let mut content = msg.content.clone().unwrap_or_default();
-                // Inject tools into the system message (first occurrence)
-                if i == 0 {
-                    if let Some(tools_list) = tools {
-                        if !tools_list.is_empty() {
-                            let mut tools_text = String::from(
-                                "\n\n# Tools\n\nYou have access to the following functions:\n\n<tools>",
-                            );
-                            for tool in tools_list {
-                                tools_text.push('\n');
-                                tools_text.push_str(
-                                    &serde_json::to_string(tool).unwrap_or_default(),
-                                );
-                            }
-                            tools_text.push_str(
-                                "\n</tools>\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>",
-                            );
-                            content.push_str(&tools_text);
-                        }
-                    }
-                }
-                prompt.push_str(&format!("<|im_start|>system\n{}<|im_end|>\n", content));
-            }
-            "user" => {
-                let content = msg.content.as_deref().unwrap_or("");
-                if msg.tool_call_id.is_some() || msg.name.is_some() {
-                    prompt.push_str(&format!(
-                        "<|im_start|>user\n<tool_response>\n{}\n</tool_response><|im_end|>\n",
+    let mut env = Environment::new();
+    env.add_template("chat", template_str)
+        .map_err(|e| format!("Template add error: {e}"))?;
+
+    // Register tojson filter (safe: Rust serde_json defaults to ensure_ascii=false)
+    env.add_filter("tojson", |value: &Value| -> String {
+        serde_json::to_string(value).unwrap_or_default()
+    });
+
+    let tmpl = env
+        .get_template("chat")
+        .map_err(|e| format!("Template get error: {e}"))?;
+
+    // Build messages as serde_json::Value for minijinja.
+    // Assistant tool calls from history are embedded directly into the content
+    // as XML (in the exact format the model is told to emit) — the template
+    // cannot accumulate `set` variables across a loop, so this is built here.
+    let mut msgs_val: Vec<Value> = Vec::new();
+    for msg in messages {
+        let mut m: HashMap<String, Value> = HashMap::new();
+        m.insert("role".into(), Value::from(msg.role.clone()));
+
+        let mut content = msg.content.clone().unwrap_or_default();
+
+        if msg.role == "assistant" {
+            if let Some(tcs) = &msg.tool_calls {
+                for tc in tcs {
+                    if tc.call_type == "function" {
                        content
-                    ));
-                } else {
-                    prompt.push_str(&format!("<|im_start|>user\n{}<|im_end|>\n", content));
-                }
-            }
-            "assistant" => {
-                let content = msg.content.as_deref().unwrap_or("");
-                if let Some(tcs) = &msg.tool_calls {
-                    let mut asst = format!("<|im_start|>assistant\n{}", content);
-                    for tc in tcs {
+                            .push_str(&format!("\n<tool_call>\n<function={}>\n", tc.function.name));
                        let args: serde_json::Value =
                            serde_json::from_str(&tc.function.arguments).unwrap_or_default();
-                        asst.push_str(&format!(
-                            "<tool_call>\n<function={}>\n",
-                            tc.function.name
-                        ));
                        if let Some(obj) = args.as_object() {
                            for (k, v) in obj {
-                                let val = match v {
+                                // Strings stay raw (no JSON quotes) so they round-trip
+                                // through parse_tool_calls unchanged.
+                                let rendered = match v {
                                    serde_json::Value::String(s) => s.clone(),
-                                    other => serde_json::to_string(other).unwrap_or_default(),
+                                    other => other.to_string(),
                                };
-                                asst.push_str(&format!("<parameter={}>\n{}\n</parameter>\n", k, val));
+                                content
+                                    .push_str(&format!("<parameter={k}>{rendered}</parameter>\n"));
                            }
                        }
-                        asst.push_str("</function>\n</tool_call>");
+                        content.push_str("</function>\n</tool_call>");
                    }
-                    asst.push_str("<|im_end|>\n");
-                    prompt.push_str(&asst);
-                } else {
-                    prompt.push_str(&format!(
-                        "<|im_start|>assistant\n{}<|im_end|>\n",
-                        content
-                    ));
                }
            }
-            _ => {
-                let content = msg.content.as_deref().unwrap_or("");
-                prompt.push_str(&format!("<|im_start|>user\n{}<|im_end|>\n", content));
-            }
+        }
+
+        // NOTE: the template wraps tool-role content in <tool_response>; do not
+        // wrap here or it would be double-wrapped.
+        m.insert("content".into(), Value::from(content));
+
+        msgs_val.push(Value::from(m));
+    }
+
+    // BOS token for sentencepiece / unigram models (template-owned).
+    let bos_token: &str = "<s>";
+
+    // Build context
+    let mut ctx: HashMap<String, Value> = HashMap::new();
+    ctx.insert("bos_token".into(), Value::from(bos_token));
+    ctx.insert("messages".into(), Value::from(msgs_val));
+    ctx.insert("add_generation_prompt".into(), Value::from(true));
+    ctx.insert("enable_thinking".into(), Value::from(true));
+
+    // Tool definitions (optional) — previously dead, now actually rendered.
+    if let Some(tools) = tools {
+        if !tools.is_empty() {
+            let tools_val: Vec<Value> = tools.iter().map(Value::from_serialize).collect();
+            ctx.insert("tools".into(), Value::from(tools_val));
        }
    }

-    // Generation prompt: non-thinking mode
-    prompt.push_str("<|im_start|>assistant\n<think>\n\n</think>\n\n");
-    prompt
+    // Render
+    let result = tmpl
+        .render(&ctx)
+        .map_err(|e| format!("Template render error: {e}"))?;
+
+    Ok(result)
+}
+
+/// Validate that the requested model matches the single served model.
+pub fn validate_model(model: &str) -> Result<(), String> {
+    if model != crate::config::MODEL_ID {
+        return Err(format!(
+            "Unknown model '{model}'. Available: {}",
+            crate::config::MODEL_ID
+        ));
+    }
+    Ok(())
 }

 /// Parameters for building a [`LlamaSampler`] chain.
@@ -135,7 +157,6 @@ pub fn build_sampler(params: &SamplerParams) -> LlamaSampler {
    let seed = params.seed;
    let mut samplers: Vec<LlamaSampler> = Vec::new();

-    // Repetition/frequency/presence penalties
    let repeat = repeat_penalty.unwrap_or(1.0);
    let freq = frequency_penalty.unwrap_or(0.0);
    let present = presence_penalty.unwrap_or(0.0);
@@ -143,22 +164,18 @@ pub fn build_sampler(params: &SamplerParams) -> LlamaSampler {
        samplers.push(LS::penalties(64, repeat, freq, present));
    }

-    // top_k
    if let Some(k) = top_k {
        samplers.push(LS::top_k(k as i32));
    }

-    // top_p
    if let Some(p) = top_p {
        samplers.push(LS::top_p(p, 1));
    }

-    // min_p
    if let Some(p) = min_p {
        samplers.push(LS::min_p(p, 1));
    }

-    // Temperature + final selector
    let temp = temperature.unwrap_or(0.0);
    if temp <= 0.0 {
        samplers.push(LS::greedy());
@@ -173,19 +190,36 @@ pub fn build_sampler(params: &SamplerParams) -> LlamaSampler {
    LlamaSampler::chain_simple(samplers)
 }

-
 // ═══════════════════════════════════════════════════════════════
 //  TEXT PROCESSING
 // ═══════════════════════════════════════════════════════════════

-/// Remove special tokens from generated text.
-pub fn clean_text(text: &str) -> String {
-    text.replace("<|im_end|>", "")
-        .replace("<|im_start|>", "")
-        .replace("<think>", "")
-        .replace("</think>", "")
-        .trim()
-        .to_string()
+/// Remove special tokens from generated text and separate reasoning.
+///
+/// For thinking models, returns (reasoning, cleaned_answer).
+pub fn clean_text(text: &str) -> (String, String) {
+    let text = text.replace("<|im_end|>", "").replace("<|im_start|>", "");
+
+    // Separate reasoning (between <think>/</think>) from answer
+    let text = text.trim();
+    let (reasoning, answer) = if let Some(close_idx) = text.find("</think>") {
+        let reasoning = text[..close_idx]
+            .trim()
+            .trim_start_matches("<think>")
+            .trim()
+            .to_string();
+        let answer = text[close_idx + 8..].trim().to_string();
+        (reasoning, answer)
+    } else if text.contains("<think>") {
+        // Still thinking — everything is reasoning
+        let reasoning = text.trim_start_matches("<think>").trim().to_string();
+        (reasoning, String::new())
+    } else {
+        (String::new(), text.to_string())
+    };
+
+    let answer = answer.replace("<think>", "").replace("</think>", "");
+    (reasoning, answer.trim().to_string())
 }

 /// Parse tool calls from generated text in the format:
@@ -240,9 +274,9 @@ pub fn parse_tool_calls(text: &str) -> (String, Vec<ToolCall>) {

        for line in lines {
            let line = line.trim();
-            if let Some(param) =
-                line.strip_prefix("<parameter=")
-                    .and_then(|s| s.strip_suffix('>'))
+            if let Some(param) = line
+                .strip_prefix("<parameter=")
+                .and_then(|s| s.strip_suffix('>'))
            {
                if let Some(p) = current_param.take() {
                    args_map.insert(
@@ -264,9 +298,11 @@ pub fn parse_tool_calls(text: &str) -> (String, Vec<ToolCall>) {
                continue;
            }
        }
-        // Save last param
        if let Some(p) = current_param.take() {
-            args_map.insert(p, serde_json::Value::String(current_value.trim().to_string()));
+            args_map.insert(
+                p,
+                serde_json::Value::String(current_value.trim().to_string()),
+            );
        }

        let args_json = serde_json::Value::Object(args_map).to_string();
@@ -283,12 +319,354 @@ pub fn parse_tool_calls(text: &str) -> (String, Vec<ToolCall>) {
        idx = end;
    }

-    // Remove tool_call blocks from the text
    clean = clean.replace("<tool_call>", "").replace("</tool_call>", "");
-    // XML-like tags are already fully parsed; remaining text is the content
    clean = clean.trim().to_string();
-    // Strip remaining XML tags that aren't part of clean
-    let cleaned = clean_text(&clean);
+    let (_reasoning, cleaned) = clean_text(&clean);

    (cleaned, tool_calls)
 }
+
+// ═══════════════════════════════════════════════════════════════
+//  STREAMING CHUNK SPLITTING
+// ═══════════════════════════════════════════════════════════════
+
+/// Remove special tokens and tool-call XML markup from a text fragment.
+///
+/// Handles both fixed tags (`<|im_end|>`, `<think>`, `<tool_call>`, …) and the
+/// attribute-bearing openers used by this model's tool format (`<function=…>`,
+/// `<parameter=…>`), even when a tag straddles a token boundary.
+pub fn strip_markup(text: &str) -> String {
+    let mut out = String::with_capacity(text.len());
+    let mut rest = text;
+
+    while !rest.is_empty() {
+        let Some(idx) = rest.find('<') else {
+            out.push_str(rest);
+            break;
+        };
+        out.push_str(&rest[..idx]);
+        let tail = &rest[idx..];
+
+        // Fixed tags (no attribute content).
+        let fixed = [
+            "<|im_start|>",
+            "<|im_end|>",
+            "<think>",
+            "</think>",
+            "<tool_call>",
+            "</tool_call>",
+            "</function>",
+            "</parameter>",
+        ];
+        if let Some(tag) = fixed.iter().find(|t| tail.starts_with(**t)) {
+            rest = &tail[tag.len()..];
+            continue;
+        }
+
+        // Attribute-bearing openers: <function=…> / <parameter=…>.
+        if let Some(attr) = tail
+            .strip_prefix("<function=")
+            .or_else(|| tail.strip_prefix("<parameter="))
+        {
+            if let Some(end) = attr.find('>') {
+                rest = &attr[end + 1..];
+                continue;
+            }
+        }
+
+        // Not a known tag — keep this char and advance one UTF-8 char.
+        let ch = tail.chars().next().expect("non-empty tail");
+        out.push(ch);
+        rest = &tail[ch.len_utf8()..];
+    }
+
+    out
+}
+
+/// Compute `(reasoning, content)` deltas for an incremental streamed fragment.
+///
+/// * `new_text` — the not-yet-emitted fragment (`text_buf[sent_len..]`).
+/// * `sent_len` — byte offset in the full buffer where `new_text` begins.
+/// * `content_start` — byte offset in the full buffer where the content phase
+///   begins (immediately after the first `</think>`); `None` while still
+///   thinking.
+///
+/// The boundary is a position in the *full* buffer, not a string search in the
+/// fragment — this stays correct even when `</think>` is split across tokens.
+/// Whitespace inside a fragment is preserved; only the edges around the
+/// boundary are trimmed. A stray second `</think>` is stripped, not re-split.
+pub fn split_stream_chunk(
+    new_text: &str,
+    sent_len: usize,
+    content_start: Option<usize>,
+) -> (Option<String>, String) {
+    match content_start {
+        // Still thinking — everything is reasoning.
+        None => {
+            let cleaned = strip_markup(new_text);
+            let reasoning = if cleaned.is_empty() {
+                None
+            } else {
+                Some(cleaned)
+            };
+            (reasoning, String::new())
+        }
+        // Boundary already emitted — everything is content.
+        Some(cs) if cs <= sent_len => (None, strip_markup(new_text)),
+        // Boundary falls inside this fragment (or beyond it, defensively).
+        Some(cs) => {
+            let rel = (cs - sent_len).min(new_text.len());
+            let (before, after) = new_text.split_at(rel);
+            let mut before = strip_markup(before);
+            let mut after = strip_markup(after);
+
+            while before.ends_with(['\n', ' ', '\t']) {
+                before.pop();
+            }
+            while after.starts_with(['\n', ' ', '\t']) {
+                after.remove(0);
+            }
+
+            let reasoning = if before.is_empty() {
+                None
+            } else {
+                Some(before)
+            };
+            (reasoning, after)
+        }
+    }
+}
+
+// ═══════════════════════════════════════════════════════════════
+//  TESTS
+// ═══════════════════════════════════════════════════════════════
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::domain::entity::{
+        ChatMessage, ToolCallFunction, ToolCallResponse, ToolDef, ToolFunction,
+    };
+
+    fn msg(role: &str, content: &str) -> ChatMessage {
+        ChatMessage {
+            role: role.into(),
+            content: Some(content.into()),
+            tool_calls: None,
+            tool_call_id: None,
+            name: None,
+        }
+    }
+
+    fn tool_def() -> ToolDef {
+        ToolDef {
+            tool_type: "function".into(),
+            function: ToolFunction {
+                name: "get_weather".into(),
+                description: "Get weather for a city".into(),
+                parameters: serde_json::json!({
+                    "type": "object",
+                    "properties": {
+                        "city": { "type": "string" }
+                    },
+                    "required": ["city"]
+                }),
+            },
+        }
+    }
+
+    // ── split_stream_chunk ──
+
+    #[test]
+    fn split_chunk_before_think_is_reasoning() {
+        let (reasoning, content) = split_stream_chunk("Hello ", 0, None);
+        assert_eq!(reasoning.as_deref(), Some("Hello "));
+        assert_eq!(content, "");
+    }
+
+    #[test]
+    fn split_chunk_preserves_internal_spaces() {
+        // Regression: trimming every fragment used to eat inter-word spaces.
+        let (r1, _) = split_stream_chunk("Hello", 0, None);
+        let (r2, _) = split_stream_chunk(" world", 5, None);
+        assert_eq!(format!("{}{}", r1.unwrap(), r2.unwrap()), "Hello world");
+    }
+
+    #[test]
+    fn split_chunk_boundary_trims_only_edges() {
+        // text_buf = "question\n</think>\n\nAnswer "; boundary right after the
+        // tag at byte 17.
+        let (reasoning, content) = split_stream_chunk("question\n</think>\n\nAnswer ", 0, Some(17));
+        assert_eq!(reasoning.as_deref(), Some("question"));
+        assert_eq!(content, "Answer ");
+    }
+
+    #[test]
+    fn split_chunk_after_think_is_content() {
+        let (reasoning, content) = split_stream_chunk(" answer", 0, Some(0));
+        assert_eq!(reasoning, None);
+        assert_eq!(content, " answer");
+    }
+
+    #[test]
+    fn split_chunk_stray_think_tag_is_stripped_not_split() {
+        // A second </think> (already past the boundary) must not restart
+        // reasoning classification.
+        let (reasoning, content) = split_stream_chunk("...</think>more", 0, Some(0));
+        assert_eq!(reasoning, None);
+        assert_eq!(content, "...more");
+    }
+
+    #[test]
+    fn split_chunk_boundary_straddling_tokens() {
+        // `</think>` split as "</think" + ">" across two fragments: the boundary
+        // is detected on the full buffer, so the answer still becomes content.
+        let (r1, _) = split_stream_chunk("reasoning...</think", 0, None);
+        assert!(r1.is_some());
+        let (r2, c2) = split_stream_chunk("\n\n2 + 2 = 4.", 18, Some(18));
+        assert_eq!(r2, None);
+        assert_eq!(c2, "\n\n2 + 2 = 4.");
+    }
+
+    #[test]
+    fn split_chunk_strips_special_and_markup() {
+        // boundary at byte 30 (right after "</think>").
+        let (reasoning, content) = split_stream_chunk(
+            "<|im_end|><think>Hello</think>\n<tool_call><function=get_weather>",
+            0,
+            Some(30),
+        );
+        assert_eq!(reasoning.as_deref(), Some("Hello"));
+        assert_eq!(content, "");
+    }
+
+    // ── clean_text ──
+
+    #[test]
+    fn clean_text_splits_reasoning_and_answer() {
+        let (reasoning, answer) =
+            clean_text("<think>Let me think\nabout it</think>\nThe answer is 42.");
+        assert_eq!(reasoning, "Let me think\nabout it");
+        assert_eq!(answer, "The answer is 42.");
+    }
+
+    #[test]
+    fn clean_text_no_think_returns_answer() {
+        let (reasoning, answer) = clean_text("Just an answer");
+        assert_eq!(reasoning, "");
+        assert_eq!(answer, "Just an answer");
+    }
+
+    // ── parse_tool_calls ──
+
+    #[test]
+    fn parse_single_tool_call() {
+        let text = "I'll look that up.\n<tool_call>\n<function=get_weather>\n<parameter=city>Jakarta</parameter>\n</function>\n</tool_call>";
+        let (cleaned, calls) = parse_tool_calls(text);
+        assert_eq!(calls.len(), 1);
+        assert_eq!(calls[0].function.name, "get_weather");
+        assert!(calls[0].function.arguments.contains("Jakarta"));
+        assert!(!cleaned.contains("<tool_call>"));
+    }
+
+    #[test]
+    fn parse_multiple_tool_calls() {
+        let text = "<tool_call>\n<function=a>\n<parameter=x>1</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=b>\n<parameter=y>2</parameter>\n</function>\n</tool_call>";
+        let (_cleaned, calls) = parse_tool_calls(text);
+        assert_eq!(calls.len(), 2);
+        assert_eq!(calls[0].function.name, "a");
+        assert_eq!(calls[1].function.name, "b");
+    }
+
+    #[test]
+    fn parse_malformed_tool_call_returns_empty() {
+        let (cleaned, calls) = parse_tool_calls("no calls here");
+        assert!(calls.is_empty());
+        assert_eq!(cleaned, "no calls here");
+    }
+
+    // ── validate_model ──
+
+    #[test]
+    fn validate_model_accepts_served_id() {
+        assert!(validate_model(crate::config::MODEL_ID).is_ok());
+    }
+
+    #[test]
+    fn validate_model_rejects_unknown_id() {
+        assert!(validate_model("minicpm-v-4.6").is_err());
+    }
+
+    // ── build_sampler (smoke — no model required) ──
+
+    #[test]
+    fn build_sampler_constructs_for_common_params() {
+        let params = SamplerParams {
+            temperature: Some(0.8),
+            top_p: Some(0.9),
+            top_k: Some(40),
+            min_p: Some(0.05),
+            repeat_penalty: Some(1.1),
+            frequency_penalty: Some(0.0),
+            presence_penalty: Some(0.0),
+            seed: Some(42),
+        };
+        let _ = build_sampler(&params);
+
+        let greedy = SamplerParams {
+            temperature: Some(0.0),
+            ..params
+        };
+        let _ = build_sampler(&greedy);
+    }
+
+    // ── build_prompt ──
+
+    #[test]
+    fn build_prompt_renders_messages() {
+        let messages = vec![msg("system", "You are helpful."), msg("user", "Hi!")];
+        let prompt = build_prompt(&messages, &None).unwrap();
+        assert!(prompt.starts_with("<s>"));
+        assert!(prompt.contains("<|im_start|>system\nYou are helpful."));
+        assert!(prompt.contains("<|im_start|>user\nHi!"));
+        assert!(prompt.contains("<|im_start|>assistant\n<think>\n"));
+    }
+
+    #[test]
+    fn build_prompt_includes_tool_definitions() {
+        let messages = vec![msg("user", "What's the weather?")];
+        let prompt = build_prompt(&messages, &Some(vec![tool_def()])).unwrap();
+        assert!(prompt.contains("<tools>"));
+        assert!(prompt.contains("get_weather"));
+        assert!(prompt.contains("Tool usage guidelines"));
+    }
+
+    #[test]
+    fn build_prompt_wraps_tool_response_once() {
+        let messages = vec![msg("user", "Weather?"), msg("tool", "Sunny")];
+        let prompt = build_prompt(&messages, &None).unwrap();
+        assert_eq!(prompt.matches("<tool_response>").count(), 1);
+        assert_eq!(prompt.matches("</tool_response>").count(), 1);
+    }
+
+    #[test]
+    fn build_prompt_renders_assistant_tool_calls_history() {
+        let assistant = ChatMessage {
+            role: "assistant".into(),
+            content: Some("".into()),
+            tool_calls: Some(vec![ToolCallResponse {
+                id: "call_1".into(),
+                call_type: "function".into(),
+                function: ToolCallFunction {
+                    name: "get_weather".into(),
+                    arguments: r#"{"city":"Jakarta"}"#.into(),
+                },
+            }]),
+            tool_call_id: None,
+            name: None,
+        };
+        let prompt = build_prompt(&[assistant], &None).unwrap();
+        assert!(prompt.contains("<function=get_weather>"));
+        assert!(prompt.contains("<parameter=city>Jakarta</parameter>"));
+    }
+}
@@ -32,15 +32,12 @@ impl Application {
    pub async fn build() -> anyhow::Result<Self> {
        // Initialize tracing
        let env_filter = EnvFilter::new(&CONFIG.log_level);
-        tracing_subscriber::fmt()
-            .with_env_filter(env_filter)
-            .init();
+        tracing_subscriber::fmt().with_env_filter(env_filter).init();
        tracing::info!("🚀 LLM API starting up...");

        // Load model (fail-fast)
-        let engine = LlamaEngine::load().map_err(|e| {
-            anyhow::anyhow!("Failed to initialize LLM engine: {e}")
-        })?;
+        let engine = LlamaEngine::load()
+            .map_err(|e| anyhow::anyhow!("Failed to initialize LLM engine: {e}"))?;
        let engine = Arc::new(engine);

        let state = Arc::new(AppState { engine });
@@ -51,10 +48,7 @@ impl Application {
        // Bind listener
        let addr = format!("0.0.0.0:{}", CONFIG.server_port);
        let listener = TcpListener::bind(&addr).await?;
-        tracing::info!(
-            "Server listening on {}",
-            listener.local_addr()?
-        );
+        tracing::info!("Server listening on {}", listener.local_addr()?);

        Ok(Self {
            port: CONFIG.server_port,
@@ -4,8 +4,16 @@

 use std::sync::LazyLock;

-const DEFAULT_MODEL_PATH: &str = "/models/MiniCPM-V-4.6-Q4_K_M.gguf";
-pub const MODEL_ID: &str = "minicpm-v-4.6";
+const DEFAULT_MODEL_PATH: &str = "/models/MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-Q8_0.gguf";
+pub const MODEL_ID: &str = "minicpm5-1b-fable5-v2-thinking";
+
+/// Read an env var as `T`, falling back to `default` on absence or parse error.
+fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
+    std::env::var(key)
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(default)
+}

 /// Application configuration loaded at startup from environment variables.
 #[derive(Debug, Clone)]
@@ -30,6 +38,9 @@ pub struct AppConfig {

    /// Number of CPU threads for inference
    pub n_threads: i32,
+
+    /// Hard cap for `max_tokens` in chat requests (0 = unlimited)
+    pub max_tokens: u32,
 }

 impl AppConfig {
@@ -42,11 +53,12 @@ impl AppConfig {
            server_port: std::env::var("SERVER_PORT")
                .ok()
                .and_then(|v| v.parse().ok())
-                .unwrap_or(8080),
+                .unwrap_or(4010),
            log_level: std::env::var("RUST_LOG").unwrap_or_else(|_| "info".to_string()),
-            n_ctx: 2048,
-            n_batch: 512,
-            n_threads: 4,
+            n_ctx: env_or("N_CTX", 8192),
+            n_batch: env_or("N_BATCH", 512),
+            n_threads: env_or("N_THREADS", 4),
+            max_tokens: env_or("MAX_TOKENS", 2048),
        }
    }
 }
@@ -96,16 +96,25 @@ pub struct Choice {
 #[derive(Serialize)]
 pub struct ResponseMessage {
    pub role: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
    pub content: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
+    pub reasoning_content: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
    pub tool_calls: Option<Vec<ToolCall>>,
 }

-#[derive(Serialize)]
+#[derive(Serialize, Clone)]
 pub struct Usage {
    pub prompt_tokens: u32,
    pub completion_tokens: u32,
    pub total_tokens: u32,
+    /// Wall-clock duration of the generation, in milliseconds.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub duration_ms: Option<u64>,
+    /// Generated tokens per second (completion_tokens / seconds).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub tokens_per_second: Option<f64>,
 }

 // ═══════════════════════════════════════════════════════════════
@@ -136,6 +145,36 @@ pub struct ToolCallResponse {
    pub function: ToolCallFunction,
 }

+// ═══════════════════════════════════════════════════════════════
+//  GENERATION FINISH REASON
+// ═══════════════════════════════════════════════════════════════
+
+/// Why a generation run stopped. Produced by the engine and rendered as the
+/// OpenAI `finish_reason` at the presentation layer.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FinishReason {
+    /// Natural end-of-generation (EOS token) or a stop sequence matched.
+    Stop,
+    /// `max_tokens` exhausted.
+    Length,
+    /// A complete `<tool_call>` block was emitted.
+    ToolCalls,
+    /// Generation aborted early (e.g. client disconnected).
+    Aborted,
+}
+
+impl FinishReason {
+    /// Map to the OpenAI-compatible `finish_reason` string.
+    pub fn as_str(&self) -> &'static str {
+        match self {
+            FinishReason::Stop => "stop",
+            FinishReason::Length => "length",
+            FinishReason::ToolCalls => "tool_calls",
+            FinishReason::Aborted => "stop",
+        }
+    }
+}
+
 // ═══════════════════════════════════════════════════════════════
 //  SSE (STREAMING) TYPES
 // ═══════════════════════════════════════════════════════════════
@@ -148,6 +187,59 @@ pub struct SseChunk {
    pub created: i64,
    pub model: String,
    pub choices: Vec<SseChoice>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub usage: Option<Usage>,
+}
+
+impl SseChunk {
+    /// A delta chunk carrying partial reasoning/content/tool-call output.
+    pub fn delta(id: String, created: i64, model: String, delta: SseDelta) -> Self {
+        Self {
+            id,
+            object: "chat.completion.chunk".into(),
+            created,
+            model,
+            choices: vec![SseChoice {
+                index: 0,
+                delta,
+                finish_reason: None,
+            }],
+            usage: None,
+        }
+    }
+
+    /// The final chunk carrying the `finish_reason` (no token deltas).
+    pub fn finish(id: String, created: i64, model: String, finish_reason: &str) -> Self {
+        Self {
+            id,
+            object: "chat.completion.chunk".into(),
+            created,
+            model,
+            choices: vec![SseChoice {
+                index: 0,
+                delta: SseDelta {
+                    role: None,
+                    content: None,
+                    tool_calls: None,
+                    reasoning_content: None,
+                },
+                finish_reason: Some(finish_reason.into()),
+            }],
+            usage: None,
+        }
+    }
+
+    /// A trailing chunk with token usage and empty choices (OpenAI convention).
+    pub fn usage(id: String, created: i64, model: String, usage: Usage) -> Self {
+        Self {
+            id,
+            object: "chat.completion.chunk".into(),
+            created,
+            model,
+            choices: vec![],
+            usage: Some(usage),
+        }
+    }
 }

 #[derive(Serialize)]
@@ -165,6 +257,8 @@ pub struct SseDelta {
    #[serde(skip_serializing_if = "Option::is_none")]
    pub content: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
+    pub reasoning_content: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
    pub tool_calls: Option<Vec<ToolCall>>,
 }

@@ -190,4 +284,13 @@ pub struct ModelInfo {
 pub struct HealthResponse {
    pub status: String,
    pub model: String,
+    /// Server process uptime in seconds.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub uptime_s: Option<u64>,
+    /// llama.cpp context size (n_ctx).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub n_ctx: Option<u32>,
+    /// Server binary version (from CARGO_PKG_VERSION).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub version: Option<String>,
 }
@@ -4,6 +4,7 @@
 //! lifetime transmute), tokenization, and generation.

 use std::num::NonZeroU32;
+use std::sync::Mutex;

 use llama_cpp_2::context::params::LlamaContextParams;
 use llama_cpp_2::context::LlamaContext;
@@ -14,10 +15,10 @@ use llama_cpp_2::model::{AddBos, LlamaModel};
 use llama_cpp_2::sampling::LlamaSampler;
 use llama_cpp_2::token::LlamaToken;
 use llama_cpp_2::TokenToStringError;
-use tokio::sync::Mutex;
 use tracing::info;

 use crate::config::CONFIG;
+use crate::domain::entity::FinishReason;
 use crate::domain::LlmError;

 // ── Thread-safe wrapper for raw llama.cpp context ──
@@ -95,17 +96,28 @@ impl CtxInner {

 // ── LlamaEngine ──

+/// Outcome of a generation run.
+pub struct GenerationOutcome {
+    /// Generated tokens (stop-sequence and EOG tokens are excluded).
+    pub tokens: Vec<LlamaToken>,
+    /// Accumulated decoded text (raw, before markup/special-token cleaning).
+    pub text: String,
+    /// Why generation stopped.
+    pub finish: FinishReason,
+}
+
 /// Safe interface to a llama.cpp model and inference context.
 ///
 /// All access to the underlying context is serialized through a `Mutex`,
 /// so only one generation can happen at a time. This is intentional —
-/// the model is designed for sequential inference.
+/// the model is designed for sequential inference. Generation is synchronous
+/// and must be invoked from the tokio blocking pool (`spawn_blocking`).
 pub struct LlamaEngine {
    /// The loaded model (read-only after load, safe to share).
    pub model: LlamaModel,

    /// The inference context (single-threaded access via Mutex).
-    pub ctx: Mutex<CtxInner>,
+    ctx: Mutex<CtxInner>,
 }

 impl LlamaEngine {
@@ -117,17 +129,17 @@ impl LlamaEngine {
    /// cannot be created.
    pub fn load() -> Result<Self, LlmError> {
        info!("Initializing llama backend...");
-        let backend = LlamaBackend::init().map_err(|e| {
-            LlmError::Model(format!("Backend init failed: {e}"))
-        })?;
+        let backend = LlamaBackend::init()
+            .map_err(|e| LlmError::Model(format!("Backend init failed: {e}")))?;

        // Backend must outlive model and context. We leak it to achieve 'static
        // lifetime since the engine lives for the program lifetime.
        let backend: &'static LlamaBackend = Box::leak(Box::new(backend));

        info!("Loading model: {}", CONFIG.model_path);
-        let model = LlamaModel::load_from_file(backend, &CONFIG.model_path, &LlamaModelParams::default())
-            .map_err(|e| LlmError::Model(format!("Failed to load model: {e}")))?;
+        let model =
+            LlamaModel::load_from_file(backend, &CONFIG.model_path, &LlamaModelParams::default())
+                .map_err(|e| LlmError::Model(format!("Failed to load model: {e}")))?;
        info!("  Vocab: {}", model.n_vocab());
        info!("  Params: {}", model.n_params());
        info!("  Layers: {}", model.n_layer());
@@ -156,9 +168,12 @@ impl LlamaEngine {
    }

    /// Tokenize a prompt string into tokens.
+    ///
+    /// `AddBos::Never`: the chat template already prepends the `<s>` BOS token,
+    /// so adding another here would produce a double BOS.
    pub fn tokenize(&self, prompt: &str) -> Result<Vec<LlamaToken>, LlmError> {
        self.model
-            .str_to_token(prompt, AddBos::Always)
+            .str_to_token(prompt, AddBos::Never)
            .map_err(|e| LlmError::Model(format!("Tokenization failed: {e}")))
    }

@@ -177,37 +192,28 @@ impl LlamaEngine {
        String::from_utf8(bytes).unwrap_or_default()
    }

-    /// Decode multiple tokens to a single string.
-    pub fn decode_tokens(&self, tokens: &[LlamaToken]) -> String {
-        let mut out = String::with_capacity(tokens.len() * 4);
-        for &token in tokens {
-            out.push_str(&self.decode_token(token));
-        }
-        out
-    }
-
-    /// Check if a token is an end-of-generation token.
-    pub fn is_eog(&self, token: LlamaToken) -> bool {
-        self.model.is_eog_token(token)
-    }
-
-    /// Return a reference to the context mutex for advanced operations.
-    pub fn ctx(&self) -> &Mutex<CtxInner> {
-        &self.ctx
-    }
-
-    /// Generate tokens (non-streaming) and return output tokens, cleaned text, and tool calls.
+    /// Generate tokens and invoke `on_token` for each one.
    ///
-    /// Locks the context mutex, prefill the prompt, then iterates sampling + decoding
-    /// until EOG, max_tokens, stop sequence, or tool call completion.
-    pub async fn generate(
+    /// Synchronous (CPU-bound) — call from `spawn_blocking`. Locks the context,
+    /// prefills the prompt, then iterates sampling + decoding until EOG,
+    /// `max_tokens`, a stop sequence, or a complete `<tool_call>` block.
+    ///
+    /// `on_token` is invoked for every generated token (after leading-EOG
+    /// skipping, before it is decoded into the KV cache) and may return `false`
+    /// to abort early (e.g. the streaming client disconnected).
+    pub fn generate(
        &self,
        input_tokens: &[LlamaToken],
-        sampler: &mut SendSampler,
+        sampler: &mut LlamaSampler,
        max_tokens: u32,
        stop: &[String],
-    ) -> Result<(Vec<LlamaToken>, String), LlmError> {
-        let mut inner = self.ctx.lock().await;
+        enable_tool_detection: bool,
+        on_token: &mut dyn FnMut(LlamaToken, &str) -> bool,
+    ) -> Result<GenerationOutcome, LlmError> {
+        let mut inner = self
+            .ctx
+            .lock()
+            .map_err(|_| LlmError::Internal("context mutex poisoned".into()))?;
        inner.clear();
        inner
            .prefill(input_tokens)
@@ -215,49 +221,66 @@ impl LlamaEngine {

        let mut output: Vec<LlamaToken> = Vec::new();
        let mut text_buf = String::new();
-        let mut stop_now = false;
+        let mut finish = FinishReason::Length;

        let mut current = inner.sample(sampler);

+        // Skip leading EOG tokens (like a stray <|im_end|> right after the prompt)
+        while output.is_empty() && self.model.is_eog_token(current) {
+            let pos = input_tokens.len() as i32 + output.len() as i32;
+            inner
+                .decode(current, pos)
+                .map_err(|e| LlmError::Model(format!("Decode: {e}")))?;
+            current = inner.sample(sampler);
+        }
+
        for _ in 0..max_tokens {
            if self.model.is_eog_token(current) {
+                finish = FinishReason::Stop;
+                break;
+            }
+
+            let piece = self.decode_token(current);
+
+            // Check stop sequences *before* committing, so the stop tokens never
+            // leak into the output text or the stream.
+            if stop
+                .iter()
+                .any(|s| !s.is_empty() && format!("{text_buf}{piece}").contains(s))
+            {
+                finish = FinishReason::Stop;
                break;
            }

            let pos = input_tokens.len() as i32 + output.len() as i32;
            output.push(current);
-
-            let piece = self.decode_token(current);
            text_buf.push_str(&piece);

-            // Check stop sequences
-            for s in stop {
-                if text_buf.contains(s) {
-                    stop_now = true;
-                    break;
-                }
-            }
-            if stop_now {
-                break;
-            }
-
-            // Check for tool_call block completion
-            if text_buf.contains("<tool_call>") {
-                let close_count = text_buf.matches("</tool_call>").count();
-                let open_count = text_buf.matches("<tool_call>").count();
-                if open_count > 0 && close_count >= open_count {
+            // Complete <tool_call> block emitted?
+            if enable_tool_detection && text_buf.contains("<tool_call>") {
+                let open = text_buf.matches("<tool_call>").count();
+                let close = text_buf.matches("</tool_call>").count();
+                if close >= open {
+                    finish = FinishReason::ToolCalls;
                    break;
                }
            }

-            if let Err(e) = inner.decode(current, pos) {
-                tracing::info!("  Decode error: {e}");
+            if !on_token(current, &piece) {
+                finish = FinishReason::Aborted;
                break;
            }

+            inner
+                .decode(current, pos)
+                .map_err(|e| LlmError::Model(format!("Decode: {e}")))?;
            current = inner.sample(sampler);
        }

-        Ok((output, text_buf))
+        Ok(GenerationOutcome {
+            tokens: output,
+            text: text_buf,
+            finish,
+        })
    }
 }
@@ -3,9 +3,11 @@
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>AI Chat</title>
+<title>AI Chat — llm-api</title>
+<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><text y='.9em' font-size='90'>✨</text></svg>">
 <style>
  *, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
+
  :root {
    --bg: #0f0f13;
    --surface: #1a1a23;
@@ -17,64 +19,172 @@
    --accent-hover: #9484ff;
    --user-msg: #2a2a48;
    --assistant-msg: #1a1a28;
+    --code-bg: #101018;
+    --error-bg: #2a1818;
+    --error-border: #4a2828;
+    --error-text: #f08080;
    --font: system-ui, -apple-system, 'Segoe UI', sans-serif;
+    --mono: ui-monospace, SFMono-Regular, 'Cascadia Code', Consolas, monospace;
+    color-scheme: dark;
  }
+
+  [data-theme="light"] {
+    --bg: #f5f5fa;
+    --surface: #ffffff;
+    --surface2: #ececf6;
+    --border: #dcdce8;
+    --text: #1c1c2b;
+    --text2: #6b6b8d;
+    --accent: #5b4de0;
+    --accent-hover: #4a3dc8;
+    --user-msg: #dcd4ff;
+    --assistant-msg: #ffffff;
+    --code-bg: #f0f0f8;
+    --error-bg: #fdeaea;
+    --error-border: #f2c6c6;
+    --error-text: #b3261e;
+    color-scheme: light;
+  }
+
  html, body { height: 100%; background: var(--bg); color: var(--text); font-family: var(--font); }
  body { display: flex; flex-direction: column; }

  header {
-    padding: 16px 24px;
+    padding: 12px 24px;
    border-bottom: 1px solid var(--border);
    background: var(--surface);
    display: flex; align-items: center; gap: 12px;
    flex-shrink: 0;
+    flex-wrap: wrap;
  }
  header h1 { font-size: 18px; font-weight: 600; }
  header .badge {
    font-size: 11px; padding: 2px 10px;
    border-radius: 99px; background: var(--accent);
-    color: #fff; font-weight: 500;
+    color: #fff; font-weight: 500; white-space: nowrap;
  }
+  header .spacer { flex: 1; }
+  .header-btn {
+    padding: 6px 12px; border: 1px solid var(--border); border-radius: 8px;
+    background: transparent; color: var(--text2); cursor: pointer;
+    font-size: 12px; font-weight: 500; transition: all .15s; white-space: nowrap;
+  }
+  .header-btn:hover { color: var(--text); border-color: var(--accent); }
+  .header-btn:disabled { opacity: .4; cursor: not-allowed; }

  #chat-container {
    flex: 1; overflow-y: auto; padding: 24px;
    display: flex; flex-direction: column; gap: 16px;
  }
-  #chat-container:empty::after {
-    content: 'Send a message to start chatting.';
-    color: var(--text2); font-size: 14px;
-    text-align: center; margin-top: 40px;
-  }
+  .welcome { text-align: center; color: var(--text2); margin-top: 48px; font-size: 14px; line-height: 1.8; }
+  .welcome h2 { color: var(--text); font-size: 22px; margin-bottom: 8px; }
+  .welcome .hints { opacity: .85; }

  .msg {
-    max-width: 720px; width: fit-content;
+    max-width: 780px; width: fit-content;
    padding: 12px 16px; border-radius: 12px;
    line-height: 1.6; font-size: 14px;
-    white-space: pre-wrap; word-break: break-word;
+    word-break: break-word;
  }
  .msg.user {
    background: var(--user-msg);
    border: 1px solid var(--border);
    align-self: flex-end;
    border-bottom-right-radius: 4px;
+    white-space: pre-wrap;
  }
  .msg.assistant {
    background: var(--assistant-msg);
    border: 1px solid var(--border);
    align-self: flex-start;
    border-bottom-left-radius: 4px;
+    min-width: 200px;
  }
-  .msg.assistant:empty::after { content: '⏳'; } /* spinner when empty */
-  .msg .tool-call {
+  .msg.error {
+    background: var(--error-bg); border-color: var(--error-border); color: var(--error-text);
+  }
+
+  /* Markdown content */
+  .md > *:first-child { margin-top: 0; }
+  .md > *:last-child { margin-bottom: 0; }
+  .md p { margin: 0.4em 0; }
+  .md h1, .md h2, .md h3, .md h4 { margin: 0.9em 0 0.4em; line-height: 1.3; }
+  .md h1 { font-size: 1.35em; } .md h2 { font-size: 1.2em; } .md h3 { font-size: 1.08em; }
+  .md ul, .md ol { margin: 0.4em 0; padding-left: 1.5em; }
+  .md li { margin: 0.2em 0; }
+  .md code {
+    font-family: var(--mono); font-size: 0.88em;
+    background: var(--code-bg); border: 1px solid var(--border);
+    padding: 1px 5px; border-radius: 5px;
+  }
+  .md pre {
+    background: var(--code-bg); border: 1px solid var(--border);
+    border-radius: 10px; padding: 12px 14px; margin: 0.6em 0;
+    overflow-x: auto;
+  }
+  .md pre code { background: none; border: none; padding: 0; font-size: 0.85em; line-height: 1.5; display: block; }
+  .md blockquote {
+    border-left: 3px solid var(--accent); padding-left: 12px;
+    color: var(--text2); margin: 0.5em 0;
+  }
+  .md a { color: var(--accent); }
+  .md table { border-collapse: collapse; margin: 0.6em 0; font-size: 0.92em; }
+  .md th, .md td { border: 1px solid var(--border); padding: 6px 10px; }
+  .md th { background: var(--surface2); }
+
+  .reasoning {
+    font-size: 12.5px; font-style: italic;
+    color: var(--text2);
+    border: 1px dashed var(--border);
+    border-radius: 8px; margin-bottom: 10px; overflow: hidden;
+  }
+  .reasoning summary {
+    cursor: pointer; user-select: none;
+    padding: 6px 10px; font-style: normal; font-weight: 600;
+    color: var(--text2); display: flex; align-items: center; gap: 6px;
+  }
+  .reasoning summary::before { content: '🧠'; }
+  .reasoning summary:hover { color: var(--text); }
+  .reasoning .reasoning-body {
+    padding: 6px 12px 10px; white-space: pre-wrap; word-break: break-word;
+    border-top: 1px dashed var(--border);
+  }
+  .reasoning[open] summary { border-bottom: 1px dashed var(--border); }
+
+  .tool-call {
    margin-top: 8px; padding: 8px 12px;
    background: var(--surface2); border-radius: 8px;
    font-size: 13px; color: var(--text2);
    display: flex; align-items: center; gap: 6px;
  }
-  .msg .tool-call::before { content: '\1F527'; }
-  .msg.error {
-    background: #2a1818; border-color: #4a2828; color: #f08080;
+  .tool-call::before { content: '🔧'; }
+
+  .msg-stats {
+    margin-top: 10px; padding-top: 8px;
+    border-top: 1px solid var(--border);
+    font-size: 11px; color: var(--text2);
+    display: flex; align-items: center; gap: 8px; flex-wrap: wrap;
  }
+  .msg-stats .stat {
+    display: inline-flex; align-items: center; gap: 4px;
+    background: var(--surface2); border-radius: 99px; padding: 2px 9px;
+  }
+  .msg-actions {
+    margin-top: 8px; display: flex; gap: 6px;
+  }
+  .msg-actions button {
+    font-size: 11px; padding: 3px 10px; border-radius: 6px;
+    border: 1px solid var(--border); background: transparent;
+    color: var(--text2); cursor: pointer; transition: all .15s;
+  }
+  .msg-actions button:hover { color: var(--text); border-color: var(--accent); }
+  .msg-actions .copied { color: #2ecc71; border-color: #2ecc71; }
+
+  .status {
+    text-align: center; font-size: 12px; color: var(--text2);
+    padding: 4px 0; display: none;
+  }
+  .status.visible { display: block; }

  #input-area {
    padding: 16px 24px;
@@ -101,60 +211,318 @@
  }
  #send-btn:hover { background: var(--accent-hover); }
  #send-btn:disabled { opacity: .4; cursor: not-allowed; }
+  #stop-btn {
+    padding: 12px 20px; border: 1px solid #e05b5b; border-radius: 12px;
+    background: transparent; color: #e05b5b;
+    font-size: 14px; font-weight: 500; cursor: pointer;
+    display: none; white-space: nowrap;
+  }
+  #stop-btn:hover { background: rgba(224, 91, 91, .12); }

-  .status {
-    text-align: center; font-size: 12px; color: var(--text2);
-    padding: 4px 0; display: none;
+  #params {
+    display: flex; gap: 16px; align-items: center;
+    padding: 10px 24px; border-top: 1px solid var(--border);
+    background: var(--surface); flex-shrink: 0; flex-wrap: wrap;
+  }
+  .param { display: flex; align-items: center; gap: 8px; font-size: 12px; color: var(--text2); }
+  .param input[type="number"], .param select {
+    width: 70px; padding: 4px 8px; border-radius: 8px;
+    border: 1px solid var(--border); background: var(--bg); color: var(--text);
+    font-size: 12px; font-family: var(--font);
+  }
+  .param select { width: auto; }
+  .param input[type="range"] { width: 90px; accent-color: var(--accent); }
+  .param label { display: flex; align-items: center; gap: 5px; cursor: pointer; }
+  .param label input[type="checkbox"] { accent-color: var(--accent); }
+
+  @media (max-width: 640px) {
+    header { padding: 10px 14px; }
+    #chat-container { padding: 14px; }
+    #input-area { padding: 12px 14px; flex-wrap: wrap; }
+    #send-btn { width: 100%; }
+    .msg { max-width: 100%; }
  }
-  .status.visible { display: block; }
 </style>
 </head>
-<body>
+<body data-theme="dark">
 <header>
  <h1>AI Chat</h1>
-  <span class="badge">llm-api</span>
+  <span class="badge" id="model-badge">llm-api</span>
+  <div class="spacer"></div>
+  <button class="header-btn" id="clear-btn" title="Clear conversation">🗑 Clear</button>
+  <button class="header-btn" id="theme-btn" title="Toggle theme">🌙</button>
 </header>
-<div id="chat-container"></div>
+
+<div id="chat-container">
+  <div class="welcome" id="welcome">
+    <h2>MiniCPM5-1B Thinking</h2>
+    <div class="hints">
+      Fast reasoning model running locally via llama.cpp.<br>
+      Markdown &amp; code supported · context window 8K · 1B params (Q8_0)
+    </div>
+  </div>
+</div>
 <div class="status" id="status"></div>
+
+<div id="params">
+  <div class="param">
+    <label title="Enable streaming"><input type="checkbox" id="param-stream" checked> Stream</label>
+  </div>
+  <div class="param" title="Sampling temperature (0 = greedy)">
+    <label for="param-temp">Temp</label>
+    <input type="number" id="param-temp" min="0" max="2" step="0.1" value="0.7">
+  </div>
+  <div class="param" title="Top-p nucleus sampling">
+    <label for="param-topp">Top-P</label>
+    <input type="number" id="param-topp" min="0" max="1" step="0.05" value="0.9">
+  </div>
+  <div class="param" title="Maximum tokens per response">
+    <label for="param-maxtokens">Max tokens</label>
+    <input type="number" id="param-maxtokens" min="16" max="2048" step="16" value="512">
+  </div>
+  <div class="param" title="System prompt for the model">
+    <label for="param-system">System</label>
+    <input type="text" id="param-system" style="width:180px" placeholder="(none)" value="">
+  </div>
+</div>
+
 <div id="input-area">
-  <textarea id="input" rows="1" placeholder="Type your message..."
-    @keydown="if (event.key === 'Enter' && !event.shiftKey) { event.preventDefault(); send() }"></textarea>
-  <button id="send-btn" onclick="send()">Send</button>
+  <textarea id="input" rows="1" placeholder="Type your message... (Enter to send, Shift+Enter for newline)"></textarea>
+  <button id="stop-btn">■ Stop</button>
+  <button id="send-btn">Send</button>
 </div>

 <script>
+(function () {
+'use strict';
+
 const CHAT_URL = '/v1/chat/completions';
+const MODELS_URL = '/v1/models';
 const $in = document.getElementById('input');
 const $btn = document.getElementById('send-btn');
+const $stop = document.getElementById('stop-btn');
 const $container = document.getElementById('chat-container');
 const $status = document.getElementById('status');
-let messages = [];
+const $welcome = document.getElementById('welcome');
+const $themeBtn = document.getElementById('theme-btn');
+const $clearBtn = document.getElementById('clear-btn');
+const $modelBadge = document.getElementById('model-badge');
+const $paramStream = document.getElementById('param-stream');
+const $paramTemp = document.getElementById('param-temp');
+const $paramTopp = document.getElementById('param-topp');
+const $paramMax = document.getElementById('param-maxtokens');
+const $paramSystem = document.getElementById('param-system');

+let messages = [];
+let modelId = 'minicpm5-1b-fable5-v2-thinking';
+let generating = false;
+let abortCtrl = null;
+
+// ── Theme (persisted) ──
+const savedTheme = localStorage.getItem('llmapi-theme') || 'dark';
+applyTheme(savedTheme);
+$themeBtn.addEventListener('click', () => {
+  const next = document.body.dataset.theme === 'dark' ? 'light' : 'dark';
+  applyTheme(next);
+  localStorage.setItem('llmapi-theme', next);
+});
+function applyTheme(t) {
+  document.body.dataset.theme = t;
+  $themeBtn.textContent = t === 'dark' ? '🌙' : '☀️';
+}
+
+// ── Model discovery ──
+fetch(MODELS_URL).then(r => r.json()).then(d => {
+  if (d && d.data && d.data.length) {
+    modelId = d.data[0].id;
+    $modelBadge.textContent = modelId;
+  }
+}).catch(() => {});
+
+// ── Markdown rendering (small, dependency-free) ──
+// XSS safety: the input is HTML-escaped in full (esc()) BEFORE any markdown
+// transformation; code blocks are escaped individually; links are restricted
+// to http(s) URLs. No raw user/model string ever reaches innerHTML.
+const ESC = { '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' };
+function esc(s) { return String(s).replace(/[&<>"']/g, c => ESC[c]); }
+
+function renderMarkdown(src) {
+  let text = String(src);
+
+  // Fenced code blocks — keep raw, escape only the content.
+  const codeBlocks = [];
+  text = text.replace(/```([\w+-]*)\n?([\s\S]*?)```/g, (_, lang, code) => {
+    const i = codeBlocks.length;
+    codeBlocks.push('<pre><code class="lang-' + esc(lang || '') + '">' + esc(code.replace(/\n$/, '')) + '</code></pre>');
+    return '\u0000' + i + '\u0000';
+  });
+
+  // Inline code
+  text = esc(text);
+  text = text.replace(/`([^`\n]+)`/g, (_, c) => '<code>' + c + '</code>');
+
+  // Headings
+  text = text.replace(/^###### (.*)$/gm, '<h6>$1</h6>');
+  text = text.replace(/^##### (.*)$/gm, '<h5>$1</h5>');
+  text = text.replace(/^#### (.*)$/gm, '<h4>$1</h4>');
+  text = text.replace(/^### (.*)$/gm, '<h3>$1</h3>');
+  text = text.replace(/^## (.*)$/gm, '<h2>$1</h2>');
+  text = text.replace(/^# (.*)$/gm, '<h1>$1</h1>');
+
+  // Bold / italic
+  text = text.replace(/\*\*([^*]+)\*\*/g, '<strong>$1</strong>');
+  text = text.replace(/(^|[^*])\*([^*\n]+)\*/g, '$1<em>$2</em>');
+  text = text.replace(/__([^_]+)__/g, '<strong>$1</strong>');
+  text = text.replace(/~~([^~]+)~~/g, '<del>$1</del>');
+
+  // Links
+  text = text.replace(/\[([^\]]+)\]\((https?:[^)\s]+)\)/g,
+    '<a href="$2" target="_blank" rel="noopener">$1</a>');
+
+  // Blockquotes
+  text = text.replace(/^&gt; (.*)$/gm, '<blockquote>$1</blockquote>');
+
+  // Unordered lists — group consecutive items into ONE <ul>
+  text = text.replace(/(?:^[-*] (.*)$\n?)+/gm, (m) => {
+    const items = m.trim().split('\n').map(l => l.replace(/^[-*] /, '').trim());
+    return '<ul>' + items.map(i => '<li>' + i + '</li>').join('') + '</ul>\n';
+  });
+  // Ordered lists — group consecutive numbered items into ONE <ol>
+  text = text.replace(/(?:^\d+\. (.*)$\n?)+/gm, (m) => {
+    const items = m.trim().split('\n').map(l => l.replace(/^\d+\. /, '').trim());
+    return '<ol>' + items.map(i => '<li>' + i + '</li>').join('') + '</ol>\n';
+  });
+
+  // Tables (simple)
+  text = text.replace(/((?:\|.*\|\n)+)/g, (m) => {
+    const rows = m.trim().split('\n').filter(r => r.trim());
+    if (rows.length < 1) return m;
+    let html = '<table>';
+    rows.forEach((row, i) => {
+      if (/^\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?$/.test(row.trim())) return; // separator
+      const cells = row.replace(/^\||\|$/g, '').split('|').map(c => c.trim());
+      const tag = i === 0 ? 'th' : 'td';
+      html += '<tr>' + cells.map(c => '<' + tag + '>' + c + '</' + tag + '>').join('') + '</tr>';
+    });
+    return html + '</table>';
+  });
+
+  // Paragraphs (split double newlines)
+  text = text.split(/\n{2,}/).map(p => {
+    const t = p.trim();
+    if (!t) return '';
+    if (/^<(h\d|ul|ol|blockquote|pre|table)/.test(t)) return t;
+    return '<p>' + t.replace(/\n/g, '<br>') + '</p>';
+  }).join('\n');
+
+  // Restore code blocks
+  text = text.replace(/\u0000(\d+)\u0000/g, (_, i) => codeBlocks[+i] || '');
+
+  return text;
+}
+
+function formatNumber(n) {
+  if (n == null) return '—';
+  if (n >= 1e6) return (n / 1e6).toFixed(1) + 'M';
+  if (n >= 1e3) return (n / 1e3).toFixed(1) + 'K';
+  return String(Math.round(n));
+}
+
+// ── Message rendering ──
 function addMsg(role, text, extra) {
+  extra = extra || {};
  const el = document.createElement('div');
  el.className = 'msg ' + role;
-  if (extra?.error) el.classList.add('error');
-  if (text) el.textContent = text;
-  if (extra?.toolCalls?.length) {
-    for (const tc of extra.toolCalls) {
-      const t = document.createElement('div');
-      t.className = 'tool-call';
-      t.textContent = tc.function?.name || 'tool call';
-      el.appendChild(t);
+  if (extra.error) el.classList.add('error');
+
+  if (role === 'assistant' && !extra.error) {
+    const md = document.createElement('div');
+    md.className = 'md';
+    el.appendChild(md);
+
+    if (extra.reasoning) {
+      const details = document.createElement('details');
+      details.className = 'reasoning';
+      details.open = false;
+      const sum = document.createElement('summary');
+      sum.textContent = 'Thinking';
+      const body = document.createElement('div');
+      body.className = 'reasoning-body';
+      body.textContent = extra.reasoning;
+      details.appendChild(sum);
+      details.appendChild(body);
+      el.appendChild(details);
    }
+
+    if (extra.toolCalls && extra.toolCalls.length) {
+      for (const tc of extra.toolCalls) {
+        const t = document.createElement('div');
+        t.className = 'tool-call';
+        t.textContent = 'Calling tool: ' + (tc.function && tc.function.name || 'tool');
+        el.appendChild(t);
+      }
+    }
+
+    // Live content element
+    const content = document.createElement('div');
+    content.className = 'md-content';
+    md.appendChild(content);
+
+    el._md = content;
+
+    // Actions (copy)
+    const actions = document.createElement('div');
+    actions.className = 'msg-actions';
+    const copyBtn = document.createElement('button');
+    copyBtn.textContent = '📋 Copy';
+    copyBtn.addEventListener('click', () => {
+      navigator.clipboard.writeText(extra.copyText || text || '').then(() => {
+        copyBtn.textContent = '✓ Copied';
+        copyBtn.classList.add('copied');
+        setTimeout(() => { copyBtn.textContent = '📋 Copy'; copyBtn.classList.remove('copied'); }, 1500);
+      });
+    });
+    actions.appendChild(copyBtn);
+    el.appendChild(actions);
+  } else {
+    if (text) el.textContent = text;
  }
+
  $container.appendChild(el);
-  el.scrollIntoView({ behavior: 'smooth' });
+  $welcome.style.display = 'none';
+  el.scrollIntoView({ behavior: 'smooth', block: 'end' });
  return el;
 }

+function setContent(el, html) {
+  if (el) el.innerHTML = html || '';
+}
+
+// ── Conversation ──
+function buildRequestBody() {
+  const body = {
+    model: modelId,
+    messages: [],
+    stream: $paramStream.checked,
+    max_tokens: parseInt($paramMax.value, 10) || 512,
+  };
+  const temp = parseFloat($paramTemp.value);
+  if (!isNaN(temp)) body.temperature = temp;
+  const topp = parseFloat($paramTopp.value);
+  if (!isNaN(topp)) body.top_p = topp;
+  const sys = $paramSystem.value.trim();
+  if (sys) body.messages.push({ role: 'system', content: sys });
+  body.messages = body.messages.concat(messages.slice(-12));
+  return body;
+}
+
 async function send() {
  const text = $in.value.trim();
-  if (!text || $btn.disabled) return;
+  if (!text || generating) return;

  $in.value = '';
  $in.style.height = 'auto';
-  $btn.disabled = true;
+  setGenerating(true);
  $status.className = 'status visible';
  $status.textContent = 'AI is thinking...';

@@ -162,31 +530,59 @@ async function send() {
  addMsg('user', text);

  const el = addMsg('assistant', '');
+  const started = Date.now();
+  let full = '';
+  let reasoning = '';
+  let toolCalls = null;
+  let lastUsage = null;
+
+  abortCtrl = new AbortController();
+  const body = buildRequestBody();

  try {
    const res = await fetch(CHAT_URL, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({
-        model: '',
-        messages: messages.slice(-5),  // keep context window manageable
-        stream: true,
-        max_tokens: 1024,
-      }),
+      body: JSON.stringify(body),
+      signal: abortCtrl.signal,
    });

    if (!res.ok) {
-      const errText = await res.text().catch(() => 'Unknown error');
-      el.textContent = `Error ${res.status}: ${errText}`;
+      let errText = '';
+      try { const j = await res.json(); errText = j.error && j.error.message || JSON.stringify(j); }
+      catch (e) { errText = await res.text().catch(() => 'Unknown error'); }
      el.classList.add('error');
+      setContent(el._md, esc('Error ' + res.status + ': ' + errText));
      return;
    }

+    // Non-streaming response
+    if (!body.stream) {
+      const data = await res.json();
+      const choice = data.choices && data.choices[0];
+      if (choice) {
+        const msg = choice.message || {};
+        reasoning = msg.reasoning_content || '';
+        full = msg.content || '';
+        toolCalls = msg.tool_calls || null;
+        lastUsage = data.usage || null;
+        if (choice.finish_reason === 'tool_calls' && toolCalls) {
+          full = full + '\n\n<em>[tool calls emitted — see logs]</em>';
+        }
+      }
+      el._reasoning = reasoning;
+      renderMessage(el, full, reasoning, toolCalls, lastUsage, started);
+      if (full) messages.push({ role: 'assistant', content: full });
+      return;
+    }
+
+    // Streaming
    const reader = res.body.getReader();
    const decoder = new TextDecoder();
    let buffer = '';
-    let full = '';
-    let toolCalls = null;
+
+    $status.textContent = 'Receiving...';
+    $stop.style.display = 'inline-block';

    while (true) {
      const { done, value } = await reader.read();
@@ -203,44 +599,134 @@ async function send() {

        try {
          const chunk = JSON.parse(data);
-          const delta = chunk.choices?.[0]?.delta;
-          const finish = chunk.choices?.[0]?.finish_reason;
-
-          if (delta?.content) {
-            full += delta.content;
-            el.textContent = full;
-          }
-          if (delta?.tool_calls) {
-            toolCalls = delta.tool_calls;
-          }
-          if (finish === 'tool_calls' && toolCalls) {
-            const t = document.createElement('div');
-            t.className = 'tool-call';
-            t.textContent = 'Calling tool: ' + toolCalls.map(tc => tc.function?.name).join(', ');
-            el.appendChild(t);
+          const delta = chunk.choices && chunk.choices[0] && chunk.choices[0].delta;
+          if (delta) {
+            if (delta.reasoning_content) { reasoning += delta.reasoning_content; }
+            if (delta.content) { full += delta.content; }
+            if (delta.tool_calls) { toolCalls = delta.tool_calls; }
          }
+          if (chunk.usage) lastUsage = chunk.usage;
        } catch (e) { /* skip malformed chunk */ }
      }
+
+      renderMessage(el, full, reasoning, toolCalls, lastUsage, started, true);
    }

+    renderMessage(el, full, reasoning, toolCalls, lastUsage, started, false);
    if (full) messages.push({ role: 'assistant', content: full });
-    $status.className = 'status';
  } catch (e) {
-    el.textContent = 'Network error: ' + e.message;
-    el.classList.add('error');
+    if (e.name === 'AbortError') {
+      el._md.textContent = '';
+      el._md.textContent = full || '(stopped)';
+      renderMessage(el, full, reasoning, toolCalls, lastUsage, started, false);
+      messages.push({ role: 'assistant', content: full });
+      $status.textContent = 'Stopped.';
+    } else {
+      el.classList.add('error');
+      setContent(el._md, esc('Network error: ' + e.message));
+      $status.textContent = 'Error';
+    }
  } finally {
-    $btn.disabled = false;
+    setGenerating(false);
+    $stop.style.display = 'none';
    $status.className = 'status';
    $in.focus();
  }
 }

+function renderMessage(el, full, reasoning, toolCalls, usage, started, live) {
+  // Reasoning: first non-live render creates the collapsible; during streaming
+  // keep it updated.
+  const finalReasoning = reasoning && reasoning.trim() ? reasoning : '';
+  if (finalReasoning && !el._reasoningEl) {
+    const details = document.createElement('details');
+    details.className = 'reasoning';
+    details.open = false;
+    const sum = document.createElement('summary');
+    sum.textContent = 'Thinking';
+    const body = document.createElement('div');
+    body.className = 'reasoning-body';
+    details.appendChild(sum);
+    details.appendChild(body);
+    el._reasoningEl = details;
+    el._reasoningBody = body;
+    el.insertBefore(details, el._md.parentNode || el.firstChild);
+  }
+  if (el._reasoningBody) el._reasoningBody.textContent = finalReasoning;
+
+  setContent(el._md, full ? renderMarkdown(full) : (live ? '<span class="cursor">▊</span>' : ''));
+
+  // Tool calls
+  if (toolCalls && toolCalls.length && !el._toolRendered) {
+    el._toolRendered = true;
+    for (const tc of toolCalls) {
+      const t = document.createElement('div');
+      t.className = 'tool-call';
+      t.textContent = 'Calling tool: ' + ((tc.function && tc.function.name) || 'tool');
+      el.appendChild(t);
+    }
+  }
+
+  // Stats — only when generation finished.
+  if (!live && (full || usage)) {
+    renderStats(el, usage, started);
+  }
+}
+
+function renderStats(el, usage, started) {
+  const elapsed = Date.now() - started;
+  if (el._statsEl) el._statsEl.remove();
+
+  const stats = document.createElement('div');
+  stats.className = 'msg-stats';
+  const add = (label, val) => {
+    const s = document.createElement('span');
+    s.className = 'stat';
+    s.textContent = label + ' ' + val;
+    stats.appendChild(s);
+  };
+
+  const dur = usage && usage.duration_ms ? usage.duration_ms : elapsed;
+  add('⏱', (dur / 1000).toFixed(1) + 's');
+  if (usage && usage.tokens_per_second) add('⚡', usage.tokens_per_second.toFixed(1) + ' tok/s');
+  if (usage) {
+    add('📝', formatNumber(usage.prompt_tokens) + ' in / ' + formatNumber(usage.completion_tokens) + ' out');
+    add('Σ', formatNumber(usage.total_tokens) + ' tok');
+  }
+  el.appendChild(stats);
+}
+
+function setGenerating(v) {
+  generating = v;
+  $btn.disabled = v;
+  $btn.textContent = v ? 'Generating…' : 'Send';
+}
+
+$stop.addEventListener('click', () => {
+  if (abortCtrl) abortCtrl.abort();
+});
+
+$clearBtn.addEventListener('click', () => {
+  messages = [];
+  $container.querySelectorAll('.msg').forEach(el => el.remove());
+  $welcome.style.display = '';
+  $in.focus();
+});
+
 // Auto-resize textarea
 $in.addEventListener('input', () => {
  $in.style.height = 'auto';
  $in.style.height = Math.min($in.scrollHeight, 160) + 'px';
 });
+$in.addEventListener('keydown', (e) => {
+  if (e.key === 'Enter' && !e.shiftKey) {
+    e.preventDefault();
+    send();
+  }
+});
 $in.focus();
+
+})();
 </script>
 </body>
 </html>
@@ -1,22 +1,31 @@
 //! Chat completions endpoint — streaming and non-streaming.
+//!
+//! Both paths share the same synchronous generation core
+//! ([`LlamaEngine::generate`]), which runs on the tokio blocking pool via
+//! `spawn_blocking` so worker threads are not hogged by CPU-bound inference.
+//! The streaming path forwards each generated token into an SSE channel.

 use std::convert::Infallible;
 use std::sync::Arc;

 use axum::extract::State;
+use axum::http::HeaderMap;
 use axum::response::sse::{Event, KeepAlive, Sse};
-use axum::{Json, response::{IntoResponse, Response}};
+use axum::response::{IntoResponse, Response};
+use axum::Json;
 use chrono::Utc;
 use tokio::sync::mpsc;
 use tokio_stream::wrappers::ReceiverStream;
 use tracing::info;

 use crate::application::chat;
+use crate::config::CONFIG;
 use crate::domain::entity::{
-    ChatRequest, ChatResponse, Choice, ResponseMessage, SseChunk, SseChoice, SseDelta, Usage,
+    ChatRequest, ChatResponse, Choice, FinishReason, ResponseMessage, SseChunk, SseDelta, Usage,
 };
 use crate::infrastructure::llama::SendSampler;
 use crate::presentation::error::AppError;
+use crate::presentation::handler::metrics;
 use crate::presentation::state::AppState;

 /// POST /v1/chat/completions
@@ -24,11 +33,15 @@ pub async fn chat_completions(
    State(state): State<Arc<AppState>>,
    Json(req): Json<ChatRequest>,
 ) -> Result<Response, AppError> {
-    let max_tokens = req.max_tokens.unwrap_or(256).min(1024);
-    let stop = req.stop.clone().unwrap_or_default();
-    let prompt = chat::build_prompt(&req.messages, &req.tools);
+    // Strict model validation — reject unknown model ids up front.
+    chat::validate_model(&req.model).map_err(AppError::BadRequest)?;

-    // Tokenize
+    // Cap max_tokens at the configured hard limit (0 = unlimited).
+    let max_tokens = req.max_tokens.unwrap_or(256).min(CONFIG.max_tokens.max(1));
+    let stop = req.stop.clone().unwrap_or_default();
+    let prompt = chat::build_prompt(&req.messages, &req.tools).map_err(AppError::LlmError)?;
+
+    // Tokenize (fast — keep on the async thread).
    let input_tokens = state
        .engine
        .tokenize(&prompt)
@@ -36,18 +49,20 @@ pub async fn chat_completions(

    let prompt_tokens = input_tokens.len() as u32;
    info!(
-        "  Chat: {} prompt tokens, max_tokens={}, tools={}",
+        "Chat: {} prompt tokens, max_tokens={}, tools={}",
        prompt_tokens,
        max_tokens,
        req.tools.as_ref().is_some_and(|t| !t.is_empty())
    );

+    metrics::count_request(req.stream.unwrap_or(false));
+
    let response = if req.stream.unwrap_or(false) {
        handle_streaming(state.clone(), req, max_tokens, stop, input_tokens).await?
    } else {
        handle_non_streaming(state.clone(), req, max_tokens, stop, input_tokens).await?
    };
-    Ok(response.into_response())
+    Ok(response)
 }

 // ── Non-streaming path ──
@@ -64,24 +79,56 @@ async fn handle_non_streaming(
    let prompt_tokens = input_tokens.len() as u32;
    let has_tools = req.tools.as_ref().is_some_and(|t| !t.is_empty());
    let params = chat::SamplerParams::from_request(&req);
-    let mut sampler = SendSampler(chat::build_sampler(&params));

-    let (output_tokens, raw_text) = state
-        .engine
-        .generate(&input_tokens, &mut sampler, max_tokens, &stop)
-        .await?;
+    let engine = state.engine.clone();
+    let gen_start = std::time::Instant::now();
+    let outcome = tokio::task::spawn_blocking(move || {
+        let mut sampler = SendSampler(chat::build_sampler(&params));
+        engine.generate(
+            &input_tokens,
+            &mut sampler,
+            max_tokens,
+            &stop,
+            has_tools,
+            &mut |_token, _piece| true,
+        )
+    })
+    .await
+    .map_err(|e| AppError::Internal(format!("Generation task panicked: {e}")))?
+    .map_err(AppError::from)?;
+    let duration_ms = gen_start.elapsed().as_millis() as u64;

-    let (output_text, tool_calls) = chat::parse_tool_calls(&chat::clean_text(&raw_text));
+    let completion_tokens = outcome.tokens.len() as u32;
+    info!(
+        "  {} generated tokens in {}ms",
+        completion_tokens, duration_ms
+    );

-    let completion_tokens = output_tokens.len() as u32;
-    info!("  {} generated tokens", completion_tokens);
+    metrics::record_tokens(prompt_tokens, completion_tokens, duration_ms);
+    if outcome.finish == FinishReason::Aborted {
+        metrics::count_aborted();
+    }

-    let finish_reason = if has_tools && !tool_calls.is_empty() {
-        "tool_calls"
-    } else if completion_tokens < max_tokens {
-        "stop"
+    let (reasoning, cleaned) = chat::clean_text(&outcome.text);
+    let (output_text, tool_calls) = chat::parse_tool_calls(&cleaned);
+
+    let finish_reason = match (outcome.finish, tool_calls.is_empty()) {
+        (FinishReason::ToolCalls, false) => "tool_calls",
+        // Model opened a <tool_call> but never completed it — don't claim a call.
+        (FinishReason::ToolCalls, true) => "stop",
+        (f, _) => f.as_str(),
+    };
+
+    let reasoning_opt = if reasoning.is_empty() {
+        None
    } else {
-        "length"
+        Some(reasoning)
+    };
+
+    let tok_per_s = if duration_ms > 0 {
+        Some(completion_tokens as f64 / (duration_ms as f64 / 1000.0))
+    } else {
+        None
    };

    Ok(Json(ChatResponse {
@@ -94,6 +141,7 @@ async fn handle_non_streaming(
            message: ResponseMessage {
                role: "assistant".into(),
                content: Some(output_text),
+                reasoning_content: reasoning_opt,
                tool_calls: if tool_calls.is_empty() {
                    None
                } else {
@@ -106,6 +154,8 @@ async fn handle_non_streaming(
            prompt_tokens,
            completion_tokens,
            total_tokens: prompt_tokens + completion_tokens,
+            duration_ms: Some(duration_ms),
+            tokens_per_second: tok_per_s,
        },
    })
    .into_response())
@@ -124,187 +174,208 @@ async fn handle_streaming(
    let created = Utc::now().timestamp();
    let has_tools = req.tools.as_ref().is_some_and(|t| !t.is_empty());
    let model_name = req.model.clone();
+    let prompt_tokens = input_tokens.len() as u32;
    let params = chat::SamplerParams::from_request(&req);
+
    let (tx, rx) = mpsc::channel::<Result<Event, Infallible>>(64);

-    tokio::spawn(async move {
-        // Role chunk
-        let role_chunk = serde_json::to_string(&SseChunk {
-            id: chat_id.clone(),
-            object: "chat.completion.chunk".into(),
-            created,
-            model: model_name.clone(),
-            choices: vec![SseChoice {
-                index: 0,
-                delta: SseDelta {
-                    role: Some("assistant".into()),
-                    content: None,
-                    tool_calls: None,
-                },
-                finish_reason: None,
-            }],
-        })
-        .unwrap();
-        if tx.send(Ok(Event::default().data(role_chunk))).await.is_err() {
-            return;
-        }
+    // First chunk: announce the assistant role.
+    let role_chunk = SseChunk::delta(
+        chat_id.clone(),
+        created,
+        model_name.clone(),
+        SseDelta {
+            role: Some("assistant".into()),
+            content: None,
+            tool_calls: None,
+            reasoning_content: None,
+        },
+    );
+    let role_event = serde_json::to_string(&role_chunk).unwrap();
+    let _ = tx.send(Ok(Event::default().data(role_event))).await;

-        // Build sampler
+    let engine = state.engine.clone();
+    tokio::task::spawn_blocking(move || {
+        let gen_start = std::time::Instant::now();
        let mut sampler = SendSampler(chat::build_sampler(&params));
-
-        // Lock context
-        let mut inner = state.engine.ctx().lock().await;
-        inner.clear();
-        if let Err(e) = inner.prefill(&input_tokens) {
-            info!("  Prefill error: {e}");
-            return;
-        }
-
-        let mut count = 0u32;
        let mut text_buf = String::new();
-        let mut current = inner.sample(&mut sampler);
+        let mut sent_len: usize = 0;
+        // Byte offset in text_buf where the content phase begins (right after
+        // the first `</think>`); None while still thinking.
+        let mut content_start: Option<usize> = None;

-        loop {
-            if count >= max_tokens {
-                let chunk = serde_json::to_string(&SseChunk {
-                    id: chat_id.clone(),
-                    object: "chat.completion.chunk".into(),
-                    created,
-                    model: model_name.clone(),
-                    choices: vec![SseChoice {
-                        index: 0,
-                        delta: SseDelta {
-                            role: None,
-                            content: None,
-                            tool_calls: None,
-                        },
-                        finish_reason: Some("length".into()),
-                    }],
-                })
-                .unwrap();
-                let _ = tx.send(Ok(Event::default().data(chunk))).await;
-                break;
-            }
+        let outcome = engine.generate(
+            &input_tokens,
+            &mut sampler,
+            max_tokens,
+            &stop,
+            has_tools,
+            &mut |_token, piece| {
+                text_buf.push_str(piece);

-            if state.engine.is_eog(current) {
-                let reason = if has_tools && text_buf.contains("<tool_call>") {
-                    "tool_calls"
-                } else {
-                    "stop"
-                };
-                let chunk = serde_json::to_string(&SseChunk {
-                    id: chat_id.clone(),
-                    object: "chat.completion.chunk".into(),
-                    created,
-                    model: model_name.clone(),
-                    choices: vec![SseChoice {
-                        index: 0,
-                        delta: SseDelta {
-                            role: None,
-                            content: None,
-                            tool_calls: None,
-                        },
-                        finish_reason: Some(reason.into()),
-                    }],
-                })
-                .unwrap();
-                let _ = tx.send(Ok(Event::default().data(chunk))).await;
-                break;
-            }
-
-            let piece = state.engine.decode_token(current);
-            let content = chat::clean_text(&piece);
-
-            if !content.is_empty() {
-                let chunk = serde_json::to_string(&SseChunk {
-                    id: chat_id.clone(),
-                    object: "chat.completion.chunk".into(),
-                    created,
-                    model: model_name.clone(),
-                    choices: vec![SseChoice {
-                        index: 0,
-                        delta: SseDelta {
-                            role: None,
-                            content: Some(content.clone()),
-                            tool_calls: None,
-                        },
-                        finish_reason: None,
-                    }],
-                })
-                .unwrap();
-                if tx.send(Ok(Event::default().data(chunk))).await.is_err() {
-                    break;
+                // Robust boundary detection on the *full* buffer — a `</think>`
+                // tag may be split across tokens, which would defeat a search
+                // over the incremental fragment only.
+                if content_start.is_none() {
+                    if let Some(pos) = text_buf.find("</think>") {
+                        content_start = Some(pos + 8);
+                    }
                }
-            }

-            text_buf.push_str(&piece);
-
-            // Check stop sequences
-            let mut stop_now = false;
-            for s in &stop {
-                if text_buf.contains(s) {
-                    stop_now = true;
-                    break;
+                let new_text = &text_buf[sent_len..];
+                if new_text.is_empty() {
+                    return true;
                }
-            }
-            if stop_now {
-                let chunk = serde_json::to_string(&SseChunk {
-                    id: chat_id.clone(),
-                    object: "chat.completion.chunk".into(),
-                    created,
-                    model: model_name.clone(),
-                    choices: vec![SseChoice {
-                        index: 0,
-                        delta: SseDelta {
-                            role: None,
-                            content: None,
-                            tool_calls: None,
-                        },
-                        finish_reason: Some("stop".into()),
-                    }],
-                })
-                .unwrap();
-                let _ = tx.send(Ok(Event::default().data(chunk))).await;
-                break;
-            }

-            // Check tool call completeness
-            if has_tools && text_buf.contains("<tool_call>") {
-                let open = text_buf.matches("<tool_call>").count();
-                let close = text_buf.matches("</tool_call>").count();
-                if close >= open {
-                    let chunk = serde_json::to_string(&SseChunk {
-                        id: chat_id.clone(),
-                        object: "chat.completion.chunk".into(),
+                let (reasoning, content) =
+                    chat::split_stream_chunk(new_text, sent_len, content_start);
+
+                if let Some(reasoning) = reasoning {
+                    let chunk = SseChunk::delta(
+                        chat_id.clone(),
                        created,
-                        model: model_name.clone(),
-                        choices: vec![SseChoice {
-                            index: 0,
-                            delta: SseDelta {
+                        model_name.clone(),
+                        SseDelta {
+                            role: None,
+                            content: None,
+                            tool_calls: None,
+                            reasoning_content: Some(reasoning),
+                        },
+                    );
+                    let event = serde_json::to_string(&chunk).unwrap();
+                    if tx.blocking_send(Ok(Event::default().data(event))).is_err() {
+                        return false;
+                    }
+                }
+
+                if !content.is_empty() {
+                    let chunk = SseChunk::delta(
+                        chat_id.clone(),
+                        created,
+                        model_name.clone(),
+                        SseDelta {
+                            role: None,
+                            content: Some(content),
+                            tool_calls: None,
+                            reasoning_content: None,
+                        },
+                    );
+                    let event = serde_json::to_string(&chunk).unwrap();
+                    if tx.blocking_send(Ok(Event::default().data(event))).is_err() {
+                        return false;
+                    }
+                }
+
+                sent_len = text_buf.len();
+                true
+            },
+        );
+
+        match outcome {
+            Ok(outcome) => {
+                let duration_ms = gen_start.elapsed().as_millis() as u64;
+                let completion_tokens = outcome.tokens.len() as u32;
+                let usage = Usage {
+                    prompt_tokens,
+                    completion_tokens,
+                    total_tokens: prompt_tokens + completion_tokens,
+                    duration_ms: Some(duration_ms),
+                    tokens_per_second: if duration_ms > 0 {
+                        Some(completion_tokens as f64 / (duration_ms as f64 / 1000.0))
+                    } else {
+                        None
+                    },
+                };
+                info!(
+                    "  stream: {} generated tokens in {}ms",
+                    completion_tokens, duration_ms
+                );
+                metrics::record_tokens(prompt_tokens, completion_tokens, duration_ms);
+                if outcome.finish == FinishReason::Aborted {
+                    metrics::count_aborted();
+                }
+
+                // If the model never emitted `</think>`, everything was streamed
+                // as reasoning_content. Flush it as content so the client always
+                // receives the response text.
+                if content_start.is_none() && !text_buf.is_empty() {
+                    let cleaned = chat::strip_markup(&text_buf);
+                    if !cleaned.is_empty() {
+                        let chunk = SseChunk::delta(
+                            chat_id.clone(),
+                            created,
+                            model_name.clone(),
+                            SseDelta {
+                                role: None,
+                                content: Some(cleaned),
+                                tool_calls: None,
+                                reasoning_content: None,
+                            },
+                        );
+                        let event = serde_json::to_string(&chunk).unwrap();
+                        let _ = tx.blocking_send(Ok(Event::default().data(event)));
+                    }
+                }
+
+                // Single-shot tool-calls delta (this model emits whole blocks).
+                let mut sent_tool_calls = false;
+                if outcome.finish == FinishReason::ToolCalls {
+                    let (_cleaned, calls) = chat::parse_tool_calls(&outcome.text);
+                    if !calls.is_empty() {
+                        sent_tool_calls = true;
+                        let chunk = SseChunk::delta(
+                            chat_id.clone(),
+                            created,
+                            model_name.clone(),
+                            SseDelta {
                                role: None,
                                content: None,
-                                tool_calls: None,
+                                tool_calls: Some(calls),
+                                reasoning_content: None,
                            },
-                            finish_reason: Some("tool_calls".into()),
-                        }],
-                    })
-                    .unwrap();
-                    let _ = tx.send(Ok(Event::default().data(chunk))).await;
-                    break;
+                        );
+                        let event = serde_json::to_string(&chunk).unwrap();
+                        let _ = tx.blocking_send(Ok(Event::default().data(event)));
+                    }
                }
-            }

-            let pos = input_tokens.len() as i32 + count as i32;
-            if let Err(e) = inner.decode(current, pos) {
-                info!("  Decode error: {e}");
-                break;
+                let finish_reason = match (outcome.finish, sent_tool_calls) {
+                    (FinishReason::ToolCalls, true) => "tool_calls",
+                    (FinishReason::ToolCalls, false) => "stop",
+                    (f, _) => f.as_str(),
+                };
+
+                let finish_chunk =
+                    SseChunk::finish(chat_id.clone(), created, model_name.clone(), finish_reason);
+                let event = serde_json::to_string(&finish_chunk).unwrap();
+                let _ = tx.blocking_send(Ok(Event::default().data(event)));
+
+                let usage_chunk = SseChunk::usage(chat_id, created, model_name, usage);
+                let event = serde_json::to_string(&usage_chunk).unwrap();
+                let _ = tx.blocking_send(Ok(Event::default().data(event)));
+            }
+            Err(e) => {
+                // Surface the error instead of silently truncating the stream.
+                metrics::count_error();
+                let error_body = serde_json::json!({
+                    "error": {
+                        "message": e.to_string(),
+                        "type": "server_error",
+                    }
+                });
+                let _ = tx.blocking_send(Ok(Event::default().data(error_body.to_string())));
            }
-            count += 1;
-            current = inner.sample(&mut sampler);
        }
+
+        // OpenAI-compatible terminator.
+        let _ = tx.blocking_send(Ok(Event::default().data("[DONE]")));
    });

    let stream = ReceiverStream::new(rx);
+    let mut headers = HeaderMap::new();
+    headers.insert("X-Accel-Buffering", "no".parse().unwrap());
+    headers.insert("Cache-Control", "no-cache".parse().unwrap());
+    headers.insert("Connection", "keep-alive".parse().unwrap());
    let sse = Sse::new(stream).keep_alive(KeepAlive::default());
-    Ok(sse.into_response())
+    Ok((headers, sse).into_response())
 }
@@ -9,7 +9,10 @@ const HTML: &str = include_str!("chat-ui/index.html");
 pub async fn chat_ui() -> Response {
    (
        StatusCode::OK,
-        [(header::CONTENT_TYPE, HeaderValue::from_static("text/html; charset=utf-8"))],
+        [(
+            header::CONTENT_TYPE,
+            HeaderValue::from_static("text/html; charset=utf-8"),
+        )],
        HTML,
    )
        .into_response()
@@ -1,13 +1,39 @@
 //! Health check endpoint.

+use std::sync::atomic::AtomicU64;
+use std::sync::LazyLock;
+use std::time::Instant;
+
 use axum::Json;

-use crate::config::MODEL_ID;
+use crate::config::{CONFIG, MODEL_ID};
 use crate::domain::entity::HealthResponse;

+/// Process start instant — used to compute uptime for /health and /metrics.
+pub static START_INSTANT: LazyLock<Instant> = LazyLock::new(Instant::now);
+
+/// Process start timestamp (unix seconds) — exported as a Prometheus gauge.
+pub static START_TIMESTAMP: LazyLock<AtomicU64> = LazyLock::new(|| {
+    AtomicU64::new(
+        std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .map(|d| d.as_secs())
+            .unwrap_or(0),
+    )
+});
+
+/// Seconds since process start.
+pub fn uptime_secs() -> u64 {
+    START_INSTANT.elapsed().as_secs()
+}
+
 pub async fn health_check() -> Json<HealthResponse> {
    Json(HealthResponse {
        status: "ok".into(),
-        model: format!("{MODEL_ID}-q4_k_m"),
+        // Report the exact model id served by /v1/models (no extra suffix).
+        model: MODEL_ID.into(),
+        uptime_s: Some(uptime_secs()),
+        n_ctx: Some(CONFIG.n_ctx),
+        version: Some(env!("CARGO_PKG_VERSION").into()),
    })
 }
@@ -0,0 +1,249 @@
+//! Prometheus metrics endpoint.
+//!
+//! Exports process-level metrics (CPU, RSS) plus request counters, token
+//! usage and generation latencies. Implemented with `std` only — no external
+//! metrics dependency — so the deploy stays dependency-free.
+//!
+//! Scrape config (VPS): `prometheus.yml` file_sd targets llm-api at
+//! `/metrics` with default `__metrics_path__`.
+
+use std::fmt::Write as _;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::LazyLock;
+use std::time::Instant;
+
+use axum::http::{header, HeaderValue, StatusCode};
+use axum::response::{IntoResponse, Response};
+
+use super::health::{uptime_secs, START_INSTANT, START_TIMESTAMP};
+
+// ── Atomic counters (updated by the chat handlers) ──
+
+/// Total /v1/chat/completions requests received.
+pub static REQUESTS_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Requests that ended in an error (any 4xx/5xx).
+pub static ERRORS_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Requests that streamed (`stream: true`).
+pub static STREAMING_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Prompt tokens accepted across all requests.
+pub static PROMPT_TOKENS_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Completion tokens generated across all requests.
+pub static COMPLETION_TOKENS_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Generation time spent across all requests, in milliseconds.
+pub static GENERATION_MS_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+/// Requests whose generation was aborted early (client disconnect).
+pub static ABORTED_TOTAL: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
+
+// ── Public helpers used by handlers ──
+
+pub fn count_request(streaming: bool) {
+    REQUESTS_TOTAL.fetch_add(1, Ordering::Relaxed);
+    if streaming {
+        STREAMING_TOTAL.fetch_add(1, Ordering::Relaxed);
+    }
+}
+
+pub fn count_error() {
+    ERRORS_TOTAL.fetch_add(1, Ordering::Relaxed);
+}
+
+pub fn count_aborted() {
+    ABORTED_TOTAL.fetch_add(1, Ordering::Relaxed);
+}
+
+pub fn record_tokens(prompt_tokens: u32, completion_tokens: u32, duration_ms: u64) {
+    PROMPT_TOKENS_TOTAL.fetch_add(prompt_tokens as u64, Ordering::Relaxed);
+    COMPLETION_TOKENS_TOTAL.fetch_add(completion_tokens as u64, Ordering::Relaxed);
+    GENERATION_MS_TOTAL.fetch_add(duration_ms, Ordering::Relaxed);
+}
+
+fn f(field: &mut String, name: &str, value: impl std::fmt::Display) {
+    let _ = writeln!(field, "{name} {value}");
+}
+
+/// GET /metrics — Prometheus text exposition format.
+pub async fn metrics() -> Response {
+    let uptime = uptime_secs();
+    let start_ts = START_TIMESTAMP.load(Ordering::Relaxed);
+
+    // Per-second rates over the process lifetime.
+    let total = REQUESTS_TOTAL.load(Ordering::Relaxed);
+    let errors = ERRORS_TOTAL.load(Ordering::Relaxed);
+    let streaming = STREAMING_TOTAL.load(Ordering::Relaxed);
+    let aborted = ABORTED_TOTAL.load(Ordering::Relaxed);
+    let prompt_tokens = PROMPT_TOKENS_TOTAL.load(Ordering::Relaxed);
+    let completion_tokens = COMPLETION_TOKENS_TOTAL.load(Ordering::Relaxed);
+    let gen_ms = GENERATION_MS_TOTAL.load(Ordering::Relaxed);
+
+    let rps = if uptime > 0 {
+        total as f64 / uptime as f64
+    } else {
+        0.0
+    };
+    let tok_per_s = if gen_ms > 0 {
+        completion_tokens as f64 / (gen_ms as f64 / 1000.0)
+    } else {
+        0.0
+    };
+    let avg_ms = if total > 0 {
+        gen_ms as f64 / total as f64
+    } else {
+        0.0
+    };
+
+    let mut body = String::with_capacity(2048);
+    body.push_str("# HELP llm_api_requests_total Total /v1/chat/completions requests received.\n");
+    body.push_str("# TYPE llm_api_requests_total counter\n");
+    f(&mut body, "llm_api_requests_total", total);
+    body.push_str("# HELP llm_api_errors_total Requests that ended in an error.\n");
+    body.push_str("# TYPE llm_api_errors_total counter\n");
+    f(&mut body, "llm_api_errors_total", errors);
+    body.push_str("# HELP llm_api_streaming_requests_total Requests that streamed.\n");
+    body.push_str("# TYPE llm_api_streaming_requests_total counter\n");
+    f(&mut body, "llm_api_streaming_requests_total", streaming);
+    body.push_str(
+        "# HELP llm_api_aborted_requests_total Generations aborted early (client disconnect).\n",
+    );
+    body.push_str("# TYPE llm_api_aborted_requests_total counter\n");
+    f(&mut body, "llm_api_aborted_requests_total", aborted);
+    body.push_str("# HELP llm_api_prompt_tokens_total Prompt tokens accepted.\n");
+    body.push_str("# TYPE llm_api_prompt_tokens_total counter\n");
+    f(&mut body, "llm_api_prompt_tokens_total", prompt_tokens);
+    body.push_str("# HELP llm_api_completion_tokens_total Completion tokens generated.\n");
+    body.push_str("# TYPE llm_api_completion_tokens_total counter\n");
+    f(
+        &mut body,
+        "llm_api_completion_tokens_total",
+        completion_tokens,
+    );
+    body.push_str("# HELP llm_api_generation_ms_total Generation time in milliseconds.\n");
+    body.push_str("# TYPE llm_api_generation_ms_total counter\n");
+    f(&mut body, "llm_api_generation_ms_total", gen_ms);
+    body.push_str("# HELP llm_api_requests_per_second Lifetime request rate.\n");
+    body.push_str("# TYPE llm_api_requests_per_second gauge\n");
+    f(
+        &mut body,
+        "llm_api_requests_per_second",
+        format!("{rps:.3}"),
+    );
+    body.push_str("# HELP llm_api_tokens_per_second Lifetime generation throughput.\n");
+    body.push_str("# TYPE llm_api_tokens_per_second gauge\n");
+    f(
+        &mut body,
+        "llm_api_tokens_per_second",
+        format!("{tok_per_s:.3}"),
+    );
+    body.push_str(
+        "# HELP llm_api_average_generation_ms Average generation duration per request.\n",
+    );
+    body.push_str("# TYPE llm_api_average_generation_ms gauge\n");
+    f(
+        &mut body,
+        "llm_api_average_generation_ms",
+        format!("{avg_ms:.1}"),
+    );
+    body.push_str("# HELP llm_api_uptime_seconds Server process uptime.\n");
+    body.push_str("# TYPE llm_api_uptime_seconds gauge\n");
+    f(&mut body, "llm_api_uptime_seconds", uptime);
+    body.push_str("# HELP llm_api_start_time_seconds Process start time (unix).\n");
+    body.push_str("# TYPE llm_api_start_time_seconds gauge\n");
+    f(&mut body, "llm_api_start_time_seconds", start_ts);
+
+    // Engine identity (helpful when multiple model servers exist).
+    body.push_str("# HELP llm_api_build_info Build information.\n");
+    body.push_str("# TYPE llm_api_build_info gauge\n");
+    let _ = writeln!(
+        body,
+        "llm_api_build_info{{version=\"{}\",model=\"{}\"}} 1",
+        env!("CARGO_PKG_VERSION"),
+        crate::config::MODEL_ID
+    );
+
+    (
+        StatusCode::OK,
+        [(
+            header::CONTENT_TYPE,
+            HeaderValue::from_static("text/plain; version=0.0.4; charset=utf-8"),
+        )],
+        body,
+    )
+        .into_response()
+}
+
+/// Snapshot helper used by tests to inspect counters.
+pub fn snapshot() -> (u64, u64, u64) {
+    (
+        REQUESTS_TOTAL.load(Ordering::Relaxed),
+        COMPLETION_TOKENS_TOTAL.load(Ordering::Relaxed),
+        GENERATION_MS_TOTAL.load(Ordering::Relaxed),
+    )
+}
+
+/// Used by tests to verify the monotonic clock source is live.
+pub fn start_instant() -> &'static Instant {
+    &START_INSTANT
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn reset() {
+        for c in [
+            &REQUESTS_TOTAL,
+            &ERRORS_TOTAL,
+            &STREAMING_TOTAL,
+            &PROMPT_TOKENS_TOTAL,
+            &COMPLETION_TOKENS_TOTAL,
+            &GENERATION_MS_TOTAL,
+            &ABORTED_TOTAL,
+        ] {
+            c.store(0, Ordering::Relaxed);
+        }
+    }
+
+    #[test]
+    fn counters_accumulate() {
+        reset();
+        count_request(true);
+        count_request(false);
+        count_error();
+        count_aborted();
+        record_tokens(100, 250, 5000);
+
+        assert_eq!(REQUESTS_TOTAL.load(Ordering::Relaxed), 2);
+        assert_eq!(STREAMING_TOTAL.load(Ordering::Relaxed), 1);
+        assert_eq!(ERRORS_TOTAL.load(Ordering::Relaxed), 1);
+        assert_eq!(ABORTED_TOTAL.load(Ordering::Relaxed), 1);
+        assert_eq!(PROMPT_TOKENS_TOTAL.load(Ordering::Relaxed), 100);
+        assert_eq!(COMPLETION_TOKENS_TOTAL.load(Ordering::Relaxed), 250);
+        assert_eq!(GENERATION_MS_TOTAL.load(Ordering::Relaxed), 5000);
+    }
+
+    #[test]
+    fn metrics_body_has_prometheus_shape() {
+        reset();
+        count_request(true);
+        record_tokens(10, 20, 1000);
+
+        let response = futures::executor::block_on(metrics());
+        let bytes =
+            futures::executor::block_on(axum::body::to_bytes(response.into_body(), usize::MAX))
+                .expect("read body");
+        let text = String::from_utf8(bytes.to_vec()).unwrap();
+
+        assert!(text.contains("# TYPE llm_api_requests_total counter"));
+        assert!(text.contains("llm_api_requests_total 1"));
+        assert!(text.contains("llm_api_completion_tokens_total 20"));
+        assert!(text.contains("llm_api_build_info{version="));
+        assert!(text.contains("llm_api_uptime_seconds "));
+    }
+
+    #[test]
+    fn uptime_is_monotonic() {
+        let a = uptime_secs();
+        std::thread::sleep(std::time::Duration::from_millis(20));
+        let b = uptime_secs();
+        assert!(b >= a);
+    }
+}
@@ -1,4 +1,7 @@
+//! HTTP handlers.
+
 pub mod chat;
 pub mod chat_ui;
 pub mod health;
+pub mod metrics;
 pub mod models;
@@ -4,18 +4,18 @@
 //! Only applied to routes that require authentication.

 use axum::extract::Request;
-use axum::http::StatusCode;
 use axum::middleware::Next;
 use axum::response::Response;

 use crate::config::CONFIG;
+use crate::presentation::error::AppError;

 /// Middleware that validates the Bearer token in the Authorization header.
 ///
 /// If `API_KEY` is not set (empty), authentication is disabled and
 /// all requests pass through. If set, the middleware rejects requests
 /// without a matching token.
-pub async fn auth_middleware(request: Request, next: Next) -> Result<Response, (StatusCode, String)> {
+pub async fn auth_middleware(request: Request, next: Next) -> Result<Response, AppError> {
    let api_key = &CONFIG.api_key;
    if api_key.is_empty() {
        return Ok(next.run(request).await);
@@ -32,8 +32,6 @@ pub async fn auth_middleware(request: Request, next: Next) -> Result<Response, (
        return Ok(next.run(request).await);
    }

-    Err((
-        StatusCode::UNAUTHORIZED,
-        "{\"error\":\"unauthorized\",\"message\":\"Invalid API key\"}".into(),
-    ))
+    // AppError renders a JSON body with the correct Content-Type.
+    Err(AppError::Unauthorized)
 }
@@ -7,7 +7,7 @@ use axum::routing::{get, post};
 use axum::Router;
 use tower_http::cors::CorsLayer;

-use super::handler::{chat, chat_ui, health, models};
+use super::handler::{chat, chat_ui, health, metrics, models};
 use crate::presentation::middleware::auth::auth_middleware;
 use crate::presentation::state::AppState;

@@ -17,6 +17,7 @@ pub fn build_router(state: Arc<AppState>) -> Router {
        // Public routes (no auth)
        .route("/", get(chat_ui::chat_ui))
        .route("/health", get(health::health_check))
+        .route("/metrics", get(metrics::metrics))
        .route("/v1/models", get(models::list_models))
        // Chat completions (auth-protected)
        .route("/v1/chat/completions", post(chat::chat_completions))
Author	SHA1	Message	Date
asepharyana	5f7ead5503	docs: README /metrics + MAX_TOKENS, smoke-test health/metrics/timing checks	2026-08-03 11:44:27 +07:00
asepharyana	e8f90fc9b1	feat(metrics): prometheus /metrics + generation timing in usage - /metrics endpoint: request/token/latency counters, tok/s gauge, build info (std-only, no deps) - usage.duration_ms + usage.tokens_per_second in non-streaming and streaming responses - /health now reports uptime_s, n_ctx, version - MAX_TOKENS env config (hard cap, default 2048; 0 = unlimited) - metrics unit tests (counters, prometheus shape, uptime monotonic)	2026-08-03 11:44:21 +07:00
asepharyanaandClaude Opus 5	7cec411cba	fix(stream): robust reasoning/content split + flush un-tagged output The incremental </think> search missed tags split across tokens, inverting reasoning/content classification. Detect the first </think> on the full buffer and track the content boundary as a byte offset. If the model never closes </think>, flush the buffered text as content so clients always receive the response. Chat UI now renders reasoning_content too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 11:03:26 +07:00
asepharyanaandClaude Opus 5	b636496497	refactor(chat): unify generation flow and fix tool/streaming bugs - Unified synchronous generate() core with callback; both streaming and non-streaming paths run it via spawn_blocking (context: std Mutex). - build_prompt now passes tool definitions to the template (was dead) and embeds assistant tool-call history as XML matching the parser format; fixes double <tool_response> wrap and template set-scoping bug. - Tokenize with AddBos::Never (template owns <s>) to remove double BOS. - Streaming: preserve inter-word spaces (per-chunk trim removed), add [DONE] + usage chunk, emit error events, single-shot tool_calls delta. - Strict model validation (400 on unknown model); health/UI/README aligned to minicpm5-1b-fable5-v2-thinking; auth returns JSON errors; n_ctx/ n_batch/n_threads env-configurable. - Added 18 unit tests; cargo check/clippy/fmt clean. - scripts/smoke-test.sh for post-deploy verification on the VPS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 08:58:13 +07:00
asepharyana	344bc195fa	docs: README Nix+Caddy deploy	2026-08-02 16:44:43 +07:00
asepharyana	81c5177249	chore: port 8080 to 4010	2026-08-02 16:18:17 +07:00
Asep Haryana	67861f384b	fix(stream): add anti-buffer headers for real SSE streaming Notify Parent Repo / dispatch (push) Canceled after 0s - Add X-Accel-Buffering: no, Cache-Control: no-cache, Connection: keep-alive - Prevents Traefik/proxy from buffering SSE events - Each token is flushed immediately as generated	2026-07-26 19:13:00 +07:00
Asep Haryana	5cba76d280	fix(stream): separate reasoning_content from content in SSE streaming - Before </think>: tokens sent as delta.reasoning_content - After </think>: tokens sent as delta.content - Handles </think> split across token boundaries - Clean <\|im_start\|>/<\|im_end\|>/<think>/</think> from streamed text - Normal models (no thinking) send all as content as before	2026-07-26 19:04:29 +07:00
Asep Haryana	6ff31b5f62	feat: render GGUF Jinja template via minijinja crate - Replaced manual prompt building with GGUF's chat template rendered through minijinja (Rust Jinja2 engine) - Simplified template: removed multi-step tool detection, reasoning extraction (not needed at template level) - Added reasoning_content separation: clean_text() returns (reasoning, answer) tuple - Added reasoning_content field to ResponseMessage and SseDelta for OpenAI-compatible output - Embedded template at build time via include_str! from templates/chat_template.jinja - Built-in support for enable_thinking, tool_definitions, tool_calls, tool_response	2026-07-26 18:21:35 +07:00
Asep Haryana	9a63ff1601	feat: use GGUF chat template via apply_chat_template - Replaced manual prompt building with LlamaModel::apply_chat_template - Uses model's baked-in Jinja template (system/user/assistant/tool format) - Added <think> trigger after template for thinking mode - Clean_text strips only <\|im_end\|>, <\|im_start\|>, <think>, </think>	2026-07-26 17:10:42 +07:00
Asep Haryana	59c77108a5	fix: skip leading EOS tokens in streaming + non-streaming - Skip <\|im_end\|> generated as first token (prevents empty responses) - Clean <think> tags as plain text in generated output	2026-07-26 16:09:04 +07:00
Asep Haryana	495b9ed126	fix: remove /think trigger, add back <think> in clean_text - Removed /think trigger from prompt (model generates better without it) - Added <think> and </think> back to clean_text (model uses plain text thinking tags, not special tokens)	2026-07-26 16:03:18 +07:00
Asep Haryana	254532458b	Revert "fix: remove /think trigger from prompt, use plain assistant prefix" This reverts commit `7c8f747faf`.	2026-07-26 15:49:04 +07:00
Asep Haryana	7c8f747faf	fix: remove /think trigger from prompt, use plain assistant prefix	2026-07-26 15:48:44 +07:00
Asep Haryana	d8425ea3b3	feat: switch to MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-Q8_0 GGUF - Updated model path + model ID for new 1B thinking model - Updated prompt builder to use MiniCPM5 native /think trigger - Updated clean_text to strip MiniCPM5 special tokens - Bumped n_ctx from 2048 to 8192	2026-07-26 15:38:05 +07:00
asepharyana	14032db705	ci: add workflow_dispatch to notify-parent for manual triggering	2026-07-25 18:13:35 +07:00
asepharyanaandClaude Code	2a5ab8b6d5	ci(llm-api): add notify-parent workflow Trigger repository_dispatch to parent hub repo on push to master. Co-Authored-By: Claude Code <noreply@anthropic.com>	2026-07-25 15:11:38 +07:00
asepharyanaandGitHub	7e717cc808	Merge pull request #1 from asepharyana/refactor/clean-architecture refactor(llm-api): implement clean architecture following scraper pat…	2026-07-25 15:09:04 +07:00