MythEclipse/llm-api
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
18 Commits 2 Branches 0 Tags
67861f384bd4f64d5236a9608b33c965fbebbc7f
Commit Graph
7 Commits
Author SHA1 Message Date
Asep Haryana 67861f384b fix(stream): add anti-buffer headers for real SSE streaming 
- Add X-Accel-Buffering: no, Cache-Control: no-cache, Connection: keep-alive
- Prevents Traefik/proxy from buffering SSE events
- Each token is flushed immediately as generated
 2026-07-26 19:13:00 +07:00
Asep Haryana 5cba76d280 fix(stream): separate reasoning_content from content in SSE streaming 
- Before </think>: tokens sent as delta.reasoning_content
- After </think>: tokens sent as delta.content
- Handles </think> split across token boundaries
- Clean <|im_start|>/<|im_end|>/<think>/</think> from streamed text
- Normal models (no thinking) send all as content as before
 2026-07-26 19:04:29 +07:00
Asep Haryana 6ff31b5f62 feat: render GGUF Jinja template via minijinja crate 
- Replaced manual prompt building with GGUF's chat template rendered through minijinja (Rust Jinja2 engine)
- Simplified template: removed multi-step tool detection, reasoning extraction (not needed at template level)
- Added reasoning_content separation: clean_text() returns (reasoning, answer) tuple
- Added reasoning_content field to ResponseMessage and SseDelta for OpenAI-compatible output
- Embedded template at build time via include_str! from templates/chat_template.jinja
- Built-in support for enable_thinking, tool_definitions, tool_calls, tool_response
 2026-07-26 18:21:35 +07:00
Asep Haryana 9a63ff1601 feat: use GGUF chat template via apply_chat_template 
- Replaced manual prompt building with LlamaModel::apply_chat_template
- Uses model's baked-in Jinja template (system/user/assistant/tool format)
- Added <think> trigger after template for thinking mode
- Clean_text strips only <|im_end|>, <|im_start|>, <think>, </think>
 2026-07-26 17:10:42 +07:00
Asep Haryana 59c77108a5 fix: skip leading EOS tokens in streaming + non-streaming 
- Skip <|im_end|> generated as first token (prevents empty responses)
- Clean <think> tags as plain text in generated output
 2026-07-26 16:09:04 +07:00
Asep Haryana d8425ea3b3 feat: switch to MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-Q8_0 GGUF 
- Updated model path + model ID for new 1B thinking model
- Updated prompt builder to use MiniCPM5 native /think trigger
- Updated clean_text to strip MiniCPM5 special tokens
- Bumped n_ctx from 2048 to 8192
 2026-07-26 15:38:05 +07:00
asepharyanaandClaude Code e351d74fa4 refactor(llm-api): implement clean architecture following scraper pattern 
Split monolithic 1012-line main.rs into layered hexagonal architecture:
- Domain: entity types and LlmError enum
- Application: prompt building, sampler construction, tool call parsing
- Infrastructure: LlamaEngine wrapping llama-cpp-2 with isolated unsafe transmute
- Presentation: Axum handlers, middleware (auth), error chain, router
- Config: type-safe AppConfig with LazyLock
- Bootstrap: Application struct with build() + run()

Resolves build_sampler/build_sampler_params duplication.
Adds simple web chat UI at GET /.

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-07-25 15:07:19 +07:00