Asep Haryana
9a63ff1601
- Replaced manual prompt building with LlamaModel::apply_chat_template - Uses model's baked-in Jinja template (system/user/assistant/tool format) - Added <think> trigger after template for thinking mode - Clean_text strips only <|im_end|>, <|im_start|>, <think>, </think>
llm-api
OpenAI-compatible LLM inference server using llama-cpp-2 (Rust).
Model: MiniCPM-V-4.6 Q4_K_M (505 MB)
Engine: llama.cpp via llama-cpp-2 crate
Domain: ai.asepharyana.my.id
API
GET /health
{"status": "ok", "model": "minicpm-v-4.6-q4_k_m"}
GET /v1/models
OpenAI-compatible model listing.
POST /v1/chat/completions
OpenAI-compatible chat completions.
curl https://ai.asepharyana.my.id/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "minicpm-v-4.6",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 100
}'
Development
# Build
cargo build --release
# Run (with model path env var)
MODEL_PATH=/path/to/model.gguf ./target/release/llm-api
# Or use default path
./target/release/llm-api
Docker
docker compose -f docker-compose.yml up -d
Benchmark
| Framework | Model Size | tok/s | vs PyTorch |
|---|---|---|---|
| PyTorch BF16 | 2.48 GB | 0.97 | 1.0x |
| llama.cpp Q4_K_M | 505 MB | 39.1 | 40.3x 🏆 |
Infrastructure
- Traefik router:
ai.asepharyana.my.id→llm-api:8080 - Network:
app-shared-net - Docker Compose: see
llm-api.yml