feat(context): tambah context::tokens dengan tiktoken-rs

Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4
di status bar) yang saling tidak konsisten dengan satu BPE tokenizer
nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi
tidak ada akses jaringan saat runtime.
This commit is contained in:
asepharyana
2026-07-16 07:49:43 +07:00
parent 8bb697a53f
commit c219b6ec58
5 changed files with 112 additions and 0 deletions
+13
View File
@@ -0,0 +1,13 @@
//! Context management: token counting, cross-call tool-result dedup,
//! per-result compression, budget-based shaping, and shared
//! context-window resolution — replaces `runtime::shortsend`.
//!
//! No facade function here: `dedup`, `shaping`, and `tokens` are called
//! directly from each call site (the per-turn auto-compaction loop in
//! `actions::run_agent_turn`, and `Action::Compact`), matching this
//! codebase's "no DI, call modules directly" convention. An orchestration
//! layer would only serve one of the two callers generically — the
//! auto-loop already needs per-stage control to decide when to emit
//! `TurnEvent::Compacted`.
pub mod tokens;