feat(context): tambah context::tokens dengan tiktoken-rs
Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4 di status bar) yang saling tidak konsisten dengan satu BPE tokenizer nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi tidak ada akses jaringan saat runtime.
This commit is contained in:
@@ -0,0 +1,13 @@
|
||||
//! Context management: token counting, cross-call tool-result dedup,
|
||||
//! per-result compression, budget-based shaping, and shared
|
||||
//! context-window resolution — replaces `runtime::shortsend`.
|
||||
//!
|
||||
//! No facade function here: `dedup`, `shaping`, and `tokens` are called
|
||||
//! directly from each call site (the per-turn auto-compaction loop in
|
||||
//! `actions::run_agent_turn`, and `Action::Compact`), matching this
|
||||
//! codebase's "no DI, call modules directly" convention. An orchestration
|
||||
//! layer would only serve one of the two callers generically — the
|
||||
//! auto-loop already needs per-stage control to decide when to emit
|
||||
//! `TurnEvent::Compacted`.
|
||||
|
||||
pub mod tokens;
|
||||
Reference in New Issue
Block a user