feat(context): tambah context::tokens dengan tiktoken-rs

Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4
di status bar) yang saling tidak konsisten dengan satu BPE tokenizer
nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi
tidak ada akses jaringan saat runtime.
This commit is contained in:
asepharyana
2026-07-16 07:49:43 +07:00
parent 8bb697a53f
commit c219b6ec58
5 changed files with 112 additions and 0 deletions
+1
View File
@@ -2,5 +2,6 @@
//! and the LLM streaming pipeline.
pub mod actions;
pub mod commands;
pub mod context;
pub mod shortsend;
pub mod stream;