feat(context): tambah context::shaping (port dari shortsend)

Perilaku should_shape/shape_messages tidak berubah, hanya sumber
penghitungan token yang sekarang lewat context::tokens (tiktoken-rs)
menggantikan heuristik char/3 bawaannya sendiri.
This commit is contained in:
asepharyana
2026-07-16 07:55:48 +07:00
parent ea0b498829
commit e080d9fc6b
7 changed files with 8 additions and 4 deletions
+1
View File
@@ -28,6 +28,7 @@ pub fn count_tokens(text: &str) -> usize {
///
/// Return: 0 for a message with no `content` (e.g. an assistant message
/// that only carries `tool_calls`).
#[allow(dead_code)]
pub fn count_message_tokens(msg: &ChatMessage) -> usize {
msg.content.as_deref().map_or(0, count_tokens)
}