zesdex/docs/superpowers/plans/2026-07-16-context-compaction-overhaul.md

Context & Compaction Overhaul Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace src/app/runtime/shortsend.rs with a src/app/runtime/context/ module (tokens, dedup, squash, shaping, window) that keeps conversation context lean from turn 1 — deduplicating repeated tool calls, compressing large tool outputs at capture time, and fixing three known inconsistencies (divergent token heuristics, manual/auto /compact asymmetry, triplicated context_window resolution) — plus an optional off-by-default "concise mode" system-prompt toggle.

Architecture: Five focused files under src/app/runtime/context/, each independently testable, called directly from run_agent_turn's auto-compaction loop and from Action::Compact (no orchestration facade — this codebase has no DI layer; see CLAUDE.md). squash runs once per tool result at construction time; dedup runs every turn unconditionally; shaping (the old shortsend logic, unchanged behavior) runs only when the hysteresis threshold trips.

Tech Stack: Rust 2021, tiktoken-rs (new dependency), sha2/hex/serde_json (already dependencies).

Global Constraints

• Every pub fn/pub struct/pub enum/pub trait and every non-trivial private function (≥10 lines) needs a doc comment covering What/Flow/Why/Return, per CLAUDE.md.
• Never add #[allow(...)] lint-bypass attributes. Fix the underlying issue instead.
• Tests live inline in #[cfg(test)] mod tests blocks in the same file, not under tests/.
• Commit convention: Conventional Commits in Bahasa Indonesia (see the commit-convention skill) — every commit step in this plan already follows that format; keep it consistent if you deviate.
• Verify each task with cargo build and cargo test <filter> (both pass cleanly today). Do not gate on cargo clippy -- -D warnings — that command already fails on main from pre-existing warnings in unrelated files (view/markdown.rs, main.rs, etc.); it is not this plan's job to fix those. Instead, after each task, spot-check for new warnings with cargo clippy --message-format=short 2>&1 | grep <files this task touched> and expect no output.
• serde_json::Value objects in this codebase are NOT built with the preserve_order feature (confirmed: absent from Cargo.toml/Cargo.lock), so serde_json::Map is backed by a BTreeMap and serde_json::to_string always emits object keys in sorted order. Don't add manual key-sorting for JSON canonicalization — it's already canonical.

---

Task 1: tiktoken-rs dependency + context/tokens.rs

Files:

• Modify: Cargo.toml
• Modify: src/app/runtime/mod.rs
• Create: src/app/runtime/context/mod.rs
• Create: src/app/runtime/context/tokens.rs

Interfaces:

• Produces: pub fn context::tokens::count_tokens(text: &str) -> usize, pub fn context::tokens::count_message_tokens(msg: &ChatMessage) -> usize — every later task that needs a token count uses these two functions exclusively.

• Step 1: Add the dependency

Run: cargo add tiktoken-rs@0.12 Expected: Cargo.toml gains a tiktoken-rs = "0.12" line under [dependencies], Cargo.lock updates. This crate embeds its BPE vocab files via include_str! at compile time (verified: no reqwest/http/network dependency anywhere in its own Cargo.toml) — no runtime network access, consistent with zesdex's InternetMode gating.

• Step 2: Register the new module directory

In src/app/runtime/mod.rs, add pub mod context; alongside the existing modules (leave pub mod shortsend; in place for now — it's deleted in Task 6 once every call site has moved over):

pub mod actions;
pub mod commands;
pub mod context;
pub mod shortsend;
pub mod stream;

Create src/app/runtime/context/mod.rs:

//! Context management: token counting, cross-call tool-result dedup,
//! per-result compression, budget-based shaping, and shared
//! context-window resolution — replaces `runtime::shortsend`.
//!
//! No facade function here: `dedup`, `shaping`, and `tokens` are called
//! directly from each call site (the per-turn auto-compaction loop in
//! `actions::run_agent_turn`, and `Action::Compact`), matching this
//! codebase's "no DI, call modules directly" convention. An orchestration
//! layer would only serve one of the two callers generically — the
//! auto-loop already needs per-stage control to decide when to emit
//! `TurnEvent::Compacted`.

pub mod tokens;

(dedup, shaping, squash, window are added to this list in later tasks as their files are created — adding pub mod lines for files that don't exist yet won't compile.)

• Step 3: Write the failing tests

Create src/app/runtime/context/tokens.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::dto::chat::message::ChatMessage;

    #[test]
    fn empty_string_has_zero_tokens() {
        assert_eq!(count_tokens(""), 0);
    }

    #[test]
    fn known_short_phrase_has_expected_token_count() {
        // Verified empirically against tiktoken-rs 0.12's o200k_base:
        // "hello world" -> [24912, 2375], i.e. 2 tokens.
        assert_eq!(count_tokens("hello world"), 2);
    }

    #[test]
    fn known_code_snippet_has_expected_token_count() {
        // Verified empirically: 9 tokens under o200k_base.
        assert_eq!(count_tokens("fn main() { println!(\"hi\"); }"), 9);
    }

    #[test]
    fn message_with_no_content_counts_zero() {
        let msg = ChatMessage::assistant(None);
        assert_eq!(count_message_tokens(&msg), 0);
    }

    #[test]
    fn message_token_count_matches_count_tokens_on_its_content() {
        let msg = ChatMessage::user("hello world");
        assert_eq!(count_message_tokens(&msg), count_tokens("hello world"));
    }
}

• Step 2: Run test to verify it fails

Run: cargo test --bin zesdex context::tokens::tests Expected: compile error — count_tokens/count_message_tokens not found (not yet defined above the test module).

• Step 3: Write the implementation

Add above the #[cfg(test)] block in the same file:

//! Unified token-count estimation for context-window budgeting.
//!
//! Flow: text -> `tiktoken_rs::o200k_base_singleton()` (BPE vocab embedded
//! in the binary via `include_str!`, no network access) -> `encode_ordinary`
//! -> token count.
//!
//! Why: replaces three independent char-count heuristics that disagreed
//! with each other (`/3` in the old `shortsend.rs`, `/4` in the turn
//! loop, `/4` again in the status bar) with one real BPE tokenizer.
//! `o200k_base` is an approximation for non-OpenAI providers but is far
//! closer than a flat byte-per-token guess; it's only used for the
//! 85%/95% budget thresholds, not for billing-accurate counts.

use crate::dto::chat::message::ChatMessage;

/// Count tokens in a single string under `o200k_base`.
///
/// Return: the BPE token count for `text`. `encode_ordinary` (not
/// `encode`/`encode_with_special_tokens`) is used deliberately — message
/// content that happens to contain a special-token-shaped substring
/// (e.g. literal text `<|endoftext|>` pasted by a user) must be counted
/// as ordinary text, not interpreted as a control token.
pub fn count_tokens(text: &str) -> usize {
    tiktoken_rs::o200k_base_singleton().encode_ordinary(text).len()
}

/// Count tokens in a `ChatMessage`'s text content.
///
/// Return: 0 for a message with no `content` (e.g. an assistant message
/// that only carries `tool_calls`).
pub fn count_message_tokens(msg: &ChatMessage) -> usize {
    msg.content.as_deref().map_or(0, count_tokens)
}

• Step 4: Run test to verify it passes

Run: cargo test --bin zesdex context::tokens::tests Expected: test result: ok. 5 passed; 0 failed

• Step 5: Commit

git add Cargo.toml Cargo.lock src/app/runtime/mod.rs src/app/runtime/context/mod.rs src/app/runtime/context/tokens.rs
git commit -m "feat(context): tambah context::tokens dengan tiktoken-rs

Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4
di status bar) yang saling tidak konsisten dengan satu BPE tokenizer
nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi
tidak ada akses jaringan saat runtime."

---

Task 2: context/window.rs

Files:

• Modify: src/app/runtime/context/mod.rs
• Create: src/app/runtime/context/window.rs

Interfaces:

• Consumes: crate::model::app_config::AppConfig, crate::model::settings::Settings (existing types, both already pub).

• Produces: pub fn context::window::resolve(app_config: &AppConfig, settings: &Settings) -> usize — Task 6 (spawn_turn/Action::Compact) and Task 8 (status bar) both call this.

• Step 1: Write the failing tests

Create src/app/runtime/context/window.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::model::app_config::{AppConfig, ModelRole};
    use crate::model::settings::Settings;

    #[test]
    fn resolves_context_window_from_matching_model_role() {
        let mut app_config = AppConfig::default();
        app_config.model_roles.insert("default".to_string(), ModelRole {
            provider: "zen".to_string(),
            model: "deepseek-v4-flash-free".to_string(),
            max_tokens: None,
            context_window: Some(128_000),
            temperature: None,
        });
        let mut settings = Settings::default();
        settings.provider = "zen".to_string();
        settings.model = "deepseek-v4-flash-free".to_string();

        assert_eq!(resolve(&app_config, &settings), 128_000);
    }

    #[test]
    fn falls_back_to_default_context_window_when_no_role_matches() {
        let app_config = AppConfig::default();
        let mut settings = Settings::default();
        settings.provider = "nonexistent".to_string();
        settings.model = "nonexistent-model".to_string();

        assert_eq!(resolve(&app_config, &settings), app_config.default_context_window as usize);
    }

    #[test]
    fn falls_back_to_default_when_matching_role_has_no_context_window_set() {
        let mut app_config = AppConfig::default();
        app_config.model_roles.insert("default".to_string(), ModelRole {
            provider: "zen".to_string(),
            model: "deepseek-v4-flash-free".to_string(),
            max_tokens: None,
            context_window: None,
            temperature: None,
        });
        let mut settings = Settings::default();
        settings.provider = "zen".to_string();
        settings.model = "deepseek-v4-flash-free".to_string();

        assert_eq!(resolve(&app_config, &settings), app_config.default_context_window as usize);
    }
}

• Step 2: Run test to verify it fails

Run: cargo test --bin zesdex context::window::tests Expected: compile error — resolve not found.

• Step 3: Write the implementation

Add above the test block:

//! Single source of truth for resolving the active model's context
//! window size, replacing three copies of the same lookup that had
//! drifted (`Action::Compact`, `spawn_turn`, and `view/status.rs` each
//! had their own inline version — the status bar's copy additionally
//! displayed "?" on no match instead of falling back like the other two,
//! an inconsistency this unifies away).

use crate::model::app_config::AppConfig;
use crate::model::settings::Settings;

/// Resolve the context-window size (in tokens) for the currently
/// configured provider/model.
///
/// Flow: find the `ModelRole` whose `provider`+`model` match
/// `settings` -> use its `context_window` if set -> otherwise fall back
/// to `app_config.default_context_window`.
///
/// Return: always a concrete token count, never "unknown".
pub fn resolve(app_config: &AppConfig, settings: &Settings) -> usize {
    app_config.model_roles.values()
        .find(|role| role.provider == settings.provider && role.model == settings.model)
        .and_then(|role| role.context_window)
        .unwrap_or(app_config.default_context_window) as usize
}

In src/app/runtime/context/mod.rs, add pub mod window; to the module list.

• Step 4: Run test to verify it passes

Run: cargo test --bin zesdex context::window::tests Expected: test result: ok. 3 passed; 0 failed

• Step 5: Commit

git add src/app/runtime/context/mod.rs src/app/runtime/context/window.rs
git commit -m "feat(context): tambah context::window::resolve

Satukan tiga salinan logika resolusi context_window (Action::Compact,
spawn_turn, status bar) yang sempat melenceng satu sama lain."

---

Task 3: context/dedup.rs

Files:

• Modify: src/app/subagent/division.rs
• Modify: src/app/runtime/context/mod.rs
• Create: src/app/runtime/context/dedup.rs

Interfaces:

• Consumes: crate::app::subagent::division::tool_scope::READ_TOOLS (made pub in this task), crate::dto::chat::message::{ChatMessage, Role}, crate::dto::chat::tool::ToolCall.

• Produces: pub fn context::dedup::collapse(messages: &[ChatMessage]) -> (Vec<ChatMessage>, bool) — the bool is true iff at least one message was replaced. Task 6 calls this from both the auto-loop and the manual /compact path.

• Step 1: Make READ_TOOLS public

In src/app/subagent/division.rs, change:

    const READ_TOOLS: &[&str] = &[

to:

    /// The read-only tool set — reused by `context::dedup` as the
    /// authoritative "safe to deduplicate" classification, so there's a
    /// single list of read-only tool names in the codebase instead of two.
    pub const READ_TOOLS: &[&str] = &[

Run: cargo build — expect it still succeeds (widening const to pub const is additive; existing test module in the same file uses tools_for and other constants unaffected).

• Step 2: Write the failing tests

Create src/app/runtime/context/dedup.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::dto::chat::message::ChatMessage;
    use crate::dto::chat::tool::{ToolCall, ToolFunction};
    use serde_json::json;

    fn assistant_with_call(id: &str, name: &str, args: serde_json::Value) -> ChatMessage {
        let mut m = ChatMessage::assistant(None);
        m.tool_calls = Some(vec![ToolCall {
            id: id.to_string(),
            type_: "function".to_string(),
            function: ToolFunction { name: name.to_string(), arguments: args },
        }]);
        m
    }

    #[test]
    fn older_result_of_same_read_tool_and_args_is_replaced() {
        let messages = vec![
            assistant_with_call("call-1", "read", json!({"path": "a.rs"})),
            ChatMessage::tool_result("call-1".to_string(), "first read of a.rs".to_string()),
            assistant_with_call("call-2", "read", json!({"path": "a.rs"})),
            ChatMessage::tool_result("call-2".to_string(), "second read of a.rs".to_string()),
        ];

        let (result, changed) = collapse(&messages);

        assert!(changed);
        assert_eq!(result[1].content.as_deref(), Some(DUPLICATE_PLACEHOLDER));
        assert_eq!(result[3].content.as_deref(), Some("second read of a.rs"));
    }

    #[test]
    fn different_arguments_are_not_deduplicated() {
        let messages = vec![
            assistant_with_call("call-1", "read", json!({"path": "a.rs"})),
            ChatMessage::tool_result("call-1".to_string(), "read of a.rs".to_string()),
            assistant_with_call("call-2", "read", json!({"path": "b.rs"})),
            ChatMessage::tool_result("call-2".to_string(), "read of b.rs".to_string()),
        ];

        let (result, changed) = collapse(&messages);

        assert!(!changed);
        assert_eq!(result[1].content.as_deref(), Some("read of a.rs"));
        assert_eq!(result[3].content.as_deref(), Some("read of b.rs"));
    }

    #[test]
    fn key_order_in_arguments_does_not_prevent_dedup() {
        let messages = vec![
            assistant_with_call("call-1", "grep", json!({"pattern": "foo", "path": "."})),
            ChatMessage::tool_result("call-1".to_string(), "first grep".to_string()),
            assistant_with_call("call-2", "grep", json!({"path": ".", "pattern": "foo"})),
            ChatMessage::tool_result("call-2".to_string(), "second grep".to_string()),
        ];

        let (result, changed) = collapse(&messages);

        assert!(changed);
        assert_eq!(result[1].content.as_deref(), Some(DUPLICATE_PLACEHOLDER));
    }

    #[test]
    fn mutating_tool_with_identical_args_is_never_deduplicated() {
        let messages = vec![
            assistant_with_call("call-1", "bash", json!({"command": "cargo test"})),
            ChatMessage::tool_result("call-1".to_string(), "first run: 3 failed".to_string()),
            assistant_with_call("call-2", "bash", json!({"command": "cargo test"})),
            ChatMessage::tool_result("call-2".to_string(), "second run: 0 failed".to_string()),
        ];

        let (result, changed) = collapse(&messages);

        assert!(!changed);
        assert_eq!(result[1].content.as_deref(), Some("first run: 3 failed"));
        assert_eq!(result[3].content.as_deref(), Some("second run: 0 failed"));
    }

    #[test]
    fn tool_result_with_no_matching_call_is_left_untouched() {
        let messages = vec![
            ChatMessage::tool_result("orphan-id".to_string(), "some result".to_string()),
        ];

        let (result, changed) = collapse(&messages);

        assert!(!changed);
        assert_eq!(result[0].content.as_deref(), Some("some result"));
    }
}

• Step 3: Run test to verify it fails

Run: cargo test --bin zesdex context::dedup::tests Expected: compile error — collapse/DUPLICATE_PLACEHOLDER not found.

• Step 4: Write the implementation

Add above the test block:

//! Cross-call tool-result deduplication: when a read-only tool is called
//! again with identical arguments, the earlier result is replaced with a
//! placeholder so only the latest copy occupies context.
//!
//! Flow: pair each `Role::Tool` message to its originating `ToolCall` via
//! `tool_call_id` -> key on `(function.name, sha256(canonical_json(args)))`
//! -> for read-only tools, keep only the last occurrence of each key in
//! full, placeholder the rest.
//!
//! Why: reading the same file (or re-running the same grep) twice in a
//! session otherwise keeps both full copies in context until compaction
//! eventually drops the older one wholesale, along with everything else
//! from that period. Mutating tools (`write`, `edit`, `bash`, `delete`,
//! `git_operator`, ...) are never touched, even with identical
//! arguments, because call order and repetition can be semantically
//! meaningful (e.g. retrying a flaky `bash` command until it passes).

use std::collections::HashMap;
use sha2::Digest;
use crate::app::subagent::division::tool_scope::READ_TOOLS;
use crate::dto::chat::message::{ChatMessage, Role};

const DUPLICATE_PLACEHOLDER: &str =
    "[duplicate result — superseded by a later identical call, see below]";

/// Replace superseded read-only tool results with a placeholder.
///
/// Return: a `Vec<ChatMessage>` the same length as `messages`, and
/// `true` iff at least one entry was replaced. The caller uses the
/// `bool` to decide whether the result is worth persisting/announcing,
/// without `ChatMessage` needing to implement `PartialEq`.
pub fn collapse(messages: &[ChatMessage]) -> (Vec<ChatMessage>, bool) {
    // tool_call_id -> (tool name, canonical JSON of its arguments)
    let mut call_info: HashMap<String, (String, String)> = HashMap::new();
    for m in messages {
        if let Some(calls) = &m.tool_calls {
            for call in calls {
                let canonical = serde_json::to_string(&call.function.arguments).unwrap_or_default();
                call_info.insert(call.id.clone(), (call.function.name.clone(), canonical));
            }
        }
    }

    // For each (tool, args-hash) key among read-only tools, find the
    // index of its LAST occurrence — that's the one kept in full.
    let mut last_index_for_key: HashMap<String, usize> = HashMap::new();
    for (idx, m) in messages.iter().enumerate() {
        if m.role != Role::Tool {
            continue;
        }
        let Some(id) = &m.tool_call_id else { continue };
        let Some((name, args)) = call_info.get(id) else { continue };
        if !READ_TOOLS.contains(&name.as_str()) {
            continue;
        }
        last_index_for_key.insert(dedup_key(name, args), idx);
    }

    let mut changed = false;
    let result = messages.iter().enumerate().map(|(idx, m)| {
        if m.role != Role::Tool {
            return m.clone();
        }
        let Some(id) = &m.tool_call_id else { return m.clone() };
        let Some((name, args)) = call_info.get(id) else { return m.clone() };
        if !READ_TOOLS.contains(&name.as_str()) {
            return m.clone();
        }
        let key = dedup_key(name, args);
        if last_index_for_key.get(&key) == Some(&idx) {
            return m.clone();
        }
        changed = true;
        ChatMessage::tool_result(id.clone(), DUPLICATE_PLACEHOLDER.to_string())
    }).collect();

    (result, changed)
}

/// Build the dedup key for a tool call.
///
/// Why hash the arguments: keeps the key a fixed, short size regardless
/// of argument payload size. `serde_json::to_string` is already
/// canonical here — this codebase doesn't enable serde_json's
/// `preserve_order` feature, so `Value::Object` is backed by a
/// `BTreeMap` and always serializes keys in sorted order.
fn dedup_key(tool_name: &str, canonical_args: &str) -> String {
    let hash = hex::encode(sha2::Sha256::digest(canonical_args.as_bytes()));
    format!("{tool_name}:{hash}")
}

In src/app/runtime/context/mod.rs, add pub mod dedup; to the module list.

• Step 5: Run test to verify it passes

Run: cargo test --bin zesdex context::dedup::tests Expected: test result: ok. 6 passed; 0 failed

• Step 6: Commit

git add src/app/subagent/division.rs src/app/runtime/context/mod.rs src/app/runtime/context/dedup.rs
git commit -m "feat(context): tambah context::dedup untuk hasil tool yang berulang

Panggilan tool read-only (read, grep, glob, dst) dengan argumen persis
sama menyisakan satu salinan penuh saja di context; entri lama diganti
placeholder tapi tool-call-nya sendiri tetap terlihat di riwayat. Tool
bersifat mutasi (write/edit/bash/dll) tidak pernah disentuh.

Jadikan tool_scope::READ_TOOLS pub supaya jadi satu-satunya sumber
klasifikasi read-only, dipakai ulang bukan didaftar dua kali."

---

Task 4: context/squash.rs

Files:

• Modify: src/app/runtime/context/mod.rs
• Create: src/app/runtime/context/squash.rs

Interfaces:

• Produces: pub fn context::squash::apply(tool_name: &str, output: &str) -> String — Task 7 calls this at the tool-result construction site.

• Step 1: Write the failing tests

Create src/app/runtime/context/squash.rs:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn output_under_the_floor_passes_through_unchanged() {
        let small = "short output";
        assert_eq!(apply("bash", small), small);
    }

    #[test]
    fn read_tool_output_is_never_squashed_even_when_huge_json() {
        let big_json = format!(
            "{{\"description\": \"{}\"}}",
            "a very long description value that repeats ".repeat(100),
        );
        assert!(big_json.len() > SQUASH_FLOOR_BYTES);
        assert_eq!(apply("read", &big_json), big_json);
    }

    #[test]
    fn json_output_over_floor_keeps_structure_and_short_values() {
        let value = serde_json::json!({
            "id": "abc123",
            "note": "hi",
            "description": "a very long description value that repeats ".repeat(100),
        });
        let text = serde_json::to_string(&value).unwrap();
        assert!(text.len() > SQUASH_FLOOR_BYTES);

        let result = apply("some_mcp_tool", &text);
        let parsed: serde_json::Value = serde_json::from_str(&result)
            .expect("squashed JSON must still be valid JSON");

        assert_eq!(parsed["id"], "abc123", "short values must survive");
        assert_eq!(parsed["note"], "hi", "short values must survive");
        assert_ne!(
            parsed["description"].as_str().unwrap().len(),
            value["description"].as_str().unwrap().len(),
            "long low-entropy value must be shrunk",
        );
    }

    #[test]
    fn json_array_elements_past_third_are_squashed_harder() {
        // A UUID-shaped value has no internal whitespace and clears the
        // entropy threshold, so under the *normal* per-value rule (which
        // still applies to array indices 0-2) it survives untouched.
        // Padding elsewhere in the object pushes total size over the
        // squash floor without affecting which array elements get kept.
        let identifier = "550e8400-e29b-41d4-a716-446655440000";
        let padding = "padding text to push this payload past the squash floor so apply() actually dispatches to squash_json ".repeat(20);
        let value = serde_json::json!({
            "padding": padding,
            "items": [identifier, identifier, identifier, identifier],
        });
        let text = serde_json::to_string(&value).unwrap();
        assert!(text.len() > SQUASH_FLOOR_BYTES);

        let result = apply("some_mcp_tool", &text);
        let parsed: serde_json::Value = serde_json::from_str(&result).unwrap();
        let items = parsed["items"].as_array().unwrap();

        assert_eq!(items[0].as_str().unwrap(), identifier, "index 0 is under the array cutoff and identifier-shaped, so it's kept under the normal rule");
        assert_eq!(items[2].as_str().unwrap(), identifier, "index 2 is still under the cutoff (past-third means index >= 3)");
        assert_ne!(items[3].as_str().unwrap(), identifier, "index 3 must be force-elided even though it's identifier-shaped and would survive at any earlier index");
    }

    #[test]
    fn log_like_output_keeps_error_lines_and_marks_omissions() {
        // `looks_log_shaped` requires >= 3 lines matching error/warn/fail/
        // panic/stack-frame patterns before routing to `squash_log` at
        // all — a single error line isn't enough and would silently fall
        // through to `squash_generic` instead, so this fixture needs at
        // least 3 such lines, spread apart, to actually exercise
        // squash_log's scoring/windowing logic (not just its fallback).
        let mut lines = vec!["build started".to_string()];
        for i in 0..200 {
            lines.push(format!("info: compiling module {i}"));
        }
        lines.push("error: something failed early in the build".to_string());
        for i in 0..200 {
            lines.push(format!("info: compiling module {}", i + 200));
        }
        lines.push("warning: deprecated api used somewhere".to_string());
        lines.push("error: something failed at the end".to_string());
        let text = lines.join("\n");
        assert!(text.len() > SQUASH_FLOOR_BYTES);

        let result = apply("bash", &text);

        assert!(result.contains("error: something failed early in the build"));
        assert!(result.contains("error: something failed at the end"));
        assert!(result.contains("lines omitted"));
        assert!(result.len() < text.len());
    }

    #[test]
    fn generic_large_text_is_truncated_with_omission_marker() {
        let lines: Vec<String> = (0..500).map(|i| format!("line number {i} of plain output")).collect();
        let text = lines.join("\n");
        assert!(text.len() > SQUASH_FLOOR_BYTES);

        let result = apply("bash", &text);

        assert!(result.contains("line number 0 of plain output"), "keeps head");
        assert!(result.contains("line number 499 of plain output"), "keeps tail");
        assert!(result.contains("lines omitted"));
        assert!(result.len() < text.len());
    }
}

• Step 2: Run test to verify it fails

Run: cargo test --bin zesdex context::squash::tests Expected: compile error — apply/SQUASH_FLOOR_BYTES not found.

• Step 3: Write the implementation

Add above the test block:

//! Per-tool-result compression: shrink large tool outputs before they
//! ever enter conversation history, dispatching by content shape.
//!
//! Flow: `apply(tool_name, output)` -> `read` tool or under the size
//! floor? pass through unchanged : valid JSON? `squash_json` : looks
//! log-shaped? `squash_log` : `squash_generic`.
//!
//! Why: a single large `bash`/`grep` result can dominate a
//! conversation's token budget even on its first occurrence, long
//! before `dedup`/`shaping` ever get a chance to act on repeats or
//! overall budget.

use std::collections::HashSet;

/// Below this size, compression isn't worth the risk of losing detail —
/// pass the output through unchanged.
const SQUASH_FLOOR_BYTES: usize = 1500;

/// Byte budget for the generic fallback compressor — double the squash
/// floor, so the fallback path still yields a real reduction on
/// anything that triggered it.
const GENERIC_BUDGET_BYTES: usize = SQUASH_FLOOR_BYTES * 2;

/// Tools whose output must never be altered. `read` is exempted because
/// its output must stay byte-exact — the agent relies on it for
/// exact-match edits afterward, and squashing a file that happens to
/// parse as JSON (e.g. `package.json`) would silently corrupt the
/// agent's view of real file content.
const NEVER_SQUASH: &[&str] = &["read"];

/// Compress a tool's raw output before it's stored in conversation
/// history.
///
/// Return: `output` unchanged if `tool_name` is in `NEVER_SQUASH` or at
/// or under `SQUASH_FLOOR_BYTES`; otherwise the compressed form from
/// whichever detector matches its content shape.
pub fn apply(tool_name: &str, output: &str) -> String {
    if NEVER_SQUASH.contains(&tool_name) || output.len() <= SQUASH_FLOOR_BYTES {
        return output.to_string();
    }
    if serde_json::from_str::<serde_json::Value>(output).is_ok() {
        return squash_json(output);
    }
    if looks_log_shaped(output) {
        return squash_log(output);
    }
    squash_generic(output, GENERIC_BUDGET_BYTES)
}

/// Compress a JSON tool result by keeping all structural content (keys,
/// array/object shape) and eliding long, low-entropy string *values*,
/// while keeping short values (<=20 chars) and high-entropy single-token
/// ones (UUIDs, hashes, paths) intact. Array elements past the first 3
/// are elided regardless of length/entropy.
///
/// Why walk a parsed `Value` instead of hand-rolling a JSON tokenizer:
/// `serde_json` already handles escaping/nesting correctly (this
/// codebase's own `dto::chat::tool::repair_json` exists specifically to
/// work around how easy it is to get that wrong by hand) — reusing it
/// is both simpler and more robust.
///
/// Return: re-serialized JSON with the same shape as the input.
fn squash_json(text: &str) -> String {
    let Ok(mut value) = serde_json::from_str::<serde_json::Value>(text) else {
        return text.to_string();
    };
    squash_json_value(&mut value, false);
    serde_json::to_string(&value).unwrap_or_else(|_| text.to_string())
}

/// Recursively elide long, low-entropy string values in place.
/// `in_late_array` is true once past the first 3 elements of an
/// enclosing array, tightening the elision rule for the rest of it.
///
/// Why the `!s.contains(' ')` gate before the entropy check: raw
/// per-character Shannon entropy alone does NOT separate "meaningful
/// prose" from "random-looking identifier" — verified empirically,
/// repeated English prose scores ~3.89 bits/char, *higher* than a UUID's
/// ~3.39 or a SHA-256 hex digest's ~3.66, because prose draws from a
/// wide, fairly-balanced character set too. What actually distinguishes
/// identifiers from prose is that identifiers are a single unbroken
/// token — this mirrors headroom's own approach (its entropy check is
/// "cheaply pre-filtered by 'no spaces'" before scoring). Multi-word
/// values never reach the entropy branch at all; only whitespace-free
/// tokens do, where entropy correctly separates "abc123" or "aaaaaaaa"
/// (low, elided if long) from a UUID/hash/API-key-shaped string (high,
/// kept).
fn squash_json_value(value: &mut serde_json::Value, in_late_array: bool) {
    match value {
        serde_json::Value::String(s) => {
            let looks_like_identifier = !s.contains(' ') && shannon_entropy(s) >= 3.0;
            let keep = !in_late_array && (s.len() <= 20 || looks_like_identifier);
            if !keep {
                *s = "…".to_string();
            }
        }
        serde_json::Value::Array(items) => {
            for (i, item) in items.iter_mut().enumerate() {
                squash_json_value(item, i >= 3);
            }
        }
        serde_json::Value::Object(map) => {
            for v in map.values_mut() {
                squash_json_value(v, false);
            }
        }
        _ => {}
    }
}

/// Shannon entropy in bits per character — used, after the `squash_json`
/// caller's own "no internal whitespace" pre-filter, to distinguish
/// high-entropy single-token strings (UUIDs, hashes, random IDs, worth
/// keeping) from low-entropy ones (e.g. `"aaaaaaaaaa"`, safe to elide).
/// 3.0 sits comfortably below a UUID's ~3.39 and a SHA-256 hex digest's
/// ~3.66 (both empirically measured with this exact formula) while
/// staying well above a degenerate repeated-character string's 0.0.
fn shannon_entropy(s: &str) -> f64 {
    if s.is_empty() {
        return 0.0;
    }
    let mut counts: std::collections::HashMap<char, usize> = std::collections::HashMap::new();
    for c in s.chars() {
        *counts.entry(c).or_insert(0) += 1;
    }
    let len = s.chars().count() as f64;
    counts.values()
        .map(|&count| {
            let p = f64::from(u32::try_from(count).unwrap_or(u32::MAX)) / len;
            -p * p.log2()
        })
        .sum()
}

#[derive(Clone, Copy, PartialEq, Eq)]
enum LogLevel {
    Error,
    Warn,
    Info,
    Debug,
}

fn classify_line(line: &str) -> LogLevel {
    let lower = line.to_lowercase();
    if lower.contains("error") || lower.contains("fail") || lower.contains("panic") {
        LogLevel::Error
    } else if lower.contains("warn") {
        LogLevel::Warn
    } else if lower.contains("debug") || lower.contains("trace") {
        LogLevel::Debug
    } else {
        LogLevel::Info
    }
}

/// Heuristic gate for routing to `squash_log` vs `squash_generic`: at
/// least 3 lines that look like error/warning/stack-trace output.
fn looks_log_shaped(text: &str) -> bool {
    let hits = text.lines().filter(|l| {
        let lower = l.to_lowercase();
        lower.contains("error") || lower.contains("warn") || lower.contains("fail")
            || lower.contains("panic") || l.trim_start().starts_with("at ")
    }).count();
    hits >= 3
}

/// Compress log-shaped output: keep up to 20 highest-scored error lines
/// and up to 10 highest-scored warning lines (score = level weight +
/// 0.3 if the line looks like a stack-trace frame), each with a
/// +/-2-line context window, replacing every gap with a `[N lines
/// omitted]` marker.
///
/// Why not a comment-shaped marker (e.g. `// N lines omitted`): the
/// `rtk` project's own regression tests found that shape gets parsed by
/// the LLM as code and triggers a retry loop.
fn squash_log(text: &str) -> String {
    let lines: Vec<&str> = text.lines().collect();
    let levels: Vec<LogLevel> = lines.iter().map(|l| classify_line(l)).collect();

    let score = |i: usize| -> f32 {
        let level_score = match levels[i] {
            LogLevel::Error => 1.0,
            LogLevel::Warn => 0.5,
            LogLevel::Info => 0.1,
            LogLevel::Debug => 0.05,
        };
        let stack_boost = if lines[i].trim_start().starts_with("at ") { 0.3 } else { 0.0 };
        level_score + stack_boost
    };

    let mut error_idxs: Vec<usize> = (0..lines.len()).filter(|&i| levels[i] == LogLevel::Error).collect();
    error_idxs.sort_by(|&a, &b| score(b).partial_cmp(&score(a)).unwrap_or(std::cmp::Ordering::Equal));
    error_idxs.truncate(20);

    let mut warn_idxs: Vec<usize> = (0..lines.len()).filter(|&i| levels[i] == LogLevel::Warn).collect();
    warn_idxs.sort_by(|&a, &b| score(b).partial_cmp(&score(a)).unwrap_or(std::cmp::Ordering::Equal));
    warn_idxs.truncate(10);

    let mut keep: HashSet<usize> = HashSet::new();
    for &i in error_idxs.iter().chain(warn_idxs.iter()) {
        let lo = i.saturating_sub(2);
        let hi = (i + 2).min(lines.len().saturating_sub(1));
        keep.extend(lo..=hi);
    }

    if keep.is_empty() {
        return squash_generic(text, GENERIC_BUDGET_BYTES);
    }

    render_kept_lines(&lines, &keep)
}

/// Importance-ranked truncation for content that isn't JSON or
/// log-shaped: keep the first 10 and last 10 lines, plus any
/// non-blank line that isn't a repeat of the one before it, until
/// `budget` bytes are used.
fn squash_generic(text: &str, budget: usize) -> String {
    let lines: Vec<&str> = text.lines().collect();
    if lines.len() <= 20 {
        return text.chars().take(budget).collect();
    }

    let head_end = 10;
    let tail_start = lines.len() - 10;
    let mut keep: HashSet<usize> = (0..head_end).chain(tail_start..lines.len()).collect();

    let mut used: usize = lines[..head_end].iter().map(|l| l.len() + 1).sum::<usize>()
        + lines[tail_start..].iter().map(|l| l.len() + 1).sum::<usize>();
    let mut prev = "";
    for (i, &line) in lines.iter().enumerate().take(tail_start).skip(head_end) {
        let non_trivial = !line.trim().is_empty() && line != prev;
        if non_trivial && used + line.len() + 1 <= budget {
            keep.insert(i);
            used += line.len() + 1;
        }
        prev = line;
    }

    render_kept_lines(&lines, &keep)
}

/// Render a subset of `lines` in order, inserting a `[N lines omitted]`
/// marker at every gap between kept lines.
fn render_kept_lines(lines: &[&str], keep: &HashSet<usize>) -> String {
    let mut kept_sorted: Vec<usize> = keep.iter().copied().collect();
    kept_sorted.sort_unstable();

    let mut out = String::new();
    let mut cursor = 0usize;
    for &i in &kept_sorted {
        if i > cursor {
            out.push_str(&format!("[{} lines omitted]\n", i - cursor));
        }
        out.push_str(lines[i]);
        out.push('\n');
        cursor = i + 1;
    }
    if cursor < lines.len() {
        out.push_str(&format!("[{} lines omitted]\n", lines.len() - cursor));
    }
    out
}

In src/app/runtime/context/mod.rs, add pub mod squash; to the module list.

• Step 4: Run test to verify it passes

Run: cargo test --bin zesdex context::squash::tests Expected: test result: ok. 6 passed; 0 failed

• Step 5: Commit

git add src/app/runtime/context/mod.rs src/app/runtime/context/squash.rs
git commit -m "feat(context): tambah context::squash untuk kompresi hasil tool

Kompresi per-jenis-konten (JSON: pertahankan struktur & value pendek/
entropi tinggi, buang value panjang bertele-tele; log: simpan baris
error/warning berskor tertinggi + konteks sekitarnya; generic: potong
importance-ranked) untuk hasil tool di atas 1.5KB. Tool read dikecualikan
total karena isinya harus tetap byte-exact untuk edit selanjutnya."

---

Task 5: context/shaping.rs (port of shortsend.rs)

Files:

• Modify: src/app/runtime/context/mod.rs
• Create: src/app/runtime/context/shaping.rs

Interfaces:

• Consumes: context::tokens::count_tokens (Task 1).

• Produces: pub fn context::shaping::should_shape(token_count: usize, max_wire_tokens: usize, prev_shaped: bool) -> bool, pub fn context::shaping::shape_messages(messages: &[ChatMessage], token_count: usize, max_wire_tokens: usize, force: bool, client: Option<&LlmClient>) -> Vec<ChatMessage> — same names and signatures as today's shortsend::should_shape/shortsend::shape_messages, so Task 6 is a pure find-and-replace of the module path.

• Step 1: Write the failing tests

Create src/app/runtime/context/shaping.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::dto::chat::message::ChatMessage;

    #[test]
    fn should_shape_triggers_at_85_percent_when_not_previously_shaped() {
        assert!(should_shape(850, 1000, false));
        assert!(!should_shape(849, 1000, false));
    }

    #[test]
    fn should_shape_uses_95_percent_threshold_once_already_shaped() {
        assert!(!should_shape(900, 1000, true), "below 95% and already shaped: no re-trigger yet");
        assert!(should_shape(950, 1000, true));
    }

    #[test]
    fn shape_messages_is_a_noop_under_budget_and_not_forced() {
        let messages = vec![
            ChatMessage::system("sys"),
            ChatMessage::user("hi"),
            ChatMessage::assistant(Some("hello".to_string())),
        ];
        let result = shape_messages(&messages, 10, 1000, false, None);
        assert_eq!(result.len(), messages.len());
    }

    /// Build a message whose real BPE token count is large enough that 20
    /// of them (~49 tokens each, ~980 total — verified empirically with
    /// `context::tokens::count_tokens`) comfortably exceed
    /// `shape_messages`'s 70%-of-1000 = 700 token target, guaranteeing
    /// several get dropped. A short fixture like `format!("message {i}")`
    /// (~8 tokens each, ~160 total for 20) stays entirely under budget
    /// with real BPE counting and would make these tests pass vacuously
    /// (nothing ever gets dropped, so "must survive shaping" and "falls
    /// back to placeholder" hold trivially without exercising the actual
    /// drop logic) — this was a real bug caught during Task 5's first
    /// implementation attempt.
    fn padded_message(i: usize) -> String {
        format!(
            "message number {i} with some padding text {}",
            "additional padding content to increase token count substantially ".repeat(5),
        )
    }

    #[test]
    fn shape_messages_always_preserves_the_first_system_message() {
        let mut messages = vec![ChatMessage::system("system prompt")];
        for i in 0..20 {
            messages.push(ChatMessage::user(padded_message(i)));
        }
        let result = shape_messages(&messages, 100_000, 1000, true, None);
        assert_eq!(result[0].content.as_deref(), Some("system prompt"));
    }

    #[test]
    fn shape_messages_without_a_client_falls_back_to_placeholder_summary() {
        let mut messages = vec![ChatMessage::system("system prompt")];
        for i in 0..20 {
            messages.push(ChatMessage::user(padded_message(i)));
        }
        let result = shape_messages(&messages, 100_000, 1000, true, None);
        let has_placeholder = result.iter().any(|m| {
            m.content.as_deref() == Some("[prior conversation compacted]")
        });
        assert!(has_placeholder);
    }

    #[test]
    fn shape_messages_keeps_most_recent_messages_over_older_ones() {
        let mut messages = vec![ChatMessage::system("system prompt")];
        for i in 0..20 {
            messages.push(ChatMessage::user(padded_message(i)));
        }
        let result = shape_messages(&messages, 100_000, 1000, true, None);
        let last_content = messages.last().unwrap().content.clone();
        assert!(result.iter().any(|m| m.content == last_content), "most recent message must survive shaping");
    }
}

• Step 2: Run test to verify it fails

Run: cargo test --bin zesdex context::shaping::tests Expected: compile error — should_shape/shape_messages not found.

• Step 3: Write the implementation

Add above the test block (ported from src/app/runtime/shortsend.rs, unchanged behavior, only the token-estimation call site changed to use tokens::count_tokens):

//! Budget-based message shaping: compacts long conversation histories so
//! they fit within the provider's context window before being sent to
//! the LLM API. Ported from the former `runtime::shortsend` — behavior
//! is unchanged, only its token-counting now goes through
//! `context::tokens` instead of an inline heuristic.

use super::tokens::count_tokens;
use crate::dto::chat::message::ChatMessage;

/// Decide whether the message list should be shaped (compacted) before
/// sending to the LLM.
///
/// Flow: trigger based on token estimate. If `token_estimate` exceeds
/// the threshold, we shape. When `prev_shaped` is true, the threshold is
/// raised (95%) to avoid fluttering — compaction only re-triggers when
/// the context is genuinely full again. When `prev_shaped` is false, the
/// threshold is lower (85%) so compaction starts proactively.
///
/// Why: hysteresis prevents repeated compaction on every turn when the
/// token count hovers near the boundary.
///
/// Return: `true` if shaping should be applied.
pub fn should_shape(token_estimate: usize, max_wire_tokens: usize, prev_shaped: bool) -> bool {
    let threshold = if prev_shaped {
        (max_wire_tokens as f32 * 0.95) as usize
    } else {
        (max_wire_tokens as f32 * 0.85) as usize
    };
    token_estimate >= threshold
}

/// Compact a long message list by dropping middle messages and inserting
/// a summary placeholder.
///
/// Flow: if the estimated token count is within budget and not forced,
/// return messages unchanged -> otherwise keep the system message and
/// the most recent messages that fit a 70%-of-budget target, with a
/// `[prior conversation compacted]` (or LLM-generated summary, if
/// `client` is `Some`) system message in between.
///
/// Why: keeps context-size overhead roughly constant regardless of
/// session length.
///
/// Return: the shaped message list, or `messages` unchanged if shaping
/// wasn't needed.
pub fn shape_messages(
    messages: &[ChatMessage],
    token_count: usize,
    max_wire_tokens: usize,
    force: bool,
    client: Option<&crate::service::provider::LlmClient>,
) -> Vec<ChatMessage> {
    if !force && (token_count <= max_wire_tokens || messages.len() < 5) {
        return messages.to_vec();
    }

    let target_tokens = (max_wire_tokens as f32 * 0.70) as usize;
    let mut current_tokens = 0;
    let mut keep_recent = Vec::new();
    let mut dropped_msgs = Vec::new();

    let mut msgs_to_eval = messages.to_vec();
    let first = if msgs_to_eval.is_empty() {
        None
    } else {
        Some(msgs_to_eval.remove(0))
    };

    for m in msgs_to_eval.into_iter().rev() {
        let text = m.content.as_deref().unwrap_or("");
        let msg_tokens = count_tokens(text);

        if current_tokens + msg_tokens <= target_tokens {
            current_tokens += msg_tokens;
            keep_recent.push(m);
        } else {
            dropped_msgs.push(m);
        }
    }

    dropped_msgs.reverse();

    let mut result = Vec::new();
    if let Some(f) = first {
        result.push(f);
    }

    if !dropped_msgs.is_empty() {
        let mut summary_text = "[prior conversation compacted]".to_string();

        if let Some(llm) = client {
            let prompt = format!(
                "Summarize the following dropped conversation history briefly. Focus on main goals, decisions made, and files modified, so the context is preserved for future turns. Keep it concise.\n\nHistory:\n{}",
                dropped_msgs.iter()
                    .map(|m| format!("[{}]: {}", if m.role == crate::dto::chat::message::Role::User { "User" } else { "Assistant" }, m.content.as_deref().unwrap_or("")))
                    .collect::<Vec<_>>()
                    .join("\n\n")
            );

            let req_msgs = vec![ChatMessage::user(prompt)];
            match llm.chat_with_tools_non_streaming(&req_msgs, None) {
                Ok(resp) => {
                    if let Some(content) = resp.0.content {
                        summary_text = format!("[Summary of compacted prior conversation:\n{content}\n]");
                    }
                }
                Err(e) => {
                    tracing::warn!(
                        "[context::shaping] LLM summarization failed: {}. \
                         Prior conversation history is lost — no summary available. \
                         This means the model will lose context about earlier parts of \
                         the conversation.",
                        e,
                    );
                }
            }
        }

        result.push(ChatMessage::system(summary_text));
    }

    result.extend(keep_recent.into_iter().rev());
    result
}

In src/app/runtime/context/mod.rs, add pub mod shaping; to the module list.

• Step 4: Run test to verify it passes

Run: cargo test --bin zesdex context::shaping::tests Expected: test result: ok. 6 passed; 0 failed

• Step 5: Commit

git add src/app/runtime/context/mod.rs src/app/runtime/context/shaping.rs
git commit -m "feat(context): tambah context::shaping (port dari shortsend)

Perilaku should_shape/shape_messages tidak berubah, hanya sumber
penghitungan token yang sekarang lewat context::tokens (tiktoken-rs)
menggantikan heuristik char/3 bawaannya sendiri."

---

Task 6: Cut over to context:: everywhere; delete shortsend.rs; fix manual /compact

This task is kept as one unit rather than split, because deleting shortsend.rs and cutting Action::Compact over to context:: are inseparable — the moment the file is deleted, Action::Compact's existing reference to shortsend::shape_messages stops compiling, so a split here would leave an intermediate task that can't build (violating "each task ends with an independently testable deliverable").

Files:

• Modify: src/app/runtime/actions/mod.rs (auto-loop at lines 1146-1174, spawn_turn at lines 616-724, Action::Compact at lines 547-563, new resolve_llm_client_config helper)
• Modify: src/app/runtime/mod.rs
• Delete: src/app/runtime/shortsend.rs

Interfaces:

• Consumes: context::dedup::collapse (Task 3), context::tokens::count_message_tokens (Task 1), context::shaping::{should_shape, shape_messages} (Task 5), context::window::resolve (Task 2).
• Produces: fn resolve_llm_client_config(state: &AppStateRest) -> Result<(String, String, Option<String>), String> (private, reused by both spawn_turn and Action::Compact) — no other task depends on it.

This task has no new unit tests of its own — Tasks 1, 2, 3, and 5 already cover the underlying logic. Verification here is that run_agent_turn's existing turn-loop behavior is preserved end-to-end, and that manual /compact now gets real LLM summarization like the automatic path already does — checked by building and running the full test suite.

• Step 1: Replace the auto-loop's compaction block

In src/app/runtime/actions/mod.rs, replace lines 1146-1174:

    loop {
        let total_chars: usize = msgs.iter()
            .filter_map(|m| m.content.as_deref())
            .map(str::len)
            .sum();
        let token_estimate = total_chars / 4;
        let max_wire_tokens = tc.context_window;

        // Skip message compaction if abort was requested — the non-streaming
        // LLM call for summarization would block without checking abort_flag.
        let wire_msgs = if !tc.abort_flag.load(std::sync::atomic::Ordering::SeqCst)
            && crate::app::runtime::shortsend::should_shape(token_estimate, max_wire_tokens, prev_shaped)
        {
            prev_shaped = true;
            let compacted = crate::app::runtime::shortsend::shape_messages(&msgs, token_estimate, max_wire_tokens, false, Some(&tc.client));
            
            // Dispatch the compacted messages to the main thread so the local session history
            // is permanently compacted and doesn't trigger shaping again immediately on next turn.
            if let Ok(mut q) = events_q.lock() {
                q.push_back(TurnEvent::Compacted(compacted.clone()));
            }
            
            // Also update our local `msgs` variable so the rest of the loop operates on the compacted version
            msgs.clone_from(&compacted);
            compacted
        } else {
            prev_shaped = false;
            msgs.clone()
        };

with:

    loop {
        // Dedup runs every iteration, unconditionally — repeated
        // read-only tool calls (same tool + same arguments) are
        // collapsed to their latest result before anything else, so
        // context stays minimal from turn 1 instead of only shrinking
        // once shaping's budget threshold trips.
        let (deduped, dedup_changed) = crate::app::runtime::context::dedup::collapse(&msgs);
        let token_count: usize = deduped.iter()
            .map(crate::app::runtime::context::tokens::count_message_tokens)
            .sum();
        let max_wire_tokens = tc.context_window;

        // Skip shaping if abort was requested — the non-streaming LLM
        // call for summarization would block without checking abort_flag.
        let wire_msgs = if !tc.abort_flag.load(std::sync::atomic::Ordering::SeqCst)
            && crate::app::runtime::context::shaping::should_shape(token_count, max_wire_tokens, prev_shaped)
        {
            prev_shaped = true;
            let compacted = crate::app::runtime::context::shaping::shape_messages(&deduped, token_count, max_wire_tokens, false, Some(&tc.client));

            // Dispatch to the main thread so the local session history is
            // permanently updated and doesn't re-trigger shaping immediately
            // on the next turn.
            if let Ok(mut q) = events_q.lock() {
                q.push_back(TurnEvent::Compacted(compacted.clone()));
            }

            msgs.clone_from(&compacted);
            compacted
        } else {
            prev_shaped = false;
            if dedup_changed {
                if let Ok(mut q) = events_q.lock() {
                    q.push_back(TurnEvent::Compacted(deduped.clone()));
                }
                msgs.clone_from(&deduped);
            }
            deduped
        };

• Step 2: Switch the module registration, delete the old file (don't build yet)

In src/app/runtime/mod.rs, remove the line pub mod shortsend;.

Run: rm src/app/runtime/shortsend.rs

The build is expected to fail after this step (Action::Compact still references shortsend::shape_messages) — that's fine, continue straight to Step 3 rather than running cargo build here.

• Step 3: Extract resolve_llm_client_config

In src/app/runtime/actions/mod.rs, add this new private function just above fn spawn_turn (around line 615):

/// Resolve the API key, model name, and base URL for the currently
/// configured provider.
///
/// Flow: look up the provider's `ProviderConfig` for its `api_base` ->
/// resolve the API key from `Settings.api_keys`, falling back to the
/// provider's `api_key_env` environment variable, then its
/// `default_api_key`, then the crate-wide empty-string default.
///
/// Why: this exact resolution was duplicated between `spawn_turn` and
/// needed again for `Action::Compact`'s background-thread LLM call —
/// factored out so both stay in sync.
///
/// Return: `Ok((api_key, model, base_url))`, or `Err(message)` — a
/// user-facing string — if the configured provider has no entry in
/// `AppConfig.providers` at all.
fn resolve_llm_client_config(state: &AppStateRest) -> Result<(String, String, Option<String>), String> {
    let base_url = state.app_config.providers.get(&state.settings.provider).map(|p| p.api_base.clone());
    let Some(base_url) = base_url else {
        return Err(format!(
            "Provider '{}' is not configured — no matching entry found. \
             Pick a different provider in Settings, or configure it.",
            state.settings.provider
        ));
    };
    let mut api_key = state.settings.api_keys.get(&state.settings.provider).cloned().unwrap_or_default();
    if api_key.is_empty() {
        if let Some(provider_cfg) = state.app_config.providers.get(&state.settings.provider) {
            api_key = provider_cfg.api_key_env.as_ref()
                .and_then(|env| std::env::var(env).ok())
                .or_else(|| provider_cfg.default_api_key.clone())
                .unwrap_or_default();
        }
    }
    if api_key.is_empty() {
        api_key = crate::service::provider::DEFAULT_API_KEY.to_string();
    }
    Ok((api_key, state.settings.model.clone(), Some(base_url)))
}

• Step 4: Simplify spawn_turn to use it

In spawn_turn, replace lines 633-668 (the inline api_key/model/base_url resolution and its base_url.is_none()/api_key.is_empty() checks):

    let mut api_key = state.settings.api_keys.get(&state.settings.provider).cloned().unwrap_or_default();
    let model = state.settings.model.clone();
    let base_url = state.app_config.providers.get(&state.settings.provider)
        .map(|p| p.api_base.clone());
    let context_window = state.app_config.model_roles.values()
        .find(|role| role.provider == state.settings.provider && role.model == state.settings.model)
        .and_then(|role| role.context_window)
        .unwrap_or(state.app_config.default_context_window) as usize;
    // The selected provider has no entry in app_config at all (e.g. the
    // Claude-settings auto-detection that registers "claude" found nothing
    // this run). Without this check, LlmClient::new silently falls back to
    // the zen default base URL while keeping this provider's model name —
    // a mismatched request that reaches a real server and comes back as a
    // confusing "Missing API key" 401 from an unrelated provider, instead
    // of the actual problem: the configured provider doesn't exist.
    if base_url.is_none() {
        if let Ok(mut q) = state.turn_events.lock() {
            q.push_back(TurnEvent::Error(format!(
                "Provider '{}' is not configured — no matching entry found. \
                 Pick a different provider in Settings, or configure it.",
                state.settings.provider
            )));
        }
        return;
    }
    if api_key.is_empty() {
        if let Some(provider_cfg) = state.app_config.providers.get(&state.settings.provider) {
            api_key = provider_cfg.api_key_env.as_ref()
                .and_then(|env| std::env::var(env).ok())
                .or_else(|| provider_cfg.default_api_key.clone())
                .unwrap_or_default();
        }
    }
    if api_key.is_empty() {
        api_key = crate::service::provider::DEFAULT_API_KEY.to_string();
    }

with:

    let (api_key, model, base_url) = match resolve_llm_client_config(state) {
        Ok(v) => v,
        Err(msg) => {
            if let Ok(mut q) = state.turn_events.lock() {
                q.push_back(TurnEvent::Error(msg));
            }
            return;
        }
    };
    let context_window = crate::app::runtime::context::window::resolve(&state.app_config, &state.settings);

(base_url is now Option<String> already — LlmClient::new(api_key, model.clone(), base_url) further down at what was line 699 stays exactly as written; no change needed there since the type didn't change.)

• Step 5: Rewrite Action::Compact

Replace the entire Action::Compact arm (lines 547-563):

        Action::Compact => {
            let max_wire_tokens = state.app_config.model_roles.values()
                .find(|role| role.provider == state.settings.provider && role.model == state.settings.model)
                .and_then(|role| role.context_window)
                .unwrap_or(state.app_config.default_context_window) as usize;
                
            if let Some(ref mut rt) = state.session_runtime {
                let total_chars: usize = rt.messages.iter()
                    .filter_map(|m| m.content.as_deref())
                    .map(str::len)
                    .sum();
                let token_estimate = total_chars / 3;
                rt.messages = crate::app::runtime::shortsend::shape_messages(&rt.messages, token_estimate, max_wire_tokens, true, None);
                state.push_toast(Toast::new(ToastKind::Success, "Conversation history compacted.".to_string()));
                state.dirty = true;
            }
        }

with:

        Action::Compact => {
            let Some(messages) = state.session_runtime.as_ref().map(|rt| rt.messages.clone()) else {
                return;
            };
            if messages.is_empty() {
                return;
            }
            let (api_key, model, base_url) = match resolve_llm_client_config(state) {
                Ok(v) => v,
                Err(msg) => {
                    state.push_toast(Toast::new(ToastKind::Error, msg));
                    return;
                }
            };
            let max_wire_tokens = crate::app::runtime::context::window::resolve(&state.app_config, &state.settings);
            let turn_events = state.turn_events.clone();

            state.push_toast(Toast::new(ToastKind::Info, "Compacting conversation history...".to_string()));

            // Manual /compact previously ran synchronously and always
            // passed `client: None` to shape_messages, so it never got
            // LLM summarization — only automatic mid-turn compaction did.
            // Running this on a background thread (same pattern as
            // spawn_turn) fixes that asymmetry: both paths now summarize
            // dropped history with the LLM instead of one silently
            // falling back to a bare placeholder.
            std::thread::spawn(move || {
                let client = crate::service::provider::LlmClient::new(api_key, model, base_url);
                let (deduped, _) = crate::app::runtime::context::dedup::collapse(&messages);
                let token_count: usize = deduped.iter()
                    .map(crate::app::runtime::context::tokens::count_message_tokens)
                    .sum();
                let compacted = crate::app::runtime::context::shaping::shape_messages(
                    &deduped, token_count, max_wire_tokens, true, Some(&client),
                );
                if let Ok(mut q) = turn_events.lock() {
                    q.push_back(TurnEvent::Compacted(compacted));
                }
            });
        }

The existing TurnEvent::Compacted handler (lines 506-512) already sets rt.messages, pushes an info toast, and marks state.dirty = true — no change needed there; it now serves both the automatic and manual paths identically, which is the point of this fix.

• Step 6: Build and run the full test suite

Run: cargo build Expected: builds clean — no remaining references to crate::app::runtime::shortsend anywhere (grep -rn "runtime::shortsend" src/ should now be empty).

Run: cargo test --bin zesdex Expected: all tests pass, including the new context::* tests from Tasks 1-5.

• Step 7: Commit

git add src/app/runtime/actions/mod.rs src/app/runtime/mod.rs
git rm src/app/runtime/shortsend.rs
git commit -m "refactor(runtime): pindah ke context::, perbaiki asimetri /compact manual

Loop auto-compaction sekarang selalu menjalankan dedup tiap iterasi
lalu shaping lewat context::, menggantikan shortsend:: yang dihapus.

Action::Compact tadinya berjalan sinkron dan selalu client: None,
sehingga hasil compact manual tidak pernah diringkas LLM (beda dengan
compaction otomatis di tengah turn). Sekarang /compact jalan di
thread background seperti spawn_turn, sehingga bisa memanggil LLM
untuk meringkas riwayat yang dibuang — perilaku manual dan otomatis
jadi setara.

Ekstrak resolve_llm_client_config() dari spawn_turn supaya logika
resolusi api_key/model/base_url tidak dua kali."

---

Task 7: Wire squash into the tool-result construction site

Files:

• Modify: src/app/runtime/actions/mod.rs:1420

Interfaces:

• Consumes: context::squash::apply (Task 4).

• Step 1: Apply squash before wrapping the tool result

In src/app/runtime/actions/mod.rs, the tool-result loop (around line 1406-1422) sends the raw output to the UI via TurnEvent::ToolResult (line 1410-1416, for the user's own transcript view — this must stay unmodified so the user always sees the tool's real output), then separately wraps output into the ChatMessage that becomes part of LLM-bound history (line 1420) — that second copy is the one to compress. Replace:

                let tool_msg = ChatMessage::tool_result(tool_call.id.clone(), output);
                archive_message(tc.db.as_ref(), &tc.session_id, &tool_msg);
                msgs.push(tool_msg);

with:

                let squashed_output = crate::app::runtime::context::squash::apply(&tool_name, &output);
                let tool_msg = ChatMessage::tool_result(tool_call.id.clone(), squashed_output);
                archive_message(tc.db.as_ref(), &tc.session_id, &tool_msg);
                msgs.push(tool_msg);

(output was already cloned into the TurnEvent::ToolResult push a few lines above at output: output.clone(), line 1413, so the original full-size string is still available here for squash::apply to read before this final output value is consumed.)

• Step 2: Build and confirm behavior

Run: cargo build Expected: builds clean.

Run: cargo test --bin zesdex Expected: all tests still pass — this task has no new tests of its own (squash::apply's logic is already covered in Task 4); this step only confirms the wiring compiles and nothing else broke.

• Step 3: Commit

git add src/app/runtime/actions/mod.rs
git commit -m "feat(runtime): kompres hasil tool lewat squash sebelum masuk context

Salinan yang dikirim ke UI (TurnEvent::ToolResult) tetap utuh; hanya
salinan yang masuk riwayat percakapan (dikirim ke LLM) yang dikompres,
supaya user tetap melihat output tool apa adanya."

---

Task 8: Switch view/status.rs to tokens/window

Files:

• Modify: src/view/status.rs

Interfaces:

• Consumes: context::tokens::count_tokens (Task 1), context::window::resolve (Task 2).

• Step 1: Replace both token/window calculations

In src/view/status.rs, replace the if let Some(ref rt) = state.session_runtime branch (lines 59-79):

    let right_str = if let Some(ref rt) = state.session_runtime {
        let max_tokens = state.app_config.model_roles.values()
            .find(|role| role.provider == state.settings.provider && role.model == state.settings.model)
            .and_then(|role| role.context_window);

        let total_chars: usize = rt.messages.iter()
            .filter_map(|m| m.content.as_deref())
            .map(str::len)
            .sum();
        let current_tokens = total_chars / 4;

        let mut parts = Vec::new();
        if rt.usage.last_tokens_in > 0 || rt.usage.last_tokens_out > 0 {
            parts.push(format!("↑{} ↓{}", rt.usage.last_tokens_in, rt.usage.last_tokens_out));
        }
        let max_str = max_tokens.map_or_else(|| "?".to_string(), |v| v.to_string());
        parts.push(format!("{current_tokens}/{max_str}"));
        parts.push(state.settings.provider.clone());
        parts.push(state.settings.model.clone());

        format!(" {} ", parts.join(" · "))
    } else {
        let max_tokens = state.app_config.model_roles.values()
            .find(|role| role.provider == state.settings.provider && role.model == state.settings.model)
            .and_then(|role| role.context_window);
        let max_str = max_tokens.map_or_else(|| "?".to_string(), |v| v.to_string());
        format!(" 0/{} · {} · {} ", max_str, state.settings.provider, state.settings.model)
    };

with:

    let max_tokens = crate::app::runtime::context::window::resolve(&state.app_config, &state.settings);

    let right_str = if let Some(ref rt) = state.session_runtime {
        let current_tokens: usize = rt.messages.iter()
            .filter_map(|m| m.content.as_deref())
            .map(crate::app::runtime::context::tokens::count_tokens)
            .sum();

        let mut parts = Vec::new();
        if rt.usage.last_tokens_in > 0 || rt.usage.last_tokens_out > 0 {
            parts.push(format!("↑{} ↓{}", rt.usage.last_tokens_in, rt.usage.last_tokens_out));
        }
        parts.push(format!("{current_tokens}/{max_tokens}"));
        parts.push(state.settings.provider.clone());
        parts.push(state.settings.model.clone());

        format!(" {} ", parts.join(" · "))
    } else {
        format!(" 0/{} · {} · {} ", max_tokens, state.settings.provider, state.settings.model)
    };

Note the behavior change: the status bar now always shows a concrete number (falling back to default_context_window like compaction itself does) instead of sometimes showing ? for a model with no explicit context_window override — this makes the display consistent with what should_shape/shape_messages actually use, closing the fourth inconsistency found during design (the status bar's own copy of this lookup didn't fall back the same way the other two copies did).

• Step 2: Build and verify

Run: cargo build Expected: builds clean.

Run: cargo test --bin zesdex Expected: all tests pass.

• Step 3: Commit

git add src/view/status.rs
git commit -m "refactor(status): pakai context::tokens dan context::window

Status bar sekarang memakai penghitungan token yang sama persis dengan
compaction (bukan heuristik /4 terpisah), dan selalu menampilkan angka
context window nyata alih-alih '?' saat model role tidak override
context_window secara eksplisit — konsisten dengan fallback yang
sudah dipakai compaction sendiri."

---

Task 9: Concise-mode system-prompt toggle

Files:

• Modify: src/model/settings.rs
• Modify: src/app/runtime/actions/mod.rs:930-936

Interfaces:

• Produces: Settings.concise_output: bool (new field, default false).

• Step 1: Write the failing test

In src/model/settings.rs, add to the existing #[cfg(test)] mod tests block:

    #[test]
    fn concise_output_defaults_to_false() {
        assert!(!Settings::default().concise_output);
    }

    #[test]
    fn missing_concise_output_field_falls_back_to_default() {
        // Simulates loading a settings.json written before this field
        // existed — #[serde(default)] must fill it in rather than
        // failing the whole parse.
        let old_json = r#"{
            "internet_mode": "Off",
            "provider": "zen",
            "model": "deepseek-v4-flash-free",
            "api_keys": {},
            "max_tokens": null,
            "temperature": null,
            "review_enabled": true,
            "review_max_lessons_per_run": 5,
            "adaptive_review_max_skip": 3,
            "verify_command": null,
            "verify_timeout_ms": 30000,
            "workflow_max_concurrency": 5,
            "session_archive_enabled": true,
            "lsp_auto_provision": true,
            "lsp_languages": []
        }"#;
        let parsed: Settings = serde_json::from_str(old_json)
            .expect("must parse even without the new field present");
        assert!(!parsed.concise_output);
    }

• Step 2: Run test to verify it fails

Run: cargo test --bin zesdex settings::tests::concise_output Expected: compile error — concise_output field doesn't exist on Settings.

• Step 3: Add the field

In src/model/settings.rs, add to the Settings struct (after hive_mind_node_timeout_ms):

    /// Off by default. When enabled, appends an instruction to the
    /// system prompt asking the model to write tersely — drop articles,
    /// filler words, hedging, and pleasantries; keep code, commands, and
    /// error text byte-exact — with an explicit exception for
    /// destructive-operation confirmations and security warnings, which
    /// always get full detail regardless of this setting.
    #[serde(default)]
    pub concise_output: bool,

And in impl Default for Settings, add concise_output: false, to the struct literal.

• Step 4: Run test to verify it passes

Run: cargo test --bin zesdex settings::tests Expected: test result: ok. — all settings tests pass, including the two new ones and the pre-existing hive_mind_node_timeout_ms ones (unaffected).

• Step 5: Wire the system-prompt fragment

In src/app/runtime/actions/mod.rs, replace the system_text assembly (lines 930-936):

    let system_text = format!(
        "{}\n\n{}\n\n{}{}",
        crate::resources::SYSTEM_PROMPT,
        crate::resources::SYSTEM_TOOLS,
        tree_info,
        memory_section,
    );

with:

    let concise_section = if state.settings.concise_output {
        "\n\nWrite tersely: drop articles (a/an/the), filler words (just/really/basically/\
         actually/simply), pleasantries (sure/certainly/of course/happy to), and hedging. \
         Fragments are fine. Code, commands, file paths, and error text must stay byte-exact \
         — never abbreviate or paraphrase those. Exception: for destructive-operation \
         confirmations and security-relevant warnings, always give full detail regardless of \
         this instruction — clarity matters more than brevity when something risky is at stake."
    } else {
        ""
    };
    let system_text = format!(
        "{}\n\n{}\n\n{}{}{}",
        crate::resources::SYSTEM_PROMPT,
        crate::resources::SYSTEM_TOOLS,
        tree_info,
        memory_section,
        concise_section,
    );

run_agent_turn takes tc: &TurnCtx, messages, and events_q only — state/Settings is not in scope at this call site, so the snippet above must read tc.concise_output, not state.settings.concise_output:

    let concise_section = if tc.concise_output {

This requires threading a new field through TurnCtx. In struct TurnCtx (the definition just after spawn_turn, currently ending hive_mind_converged: bool,\n}), add the field:

    hive_mind_converged: bool,
    /// Snapshot of `Settings.concise_output` taken at the start of this
    /// turn, so the system-prompt assembly above can read it without
    /// `TurnCtx` needing a `Settings` reference.
    concise_output: bool,
}

In spawn_turn, after the line added in Task 6 (let context_window = crate::app::runtime::context::window::resolve(&state.app_config, &state.settings);), add:

    let concise_output = state.settings.concise_output;

And in the TurnCtx { ... } literal built inside the std::thread::spawn(move || { ... }) closure (which already lists context_window, and hive_mind_converged,), add concise_output, as its own line:

            context_window,
            concise_output,

            workspace_roots,

• Step 6: Build and verify

Run: cargo build Expected: builds clean.

Run: cargo test --bin zesdex Expected: all tests pass.

• Step 7: Commit

git add src/model/settings.rs src/app/runtime/actions/mod.rs
git commit -m "feat(settings): tambah mode ringkas opsional (concise_output)

Off by default, diaktifkan lewat settings.json (belum ada UI toggle —
sama seperti review_enabled/session_archive_enabled/lsp_auto_provision
yang juga cuma bisa diedit manual hari ini). Saat aktif, system prompt
diberi instruksi menulis ringkas, dengan pengecualian eksplisit untuk
konfirmasi operasi destruktif dan peringatan keamanan yang tetap harus
detail penuh."

---

Final verification (after Task 9)

• Run: cargo build --release — matches CI's "Build" step, expect success.
• Run: cargo test — matches CI's "Test" step, expect all tests pass (context::* plus every pre-existing test in the suite, unaffected).
• Run: grep -rn "runtime::shortsend" src/ — expect no output; the module is fully removed and every caller migrated to context::.
• Run: cargo clippy --message-format=short 2>&1 | grep -E "app/runtime/context/|actions/mod.rs|view/status.rs|model/settings.rs|subagent/division.rs" — expect no output (no new warnings introduced in touched/created files; pre-existing warnings in untouched files like view/markdown.rs/main.rs are out of scope, see Global Constraints).