fix(plan): perbaiki bug entropy gate dan fixture test squash.rs

Ditemukan implementer Task 4 sebelum commit apapun (BLOCKED, bukan
kode salah): entropi Shannon mentah per-karakter tidak membedakan
prosa dari identifier acak — prosa berulang skor ~3.89 bit/char,
lebih tinggi dari UUID (~3.39). Tambah syarat "tanpa spasi" sebelum
cek entropi (meniru pre-filter headroom sendiri), turunkan ambang ke
3.0 pada skala mentah. Fixture test array JSON juga diperbesar
(repeat 5 -> 8) karena sebelumnya tidak pernah melewati SQUASH_FLOOR_BYTES
yang diasumsikan test itu sendiri.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
asepharyana
2026-07-16 07:49:43 +07:00
co-authored by Claude Sonnet 5
parent 12a03fd3d1
commit ceb84790bb
2 changed files with 39 additions and 14 deletions
@@ -148,12 +148,18 @@ bytes pass through unchanged (compression only pays off on large output, and tou
results risks losing detail with no token benefit). Above the floor, dispatch by content
shape:
- `squash_json(&str) -> String`hand-rolled JSON tokenizer; structural tokens (keys,
brackets, colons, commas, booleans, null) always kept; string values kept if ≤20 chars or
high-entropy (Shannon entropy ≥0.85 bits/char, catches UUIDs/hashes/paths — same threshold
headroom uses), otherwise replaced with `"…"` in place; array elements past the first 3
compressed harder (values elided regardless of length/entropy). Applied when
`serde_json::from_str` on the output succeeds.
- `squash_json(&str) -> String`walks a parsed `serde_json::Value` (not a hand-rolled
tokenizer — `serde_json` already handles escaping/nesting correctly, reusing it is simpler
and more robust); structural tokens (keys, brackets, colons, commas, booleans, null) always
kept; string values kept if ≤20 chars or "identifier-shaped" (no internal whitespace *and*
Shannon entropy ≥3.0 bits/char — catches UUIDs/hashes/paths), otherwise replaced with `"…"`
in place; array elements past the first 3 compressed harder (values elided regardless of
length/entropy). Applied when `serde_json::from_str` on the output succeeds. The
no-whitespace pre-filter matters: raw per-character entropy alone doesn't separate prose
from identifiers — repeated English prose measures ~3.89 bits/char, higher than a UUID's
~3.39 — because prose also draws from a wide character set. headroom's own entropy gate is
"cheaply pre-filtered by 'no spaces'" before scoring for the same reason; multi-word values
never reach the entropy check at all under this rule.
- `squash_log(&str) -> String` — line classifier (error/fail/warn/info/debug/trace by
keyword + stack-trace-frame detection) → score
(`level_score {1.0 error/fail, 0.5 warn, 0.1 info, 0.05 debug/trace} + 0.3 if