fix(plan): perbaiki bug entropy gate dan fixture test squash.rs
Ditemukan implementer Task 4 sebelum commit apapun (BLOCKED, bukan kode salah): entropi Shannon mentah per-karakter tidak membedakan prosa dari identifier acak — prosa berulang skor ~3.89 bit/char, lebih tinggi dari UUID (~3.39). Tambah syarat "tanpa spasi" sebelum cek entropi (meniru pre-filter headroom sendiri), turunkan ambang ke 3.0 pada skala mentah. Fixture test array JSON juga diperbesar (repeat 5 -> 8) karena sebelumnya tidak pernah melewati SQUASH_FLOOR_BYTES yang diasumsikan test itu sendiri. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
12a03fd3d1
commit
ceb84790bb
@@ -148,12 +148,18 @@ bytes pass through unchanged (compression only pays off on large output, and tou
|
||||
results risks losing detail with no token benefit). Above the floor, dispatch by content
|
||||
shape:
|
||||
|
||||
- `squash_json(&str) -> String` — hand-rolled JSON tokenizer; structural tokens (keys,
|
||||
brackets, colons, commas, booleans, null) always kept; string values kept if ≤20 chars or
|
||||
high-entropy (Shannon entropy ≥0.85 bits/char, catches UUIDs/hashes/paths — same threshold
|
||||
headroom uses), otherwise replaced with `"…"` in place; array elements past the first 3
|
||||
compressed harder (values elided regardless of length/entropy). Applied when
|
||||
`serde_json::from_str` on the output succeeds.
|
||||
- `squash_json(&str) -> String` — walks a parsed `serde_json::Value` (not a hand-rolled
|
||||
tokenizer — `serde_json` already handles escaping/nesting correctly, reusing it is simpler
|
||||
and more robust); structural tokens (keys, brackets, colons, commas, booleans, null) always
|
||||
kept; string values kept if ≤20 chars or "identifier-shaped" (no internal whitespace *and*
|
||||
Shannon entropy ≥3.0 bits/char — catches UUIDs/hashes/paths), otherwise replaced with `"…"`
|
||||
in place; array elements past the first 3 compressed harder (values elided regardless of
|
||||
length/entropy). Applied when `serde_json::from_str` on the output succeeds. The
|
||||
no-whitespace pre-filter matters: raw per-character entropy alone doesn't separate prose
|
||||
from identifiers — repeated English prose measures ~3.89 bits/char, higher than a UUID's
|
||||
~3.39 — because prose also draws from a wide character set. headroom's own entropy gate is
|
||||
"cheaply pre-filtered by 'no spaces'" before scoring for the same reason; multi-word values
|
||||
never reach the entropy check at all under this rule.
|
||||
- `squash_log(&str) -> String` — line classifier (error/fail/warn/info/debug/trace by
|
||||
keyword + stack-trace-frame detection) → score
|
||||
(`level_score {1.0 error/fail, 0.5 warn, 0.1 info, 0.05 debug/trace} + 0.3 if
|
||||
|
||||
Reference in New Issue
Block a user