fix(gateway): enable stream for all LLM calls — router always streams SSE
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 3m36s
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 4m9s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 11m29s
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 3m36s
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 4m9s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 11m29s
Audit lanjutan: 6x 'LLM API request failed: Request was aborted' per jam. Root cause: 9router/omniroute SELALU balas SSE (data: chunks) walau request tanpa stream:true — SDK OpenAI non-stream menunggu FULL body sebelum parse, jadi batch moderasi besar yang upstream-nya lambat kena timeout 30-60s dan di-abort. llmClient sudah punya agregasi streaming (chunks → ChatCompletion). Fix: stream:true di llmCaller (moderasi batch/individual), llmVision, cultureLearner, userProfileLearner. Verified: SDK stream test 806ms vs sebelumnya abort. Caller lain (recovery worker dll) lewat llmCaller sama.
This commit is contained in:
@@ -87,6 +87,7 @@ hal dasar ke newbie tanpa judge."`;
|
|||||||
max_tokens: 500,
|
max_tokens: 500,
|
||||||
temperature: 0.7, // Higher temp for summarization
|
temperature: 0.7, // Higher temp for summarization
|
||||||
retries: 2,
|
retries: 2,
|
||||||
|
stream: true, // router always streams SSE; non-stream waits for full body and times out
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!completion) throw new Error("Empty response from LLM");
|
if (!completion) throw new Error("Empty response from LLM");
|
||||||
|
|||||||
@@ -79,6 +79,13 @@ export async function callModerationLLM(
|
|||||||
jsonResponse: { type: "json_object" },
|
jsonResponse: { type: "json_object" },
|
||||||
retries: 0,
|
retries: 0,
|
||||||
signal,
|
signal,
|
||||||
|
// Router (9router/omniroute) always streams SSE even when the
|
||||||
|
// request omits `stream`. In non-stream mode the OpenAI SDK waits
|
||||||
|
// for the FULL body before parsing, so slow/long upstream streams
|
||||||
|
// hit the 30s/60s timeout and abort mid-generation. Streaming mode
|
||||||
|
// consumes chunks incrementally — timeout only fires on a real
|
||||||
|
// stall. llmClient aggregates the stream into a ChatCompletion.
|
||||||
|
stream: true,
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!completion)
|
if (!completion)
|
||||||
|
|||||||
@@ -272,6 +272,7 @@ export async function llmVision(
|
|||||||
temperature: 0.1,
|
temperature: 0.1,
|
||||||
top_p: 0.9,
|
top_p: 0.9,
|
||||||
retries: 0,
|
retries: 0,
|
||||||
|
stream: true, // router always streams SSE; non-stream waits for full body and times out
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!completion) return null;
|
if (!completion) return null;
|
||||||
|
|||||||
@@ -116,6 +116,7 @@ atau konten SARA, itu akan SANGAT tidak sesuai dengan karakternya dan patut dicu
|
|||||||
max_tokens: 2000,
|
max_tokens: 2000,
|
||||||
temperature: 0.7, // Higher temp for summarization
|
temperature: 0.7, // Higher temp for summarization
|
||||||
retries: 2,
|
retries: 2,
|
||||||
|
stream: true, // router always streams SSE; non-stream waits for full body and times out
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!completion) throw new Error("Empty response from LLM");
|
if (!completion) throw new Error("Empty response from LLM");
|
||||||
|
|||||||
Reference in New Issue
Block a user