Add configurable SystemOne batching and release version 4.1.0
This commit is contained in:
parent
9a5404cb62
commit
7edae937b3
13 changed files with 400 additions and 31 deletions
27
AGENTS.md
27
AGENTS.md
|
|
@ -48,8 +48,10 @@ sent only for Chat Completions.
|
|||
|
||||
Chat Completions sends system/user messages and extracts the last JSON object from
|
||||
`choices[0].message.content`, requiring a `match` or `matched` boolean. SystemOne
|
||||
sends prepared email text as `state` and a single `questions.match` of type `noul`.
|
||||
It requires `answers.match.type === "noul"` and a finite numeric `noul` in `[0, 1]`.
|
||||
sends prepared email text as `state` with `noul` questions identified by stable
|
||||
`rule_<snapshot index>` IDs. The single-question API still uses `match` for backward
|
||||
compatibility and cache fingerprinting. Every requested answer must have type
|
||||
`noul` and a finite numeric probability in `[0, 1]`; response order is irrelevant.
|
||||
No chat generation parameters or system prompt are sent to SystemOne.
|
||||
|
||||
The global `decisionThreshold` defaults to 0.5; optional rule-level
|
||||
|
|
@ -67,6 +69,27 @@ error, and continues the next queued email. Snapshot settings and rules per emai
|
|||
`inputCharacterLimit` replaces coupling input truncation to output `max_tokens`;
|
||||
it applies with aggressive reduction and migrates from the old effective limit.
|
||||
|
||||
### SystemOne batching
|
||||
|
||||
`systemOneProcessAll` defaults to true; `systemOneBatchSize` defaults to 8 and must
|
||||
be a positive safe integer. Only SystemOne shows these controls. The numeric input
|
||||
is disabled while Process all is enabled, retaining its value. Include both in
|
||||
settings import/export, startup loading, live updates, and per-email snapshots.
|
||||
|
||||
`classifyRules()` is a lazy ordered async iterator over precomputed candidate keys.
|
||||
Chat requests stay sequential. SystemOne batches the next N unique uncached keys
|
||||
(or all), sends state once, validates the whole response, and persists one batch
|
||||
of evidence before yielding ordered results. Per-rule thresholds remain independent.
|
||||
Do not change existing cache keys or include batch settings in their fingerprint.
|
||||
Prepare eligibility and keys before actions, using the existing initial mail metadata.
|
||||
|
||||
Stop-after-match ends action iteration and prevents further requests; already
|
||||
computed later answers remain cached. A malformed answer fails its entire batch:
|
||||
no partial cache writes or new actions. Keep cache-generation guards for imports
|
||||
and clearing during inference. No automatic splitting or retries. Tests in
|
||||
`tests/batching.test.mjs` cover batching, cache reuse, failure atomicity, and actual
|
||||
rule-runner stop behavior with the real classifier.
|
||||
|
||||
## Documentation
|
||||
|
||||
Additional documentation exists outside this repository.
|
||||
|
|
|
|||
20
README.md
20
README.md
|
|
@ -74,7 +74,7 @@ llama-server -hf ggml-org/Laya-GGUF
|
|||
for a single loaded model. Router deployments require the appropriate model ID.
|
||||
2. Set the default match probability threshold (initially **50%**). Each rule has
|
||||
an optional percentage override; leave it blank to inherit the default.
|
||||
3. Apply rules normally. One `noul` question is sent per evaluated rule. A rule
|
||||
3. Apply rules normally. Each uncached criterion becomes a `noul` question. A rule
|
||||
matches when the unrounded probability is **greater than or equal to** its
|
||||
effective threshold. For example, 80% matches an 80% threshold.
|
||||
4. Open Details to see a probability bar with the percentage inside. Hover over
|
||||
|
|
@ -83,6 +83,24 @@ llama-server -hf ggml-org/Laya-GGUF
|
|||
not automatically rerun actions. Decision models provide no textual explanation;
|
||||
no second model is called to invent one. Use Chat Completions for explanations.
|
||||
|
||||
**SystemOne batching:** **Process all** is enabled by default and sends all eligible,
|
||||
uncached criteria for an email in one request, with the email state included once.
|
||||
Turn it off to set **Rules per request** (initially 8); use 1 for sequential
|
||||
requests. The number is retained while disabled. These controls apply only to
|
||||
SystemOne; Chat Completions continues processing one rule per request.
|
||||
|
||||
Cached rules consume no batch capacity, and identical criteria share a question
|
||||
while retaining their individual thresholds. Actions execute in rule order and
|
||||
honor stop-after-match. A batch can evaluate later rules whose actions are then
|
||||
skipped; their probabilities remain available in Details. No further batches are
|
||||
requested after stopping. Changing batch settings preserves existing cached results.
|
||||
|
||||
If any requested answer is missing or invalid, the entire batch fails without
|
||||
caching its new answers or executing their actions. Previously completed actions
|
||||
remain applied. Sortana does not automatically retry or split rejected batches;
|
||||
if the server rejects a large request, disable Process all and reduce Rules per
|
||||
request. Limits and performance depend on the server and model.
|
||||
|
||||
Probabilities are model estimates, not guaranteed accuracy. Evaluate thresholds
|
||||
against representative labeled emails, including long messages and attachments.
|
||||
SystemOne uses the existing text extraction and attachment summaries; it does not
|
||||
|
|
|
|||
|
|
@ -347,7 +347,8 @@ async function processMessage(id) {
|
|||
identityId = null;
|
||||
}
|
||||
|
||||
for (const rule of rules) {
|
||||
const candidates = [];
|
||||
for (const [index, rule] of rules.entries()) {
|
||||
if (rule.enabled === false) {
|
||||
continue;
|
||||
}
|
||||
|
|
@ -375,7 +376,9 @@ async function processMessage(id) {
|
|||
}
|
||||
}
|
||||
const cacheKey = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot);
|
||||
const { matched } = await AiClassifier.classify(text, rule.criterion, { cacheKey, snapshot, threshold: rule.decisionThreshold ?? snapshot.decisionThreshold });
|
||||
candidates.push({ id: `rule_${index}`, rule, cacheKey });
|
||||
}
|
||||
for await (const { rule, result: { matched } } of AiClassifier.classifyRules(text, candidates, snapshot)) {
|
||||
if (matched) {
|
||||
for (const act of (rule.actions || [])) {
|
||||
if (act.type === 'tag' && act.tagKey) {
|
||||
|
|
@ -476,7 +479,7 @@ async function clearCacheForMessages(idsInput) {
|
|||
}
|
||||
|
||||
try {
|
||||
const store = await storage.local.get(["apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]);
|
||||
const store = await storage.local.get(["systemOneProcessAll", "systemOneBatchSize", "apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]);
|
||||
logger.setDebug(store.debugLogging);
|
||||
const { migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js'));
|
||||
if (store.inputCharacterLimit === undefined) {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Sortana",
|
||||
"version": "4.0.0",
|
||||
"version": "4.1.0",
|
||||
"default_locale": "en-US",
|
||||
"applications": {
|
||||
"gecko": {
|
||||
|
|
|
|||
|
|
@ -97,14 +97,7 @@ export function getCachedDetails(key, threshold = config.decisionThreshold) {
|
|||
}
|
||||
export function getCachedResult(key, threshold) { return getCachedDetails(key, threshold)?.matched ?? null; }
|
||||
export function getReason(key) { return cache.get(key)?.reason ?? null; }
|
||||
export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) {
|
||||
await init();
|
||||
if (snapshot.apiType === 'systemone') validateThreshold(threshold);
|
||||
const cached = getCachedDetails(cacheKey, threshold);
|
||||
if (cached) return cached;
|
||||
const epoch = generation;
|
||||
const adapter = adapters[snapshot.apiType];
|
||||
const payload = adapter.buildRequest(text, criterion, snapshot);
|
||||
async function sendRequest(payload, snapshot) {
|
||||
try { await storage.local.set({ lastPayload: payload }); }
|
||||
catch (error) { aiLog('Failed to save last payload', { level: 'warn' }, error); }
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
|
|
@ -126,7 +119,18 @@ export async function classify(text, criterion, { cacheKey = null, snapshot = ge
|
|||
? ' Verify that the running server build supports /v1/systemone and the selected decision model.' : '';
|
||||
throw new Error(`Classification request failed: HTTP ${response.status} at ${endpoint}.${detail}${hint}`);
|
||||
}
|
||||
const raw = await response.json();
|
||||
return response.json();
|
||||
}
|
||||
|
||||
export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) {
|
||||
await init();
|
||||
if (snapshot.apiType === 'systemone') validateThreshold(threshold);
|
||||
const cached = getCachedDetails(cacheKey, threshold);
|
||||
if (cached) return cached;
|
||||
const epoch = generation;
|
||||
const adapter = adapters[snapshot.apiType];
|
||||
const payload = adapter.buildRequest(text, criterion, snapshot);
|
||||
const raw = await sendRequest(payload, snapshot);
|
||||
const entry = { ...adapter.parseResponse(raw), backend: snapshot.apiType,
|
||||
model: typeof raw.model === 'string' ? raw.model : snapshot.model, version: 2,
|
||||
messageHash: cacheKey?.split(':')[1] };
|
||||
|
|
@ -136,6 +140,60 @@ export async function classify(text, criterion, { cacheKey = null, snapshot = ge
|
|||
}
|
||||
return withPolicy(entry, threshold);
|
||||
}
|
||||
// Yield lazily so the action runner can stop without starting the next request.
|
||||
// Candidates have precomputed keys, allowing actions to move/delete the message.
|
||||
export async function* classifyRules(text, candidates, snapshot = getConfig()) {
|
||||
await init();
|
||||
validateSettings(snapshot);
|
||||
if (snapshot.apiType !== 'systemone') {
|
||||
for (const candidate of candidates) {
|
||||
const result = await classify(text, candidate.rule.criterion, {
|
||||
cacheKey: candidate.cacheKey, snapshot,
|
||||
threshold: candidate.rule.decisionThreshold ?? snapshot.decisionThreshold
|
||||
});
|
||||
yield { rule: candidate.rule, result };
|
||||
}
|
||||
return;
|
||||
}
|
||||
const limit = snapshot.systemOneProcessAll ? Infinity : snapshot.systemOneBatchSize;
|
||||
// Keep received evidence locally even if persistence is cleared during this run.
|
||||
const evidence = new Map();
|
||||
for (const candidate of candidates) {
|
||||
validateThreshold(candidate.rule.decisionThreshold ?? snapshot.decisionThreshold);
|
||||
const entry = cache.get(candidate.cacheKey);
|
||||
if (entry) evidence.set(candidate.cacheKey, entry);
|
||||
}
|
||||
for (let index = 0; index < candidates.length; index++) {
|
||||
const candidate = candidates[index];
|
||||
if (!evidence.has(candidate.cacheKey)) {
|
||||
const pending = new Map();
|
||||
for (let next = index; next < candidates.length && pending.size < limit; next++) {
|
||||
const item = candidates[next];
|
||||
if (!evidence.has(item.cacheKey) && !pending.has(item.cacheKey)) {
|
||||
pending.set(item.cacheKey, { ...item, criterion: item.rule.criterion });
|
||||
}
|
||||
}
|
||||
const questions = [...pending.values()];
|
||||
const epoch = generation;
|
||||
const raw = await sendRequest(systemone.buildBatchRequest(text, questions, snapshot), snapshot);
|
||||
// Parse the complete batch before accepting any of its evidence.
|
||||
const parsed = systemone.parseBatchResponse(raw, questions);
|
||||
const entries = questions.map((question, i) => [question.cacheKey, {
|
||||
...parsed[i], backend: snapshot.apiType, version: 2,
|
||||
model: typeof raw.model === 'string' ? raw.model : snapshot.model,
|
||||
messageHash: question.cacheKey.split(':')[1]
|
||||
}]);
|
||||
for (const [key, entry] of entries) evidence.set(key, entry);
|
||||
if (epoch === generation) {
|
||||
for (const [key, entry] of entries) cache.set(key, entry);
|
||||
await saveCache();
|
||||
}
|
||||
}
|
||||
yield { rule: candidate.rule, result: withPolicy(evidence.get(candidate.cacheKey),
|
||||
candidate.rule.decisionThreshold ?? snapshot.decisionThreshold) };
|
||||
}
|
||||
}
|
||||
|
||||
export async function classifyText(text, criterion, cacheKey = null) { return (await classify(text, criterion, { cacheKey })).matched; }
|
||||
export async function removeCacheEntries(keys = []) {
|
||||
await init(); generation++;
|
||||
|
|
|
|||
|
|
@ -1,17 +1,22 @@
|
|||
export const path = '/v1/systemone';
|
||||
export function buildRequest(text, criterion, config) {
|
||||
const payload = {
|
||||
state: text,
|
||||
questions: { match: { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` } }
|
||||
};
|
||||
return buildBatchRequest(text, [{ id: 'match', criterion }], config);
|
||||
}
|
||||
export function buildBatchRequest(text, questions, config) {
|
||||
const payload = { state: text, questions: Object.fromEntries(questions.map(({ id, criterion }) =>
|
||||
[id, { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` }])) };
|
||||
if (config.model) payload.model = config.model;
|
||||
return payload;
|
||||
}
|
||||
export function parseResponse(result) {
|
||||
const answer = result?.answers?.match;
|
||||
export function parseResponse(result) { return parseAnswer(result, 'match'); }
|
||||
export function parseBatchResponse(result, questions) {
|
||||
return questions.map(({ id }) => parseAnswer(result, id));
|
||||
}
|
||||
function parseAnswer(result, id) {
|
||||
const answer = result?.answers?.[id];
|
||||
if (answer?.type !== 'noul' || typeof answer.noul !== 'number' ||
|
||||
!Number.isFinite(answer.noul) || answer.noul < 0 || answer.noul > 1) {
|
||||
throw new Error('SystemOne response must contain answers.match with type noul and a probability from 0 to 1.');
|
||||
throw new Error(`SystemOne response must contain answers.${id} with type noul and a probability from 0 to 1.`);
|
||||
}
|
||||
return { matched: null, reason: null, probability: answer.noul };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { DEFAULT_AI_PARAMS } from './defaultParams.js';
|
|||
export const CONFIG_DEFAULTS = {
|
||||
apiType: 'chat-completions', endpoint: 'http://127.0.0.1:5000', model: '',
|
||||
apiKey: '', openaiOrganization: '', openaiProject: '', customSystemPrompt: '',
|
||||
systemOneProcessAll: true, systemOneBatchSize: 8,
|
||||
decisionThreshold: 0.5, inputCharacterLimit: 3686, aiParams: DEFAULT_AI_PARAMS,
|
||||
htmlToMarkdown: false, stripUrlParams: false, altTextImages: false,
|
||||
collapseWhitespace: false, tokenReduction: false, debugLogging: false
|
||||
|
|
@ -13,6 +14,12 @@ export function validateThreshold(value) {
|
|||
return value;
|
||||
}
|
||||
export function validateSettings(data) {
|
||||
if (data.systemOneProcessAll !== undefined && typeof data.systemOneProcessAll !== 'boolean') {
|
||||
throw new Error('Process all must be a boolean.');
|
||||
}
|
||||
if (data.systemOneBatchSize !== undefined && (!Number.isSafeInteger(data.systemOneBatchSize) || data.systemOneBatchSize < 1)) {
|
||||
throw new Error('Rules per request must be a positive whole number.');
|
||||
}
|
||||
if (data.apiType !== undefined && !['chat-completions', 'systemone'].includes(data.apiType)) throw new Error('Unknown API type.');
|
||||
if (data.decisionThreshold !== undefined) validateThreshold(data.decisionThreshold);
|
||||
if (data.inputCharacterLimit !== undefined && (!Number.isSafeInteger(data.inputCharacterLimit) || data.inputCharacterLimit < 1)) {
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { validCacheEntry } from '../modules/AiClassifier.js';
|
|||
const storage = (globalThis.messenger ?? browser).storage;
|
||||
const KEY_GROUPS = {
|
||||
settings: [
|
||||
'systemOneProcessAll', 'systemOneBatchSize',
|
||||
'apiType', 'decisionThreshold', 'inputCharacterLimit', 'tokenReduction',
|
||||
'endpoint',
|
||||
'model',
|
||||
|
|
|
|||
|
|
@ -78,6 +78,15 @@
|
|||
<option value="systemone">SystemOne</option>
|
||||
</select></div>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="checkbox"><input type="checkbox" id="systemone-process-all"> Process all</label>
|
||||
<p class="help">Group all eligible, uncached rules for each email into one request. Actions still follow rule order and stop-after-match.</p>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="label" for="systemone-batch-size">Rules per request</label>
|
||||
<input class="input" id="systemone-batch-size" type="number" min="1" step="1" required>
|
||||
<p class="help">Turn off Process all to limit each request. Use 1 for one rule at a time.</p>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="label" for="decision-threshold">Default match probability threshold (%)</label>
|
||||
<input class="input" id="decision-threshold" type="number" min="0" max="100" step="any">
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||
const { DEFAULT_AI_PARAMS } = await import(browser.runtime.getURL('modules/defaultParams.js'));
|
||||
const { validateSettings, migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js'));
|
||||
const defaults = await storage.local.get([
|
||||
'systemOneProcessAll', 'systemOneBatchSize',
|
||||
'apiType', 'decisionThreshold', 'inputCharacterLimit',
|
||||
'endpoint',
|
||||
'customSystemPrompt',
|
||||
|
|
@ -90,6 +91,15 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||
});
|
||||
const apiTypeInput = document.getElementById('api-type');
|
||||
apiTypeInput.value = defaults.apiType || 'chat-completions';
|
||||
const processAllInput = document.getElementById('systemone-process-all');
|
||||
const batchSizeInput = document.getElementById('systemone-batch-size');
|
||||
processAllInput.checked = defaults.systemOneProcessAll ?? true;
|
||||
let savedBatchSize = defaults.systemOneBatchSize ?? 8;
|
||||
batchSizeInput.value = savedBatchSize;
|
||||
function updateBatchControls() {
|
||||
batchSizeInput.disabled = processAllInput.checked || apiTypeInput.value !== 'systemone';
|
||||
}
|
||||
processAllInput.addEventListener('change', updateBatchControls);
|
||||
const thresholdInput = document.getElementById('decision-threshold');
|
||||
thresholdInput.value = (defaults.decisionThreshold ?? 0.5) * 100;
|
||||
const inputLimit = document.getElementById('input-character-limit');
|
||||
|
|
@ -97,6 +107,7 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||
const settingsError = document.getElementById('settings-error');
|
||||
function updateApiControls() {
|
||||
const decision = apiTypeInput.value === 'systemone';
|
||||
updateBatchControls();
|
||||
document.querySelectorAll('.systemone-only').forEach(el => el.classList.toggle('is-hidden', !decision));
|
||||
for (const id of ['system-instructions', 'openai-organization', 'openai-project', ...Object.keys(DEFAULT_AI_PARAMS)]) {
|
||||
document.getElementById(id)?.closest('.field')?.classList.toggle('is-hidden', decision);
|
||||
|
|
@ -1097,6 +1108,14 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||
document.getElementById('save').addEventListener('click', async () => {
|
||||
settingsError.textContent = '';
|
||||
const apiType = apiTypeInput.value;
|
||||
const systemOneProcessAll = processAllInput.checked;
|
||||
const enteredBatchSize = Number(batchSizeInput.value);
|
||||
const validBatchSize = Number.isSafeInteger(enteredBatchSize) && enteredBatchSize > 0;
|
||||
if (!batchSizeInput.disabled && !validBatchSize) {
|
||||
settingsError.textContent = 'Rules per request must be a positive whole number.';
|
||||
return;
|
||||
}
|
||||
const systemOneBatchSize = validBatchSize ? enteredBatchSize : savedBatchSize;
|
||||
const decisionThreshold = thresholdInput.value.trim() === '' ? NaN : Number(thresholdInput.value) / 100;
|
||||
const inputCharacterLimit = Number(inputLimit.value);
|
||||
const endpoint = endpointInput.value.trim();
|
||||
|
|
@ -1127,13 +1146,15 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||
const tokenReduction = tokenReductionToggle.checked;
|
||||
const showDebugTab = debugTabToggle.checked;
|
||||
const theme = themeSelect.value;
|
||||
try { validateSettings({ apiType, decisionThreshold, inputCharacterLimit, aiRules: rules }); }
|
||||
try { validateSettings({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, aiRules: rules }); }
|
||||
catch (error) { settingsError.textContent = error.message; return; }
|
||||
await storage.local.set({ apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, aiRules: rules, theme, showDebugTab });
|
||||
await storage.local.set({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, aiRules: rules, theme, showDebugTab });
|
||||
savedBatchSize = systemOneBatchSize;
|
||||
batchSizeInput.value = savedBatchSize;
|
||||
await storage.local.remove(['templateName', 'customTemplate']);
|
||||
await applyTheme(theme);
|
||||
try {
|
||||
await AiClassifier.setConfig({ apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging });
|
||||
await AiClassifier.setConfig({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging });
|
||||
logger.setDebug(debugLogging);
|
||||
} catch (e) {
|
||||
logger.aiLog('[options] failed to apply config', {level: 'error'}, e);
|
||||
|
|
|
|||
219
tests/batching.test.mjs
Normal file
219
tests/batching.test.mjs
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import vm from 'node:vm';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { CONFIG_DEFAULTS, validateSettings } from '../modules/classificationConfig.js';
|
||||
import * as systemone from '../modules/backends/systemOne.js';
|
||||
|
||||
const data = {};
|
||||
let cacheWrites = 0;
|
||||
globalThis.messenger = { storage: { local: {
|
||||
async get(keys) { return Object.fromEntries((Array.isArray(keys) ? keys : [keys]).map(key => [key, data[key]])); },
|
||||
async set(values) { Object.assign(data, structuredClone(values)); if ('aiCache' in values) cacheWrites++; },
|
||||
async remove() {}
|
||||
} }, messages: { async get(id) { return { headerMessageId: `mail-${id}` }; } } };
|
||||
const classifier = await import('../modules/AiClassifier.js');
|
||||
const transfer = await import('../options/dataTransfer.js');
|
||||
let requests = [];
|
||||
async function setup(update = {}) {
|
||||
await classifier.clearCache();
|
||||
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone', ...update });
|
||||
requests = []; cacheWrites = 0;
|
||||
globalThis.fetch = async (url, request) => {
|
||||
const body = JSON.parse(request.body);
|
||||
requests.push({ url, body, headers: request.headers });
|
||||
// Intentionally reverse response order and include an unsolicited answer.
|
||||
return { ok: true, json: async () => ({ model: 'served', answers: {
|
||||
unsolicited: { type: 'noul', noul: 1 },
|
||||
...Object.fromEntries(Object.keys(body.questions).reverse().map(id => [id, { type: 'noul', noul: 0.75 }]))
|
||||
} }) };
|
||||
};
|
||||
return classifier.getConfig();
|
||||
}
|
||||
async function candidates(count, snapshot, criteria = []) {
|
||||
return Promise.all(Array.from({ length: count }, async (_, i) => {
|
||||
const criterion = criteria[i] ?? `criterion ${i}`;
|
||||
return { id: `rule_${i}`, rule: { criterion, actions: [] }, cacheKey: await classifier.buildCacheKey(1, criterion, snapshot) };
|
||||
}));
|
||||
}
|
||||
async function collect(items, snapshot) {
|
||||
const results = [];
|
||||
for await (const value of classifier.classifyRules('email state', items, snapshot)) results.push(value);
|
||||
return results;
|
||||
}
|
||||
|
||||
test('Process all, explicit batch sizes, partial batches, empty input and cache reuse', async () => {
|
||||
for (const [settings, expected] of [
|
||||
[{}, [10]], [{ systemOneProcessAll: false, systemOneBatchSize: 8 }, [8, 2]],
|
||||
[{ systemOneProcessAll: false, systemOneBatchSize: 1 }, Array(10).fill(1)]
|
||||
]) {
|
||||
const snapshot = await setup(settings);
|
||||
const items = await candidates(10, snapshot);
|
||||
const results = await collect(items, snapshot);
|
||||
assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), expected);
|
||||
assert.equal(cacheWrites, expected.length);
|
||||
assert.deepEqual(results.map(r => r.rule.criterion), items.map(r => r.rule.criterion));
|
||||
assert.ok(results.every(r => r.result.probability === 0.75));
|
||||
assert.ok(requests.every(r => r.body.state === 'email state' && !('messages' in r.body)));
|
||||
await collect(items, snapshot);
|
||||
assert.equal(requests.length, expected.length);
|
||||
await collect([], snapshot);
|
||||
assert.equal(requests.length, expected.length);
|
||||
}
|
||||
});
|
||||
|
||||
test('cached rules consume no capacity and duplicate criteria use independent thresholds', async () => {
|
||||
const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 });
|
||||
const items = await candidates(5, snapshot, ['cached', 'same', 'cached', 'other', 'same']);
|
||||
await collect([items[0]], snapshot);
|
||||
requests = []; cacheWrites = 0;
|
||||
items[1].rule.decisionThreshold = 0.8;
|
||||
items[4].rule.decisionThreshold = 0.7;
|
||||
const results = await collect(items, snapshot);
|
||||
assert.equal(requests.length, 1);
|
||||
assert.deepEqual(Object.keys(requests[0].body.questions), ['rule_1', 'rule_3']);
|
||||
assert.equal(results[1].result.matched, false);
|
||||
assert.equal(results[4].result.matched, true);
|
||||
assert.equal(cacheWrites, 1);
|
||||
assert.equal(await classifier.buildCacheKey(1, 'same', { ...snapshot, systemOneProcessAll: true, systemOneBatchSize: 100 }), items[1].cacheKey);
|
||||
});
|
||||
|
||||
test('answer identity determines probability, not answer order', () => {
|
||||
const result = { answers: { rule_2: { type: 'noul', noul: 0 }, rule_1: { type: 'noul', noul: 1 } } };
|
||||
assert.deepEqual(systemone.parseBatchResponse(result, [{ id: 'rule_1' }, { id: 'rule_2' }]).map(r => r.probability), [1, 0]);
|
||||
});
|
||||
|
||||
test('failed batches neither yield nor cache partial results and do not retry', async () => {
|
||||
for (const failure of ['missing', 'invalid', 'http']) {
|
||||
const snapshot = await setup();
|
||||
const items = await candidates(2, snapshot);
|
||||
let calls = 0, yielded = 0;
|
||||
globalThis.fetch = async () => {
|
||||
calls++;
|
||||
return { ok: failure !== 'http', status: 413, json: async () => ({ answers: {
|
||||
rule_0: { type: 'noul', noul: 0.9 },
|
||||
...(failure === 'invalid' ? { rule_1: { type: 'noul', noul: '0.9' } } : {})
|
||||
} }) };
|
||||
};
|
||||
await assert.rejects(async () => {
|
||||
for await (const value of classifier.classifyRules('state', items, snapshot)) yielded++;
|
||||
});
|
||||
assert.equal(yielded, 0);
|
||||
assert.equal(calls, 1);
|
||||
assert.equal(cacheWrites, 0);
|
||||
assert.equal(await classifier.getCacheSize(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('clearing or importing during a batch prevents stale cache writes', async () => {
|
||||
for (const replace of [() => classifier.clearCache(), () => classifier.replaceCache({})]) {
|
||||
const snapshot = await setup();
|
||||
const items = await candidates(2, snapshot);
|
||||
const fetch = globalThis.fetch;
|
||||
globalThis.fetch = async (...args) => { await replace(); return fetch(...args); };
|
||||
const results = await collect(items, snapshot);
|
||||
assert.equal(results.length, 2);
|
||||
assert.equal(requests.length, 1);
|
||||
assert.equal(await classifier.getCacheSize(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('Chat Completions remains lazy and sequential', async () => {
|
||||
const snapshot = await setup({ apiType: 'chat-completions' });
|
||||
const items = await candidates(3, snapshot);
|
||||
globalThis.fetch = async (url, request) => {
|
||||
requests.push(JSON.parse(request.body));
|
||||
return { ok: true, json: async () => ({ choices: [{ message: { content: '{"match":true}' } }] }) };
|
||||
};
|
||||
for await (const value of classifier.classifyRules('state', items, snapshot)) break;
|
||||
assert.equal(requests.length, 1);
|
||||
assert.ok(Array.isArray(requests[0].messages));
|
||||
});
|
||||
|
||||
test('batch settings validate and survive import', async () => {
|
||||
for (const value of [0, -1, 1.2, Infinity, '8', null]) assert.throws(() => validateSettings({ systemOneBatchSize: value }));
|
||||
assert.throws(() => validateSettings({ systemOneProcessAll: 'true' }));
|
||||
await transfer.importData({ text: async () => JSON.stringify({ systemOneProcessAll: false, systemOneBatchSize: 3 }) }, ['settings']);
|
||||
assert.equal(data.systemOneProcessAll, false);
|
||||
assert.equal(data.systemOneBatchSize, 3);
|
||||
await assert.rejects(transfer.importData({ text: async () => '{"systemOneBatchSize":0}' }, ['settings']));
|
||||
});
|
||||
|
||||
// Exercise the action runner with the real lazy classifier and mocked mail APIs.
|
||||
const source = await readFile(new URL('../background.js', import.meta.url), 'utf8');
|
||||
const functions = source.slice(source.indexOf('async function processMessage('), source.indexOf('async function clearCacheForMessages('));
|
||||
test('stop after match skips later actions and batches; all-mode retains later evidence', async () => {
|
||||
for (const processAll of [true, false]) {
|
||||
const snapshot = await setup({ systemOneProcessAll: processAll, systemOneBatchSize: 2 });
|
||||
const actions = [], errors = [];
|
||||
const rules = Array.from({ length: 5 }, (_, i) => ({ criterion: `rule ${i}`, actions: [{ type: 'flag', flagged: true }], stopProcessing: i === 1 }));
|
||||
const context = vm.createContext({
|
||||
structuredClone, Date, Error, aiRules: rules, AiClassifier: classifier,
|
||||
queue: Promise.resolve(), queuedCount: 0, processing: false, currentStart: 0,
|
||||
tokenReduction: false, showDebugTab: false, timingStats: {},
|
||||
updateActionIcon() {}, updateTimingStats() {}, showTransientIcon() {},
|
||||
buildEmailText() { return 'state'; }, ICONS: { circle: 'circle' },
|
||||
logger: { aiLog() {} }, recordError(...args) { errors.push(args); }, ERROR_NOTIFICATION_ID: 'error',
|
||||
storage: messenger.storage,
|
||||
messenger: { messages: {
|
||||
async getFull() { return {}; }, async get() { return { tags: [], folder: { accountId: 'account' } }; },
|
||||
async update(id, value) { actions.push({ id, value }); }
|
||||
}, identities: { async list() { return []; } } },
|
||||
browser: { runtime: { getURL: x => x }, notifications: { async create() {} } }
|
||||
});
|
||||
vm.runInContext(functions, context);
|
||||
await vm.runInContext('applyAiRules([1])', context);
|
||||
assert.equal(errors.length, 0);
|
||||
assert.equal(actions.length, 2);
|
||||
assert.equal(requests.length, 1);
|
||||
assert.equal(Object.keys(requests[0].body.questions).length, processAll ? 5 : 2);
|
||||
assert.equal(await classifier.getCacheSize(), processAll ? 5 : 2);
|
||||
}
|
||||
});
|
||||
|
||||
test('batch controls default to all and retain numeric input across toggles and API switches', async () => {
|
||||
const optionsSource = await readFile(new URL('../options/options.js', import.meta.url), 'utf8');
|
||||
const section = optionsSource.slice(optionsSource.indexOf(' const apiTypeInput ='), optionsSource.indexOf(' const endpointInput ='));
|
||||
const fields = new Map();
|
||||
const field = id => {
|
||||
if (!fields.has(id)) fields.set(id, {
|
||||
value: '', checked: false, disabled: false, listeners: {},
|
||||
addEventListener(event, fn) { this.listeners[event] = fn; },
|
||||
classList: { toggle() {} }, closest() { return this; }
|
||||
});
|
||||
return fields.get(id);
|
||||
};
|
||||
const context = vm.createContext({
|
||||
defaults: { apiType: 'systemone' }, DEFAULT_AI_PARAMS: {}, migratedInputLimit() { return 3686; },
|
||||
updateEndpointPreview() {}, document: { getElementById: field, querySelectorAll() { return []; } }
|
||||
});
|
||||
vm.runInContext(`${section}\nupdateApiControls();`, context);
|
||||
const all = field('systemone-process-all'), size = field('systemone-batch-size'), api = field('api-type');
|
||||
assert.equal(all.checked, true);
|
||||
assert.equal(size.value, 8);
|
||||
assert.equal(size.disabled, true);
|
||||
all.checked = false; all.listeners.change();
|
||||
assert.equal(size.disabled, false);
|
||||
size.value = '3';
|
||||
all.checked = true; all.listeners.change();
|
||||
assert.equal(size.disabled, true);
|
||||
all.checked = false; all.listeners.change();
|
||||
assert.equal(size.value, '3');
|
||||
api.value = 'chat-completions'; api.listeners.change();
|
||||
assert.equal(size.disabled, true);
|
||||
api.value = 'systemone'; api.listeners.change();
|
||||
assert.equal(size.disabled, false);
|
||||
assert.equal(size.value, '3');
|
||||
});
|
||||
|
||||
test('batch settings remain fixed while a per-email snapshot is running', async () => {
|
||||
const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 });
|
||||
const items = await candidates(5, snapshot);
|
||||
const fetch = globalThis.fetch;
|
||||
globalThis.fetch = async (...args) => {
|
||||
classifier.setConfig({ systemOneProcessAll: true, systemOneBatchSize: 1 });
|
||||
return fetch(...args);
|
||||
};
|
||||
await collect(items, snapshot);
|
||||
assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), [2, 2, 1]);
|
||||
});
|
||||
|
|
@ -51,7 +51,7 @@ const messageStart = source.indexOf('browser.runtime.onMessage.addListener(');
|
|||
vm.runInContext(source.slice(messageStart, source.indexOf('\n\n // Automatically classify', messageStart)), context);
|
||||
|
||||
test('export/reinstall/import restores Details immediately with changed Thunderbird IDs', async () => {
|
||||
const settings = { ...CONFIG_DEFAULTS, apiType: 'systemone', endpoint: 'http://decision-server:8080', tokenReduction: true, inputCharacterLimit: 5000 };
|
||||
const settings = { ...CONFIG_DEFAULTS, systemOneProcessAll: false, systemOneBatchSize: 3, apiType: 'systemone', endpoint: 'http://decision-server:8080', tokenReduction: true, inputCharacterLimit: 5000 };
|
||||
const rules = [{ criterion: 'invoice', decisionThreshold: 0.8, actions: [] }];
|
||||
await storage.local.set({ ...settings, aiRules: rules });
|
||||
globalThis.fetch = async () => ({ ok: true, json: async () => ({ answers: { match: { type: 'noul', noul: 0.9 } } }) });
|
||||
|
|
@ -73,6 +73,8 @@ test('export/reinstall/import restores Details immediately with changed Thunderb
|
|||
assert.equal((await onMessage({ type: 'sortana:getDetails', id: 99 })).results.length, 0);
|
||||
await transfer.importData(exported, ['settings', 'rules', 'cache']);
|
||||
assert.equal((await onMessage({ type: 'sortana:getCacheSize' })).count, 1);
|
||||
assert.equal(classifier.getConfig().systemOneProcessAll, false);
|
||||
assert.equal(classifier.getConfig().systemOneBatchSize, 3);
|
||||
const details = await onMessage({ type: 'sortana:getDetails', id: 99 });
|
||||
assert.equal(details.results.length, 1);
|
||||
assert.equal(details.results[0].criterion, 'invoice');
|
||||
|
|
|
|||
|
|
@ -39,10 +39,13 @@ function runner({ fail = false, changeSettings = false } = {}) {
|
|||
AiClassifier: {
|
||||
getConfig() { return structuredClone(currentConfig); },
|
||||
async buildCacheKey(id, criterion) { return `${id}:${criterion}`; },
|
||||
async classify(text, criterion, options) {
|
||||
evaluated.push({ text, criterion, ...options });
|
||||
if (fail && options.cacheKey.startsWith('1:')) throw new Error('inference failed');
|
||||
return { matched: true };
|
||||
async *classifyRules(text, candidates, snapshot) {
|
||||
for (const { rule, cacheKey } of candidates) {
|
||||
evaluated.push({ text, criterion: rule.criterion, cacheKey, snapshot,
|
||||
threshold: rule.decisionThreshold ?? snapshot.decisionThreshold });
|
||||
if (fail && cacheKey.startsWith('1:')) throw new Error('inference failed');
|
||||
yield { rule, result: { matched: true } };
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in a new issue