diff --git a/AGENTS.md b/AGENTS.md index 673e422..6a3d1e8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,8 +48,10 @@ sent only for Chat Completions. Chat Completions sends system/user messages and extracts the last JSON object from `choices[0].message.content`, requiring a `match` or `matched` boolean. SystemOne -sends prepared email text as `state` and a single `questions.match` of type `noul`. -It requires `answers.match.type === "noul"` and a finite numeric `noul` in `[0, 1]`. +sends prepared email text as `state` with `noul` questions identified by stable +`rule_` IDs. The single-question API still uses `match` for backward +compatibility and cache fingerprinting. Every requested answer must have type +`noul` and a finite numeric probability in `[0, 1]`; response order is irrelevant. No chat generation parameters or system prompt are sent to SystemOne. The global `decisionThreshold` defaults to 0.5; optional rule-level @@ -67,6 +69,27 @@ error, and continues the next queued email. Snapshot settings and rules per emai `inputCharacterLimit` replaces coupling input truncation to output `max_tokens`; it applies with aggressive reduction and migrates from the old effective limit. +### SystemOne batching + +`systemOneProcessAll` defaults to true; `systemOneBatchSize` defaults to 8 and must +be a positive safe integer. Only SystemOne shows these controls. The numeric input +is disabled while Process all is enabled, retaining its value. Include both in +settings import/export, startup loading, live updates, and per-email snapshots. + +`classifyRules()` is a lazy ordered async iterator over precomputed candidate keys. +Chat requests stay sequential. SystemOne batches the next N unique uncached keys +(or all), sends state once, validates the whole response, and persists one batch +of evidence before yielding ordered results. Per-rule thresholds remain independent. +Do not change existing cache keys or include batch settings in their fingerprint. +Prepare eligibility and keys before actions, using the existing initial mail metadata. + +Stop-after-match ends action iteration and prevents further requests; already +computed later answers remain cached. A malformed answer fails its entire batch: +no partial cache writes or new actions. Keep cache-generation guards for imports +and clearing during inference. No automatic splitting or retries. Tests in +`tests/batching.test.mjs` cover batching, cache reuse, failure atomicity, and actual +rule-runner stop behavior with the real classifier. + ## Documentation Additional documentation exists outside this repository. diff --git a/README.md b/README.md index b7fb0f5..999d826 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ llama-server -hf ggml-org/Laya-GGUF for a single loaded model. Router deployments require the appropriate model ID. 2. Set the default match probability threshold (initially **50%**). Each rule has an optional percentage override; leave it blank to inherit the default. -3. Apply rules normally. One `noul` question is sent per evaluated rule. A rule +3. Apply rules normally. Each uncached criterion becomes a `noul` question. A rule matches when the unrounded probability is **greater than or equal to** its effective threshold. For example, 80% matches an 80% threshold. 4. Open Details to see a probability bar with the percentage inside. Hover over @@ -83,6 +83,24 @@ llama-server -hf ggml-org/Laya-GGUF not automatically rerun actions. Decision models provide no textual explanation; no second model is called to invent one. Use Chat Completions for explanations. +**SystemOne batching:** **Process all** is enabled by default and sends all eligible, +uncached criteria for an email in one request, with the email state included once. +Turn it off to set **Rules per request** (initially 8); use 1 for sequential +requests. The number is retained while disabled. These controls apply only to +SystemOne; Chat Completions continues processing one rule per request. + +Cached rules consume no batch capacity, and identical criteria share a question +while retaining their individual thresholds. Actions execute in rule order and +honor stop-after-match. A batch can evaluate later rules whose actions are then +skipped; their probabilities remain available in Details. No further batches are +requested after stopping. Changing batch settings preserves existing cached results. + +If any requested answer is missing or invalid, the entire batch fails without +caching its new answers or executing their actions. Previously completed actions +remain applied. Sortana does not automatically retry or split rejected batches; +if the server rejects a large request, disable Process all and reduce Rules per +request. Limits and performance depend on the server and model. + Probabilities are model estimates, not guaranteed accuracy. Evaluate thresholds against representative labeled emails, including long messages and attachments. SystemOne uses the existing text extraction and attachment summaries; it does not diff --git a/background.js b/background.js index 1b9f554..602fe1f 100644 --- a/background.js +++ b/background.js @@ -347,7 +347,8 @@ async function processMessage(id) { identityId = null; } - for (const rule of rules) { + const candidates = []; + for (const [index, rule] of rules.entries()) { if (rule.enabled === false) { continue; } @@ -375,7 +376,9 @@ async function processMessage(id) { } } const cacheKey = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot); - const { matched } = await AiClassifier.classify(text, rule.criterion, { cacheKey, snapshot, threshold: rule.decisionThreshold ?? snapshot.decisionThreshold }); + candidates.push({ id: `rule_${index}`, rule, cacheKey }); + } + for await (const { rule, result: { matched } } of AiClassifier.classifyRules(text, candidates, snapshot)) { if (matched) { for (const act of (rule.actions || [])) { if (act.type === 'tag' && act.tagKey) { @@ -476,7 +479,7 @@ async function clearCacheForMessages(idsInput) { } try { - const store = await storage.local.get(["apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]); + const store = await storage.local.get(["systemOneProcessAll", "systemOneBatchSize", "apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]); logger.setDebug(store.debugLogging); const { migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js')); if (store.inputCharacterLimit === undefined) { diff --git a/manifest.json b/manifest.json index e9a3088..b304bc0 100644 --- a/manifest.json +++ b/manifest.json @@ -1,7 +1,7 @@ { "manifest_version": 2, "name": "Sortana", - "version": "4.0.0", + "version": "4.1.0", "default_locale": "en-US", "applications": { "gecko": { diff --git a/modules/AiClassifier.js b/modules/AiClassifier.js index 51951e6..603816f 100644 --- a/modules/AiClassifier.js +++ b/modules/AiClassifier.js @@ -97,14 +97,7 @@ export function getCachedDetails(key, threshold = config.decisionThreshold) { } export function getCachedResult(key, threshold) { return getCachedDetails(key, threshold)?.matched ?? null; } export function getReason(key) { return cache.get(key)?.reason ?? null; } -export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) { - await init(); - if (snapshot.apiType === 'systemone') validateThreshold(threshold); - const cached = getCachedDetails(cacheKey, threshold); - if (cached) return cached; - const epoch = generation; - const adapter = adapters[snapshot.apiType]; - const payload = adapter.buildRequest(text, criterion, snapshot); +async function sendRequest(payload, snapshot) { try { await storage.local.set({ lastPayload: payload }); } catch (error) { aiLog('Failed to save last payload', { level: 'warn' }, error); } const headers = { 'Content-Type': 'application/json' }; @@ -126,7 +119,18 @@ export async function classify(text, criterion, { cacheKey = null, snapshot = ge ? ' Verify that the running server build supports /v1/systemone and the selected decision model.' : ''; throw new Error(`Classification request failed: HTTP ${response.status} at ${endpoint}.${detail}${hint}`); } - const raw = await response.json(); + return response.json(); +} + +export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) { + await init(); + if (snapshot.apiType === 'systemone') validateThreshold(threshold); + const cached = getCachedDetails(cacheKey, threshold); + if (cached) return cached; + const epoch = generation; + const adapter = adapters[snapshot.apiType]; + const payload = adapter.buildRequest(text, criterion, snapshot); + const raw = await sendRequest(payload, snapshot); const entry = { ...adapter.parseResponse(raw), backend: snapshot.apiType, model: typeof raw.model === 'string' ? raw.model : snapshot.model, version: 2, messageHash: cacheKey?.split(':')[1] }; @@ -136,6 +140,60 @@ export async function classify(text, criterion, { cacheKey = null, snapshot = ge } return withPolicy(entry, threshold); } +// Yield lazily so the action runner can stop without starting the next request. +// Candidates have precomputed keys, allowing actions to move/delete the message. +export async function* classifyRules(text, candidates, snapshot = getConfig()) { + await init(); + validateSettings(snapshot); + if (snapshot.apiType !== 'systemone') { + for (const candidate of candidates) { + const result = await classify(text, candidate.rule.criterion, { + cacheKey: candidate.cacheKey, snapshot, + threshold: candidate.rule.decisionThreshold ?? snapshot.decisionThreshold + }); + yield { rule: candidate.rule, result }; + } + return; + } + const limit = snapshot.systemOneProcessAll ? Infinity : snapshot.systemOneBatchSize; + // Keep received evidence locally even if persistence is cleared during this run. + const evidence = new Map(); + for (const candidate of candidates) { + validateThreshold(candidate.rule.decisionThreshold ?? snapshot.decisionThreshold); + const entry = cache.get(candidate.cacheKey); + if (entry) evidence.set(candidate.cacheKey, entry); + } + for (let index = 0; index < candidates.length; index++) { + const candidate = candidates[index]; + if (!evidence.has(candidate.cacheKey)) { + const pending = new Map(); + for (let next = index; next < candidates.length && pending.size < limit; next++) { + const item = candidates[next]; + if (!evidence.has(item.cacheKey) && !pending.has(item.cacheKey)) { + pending.set(item.cacheKey, { ...item, criterion: item.rule.criterion }); + } + } + const questions = [...pending.values()]; + const epoch = generation; + const raw = await sendRequest(systemone.buildBatchRequest(text, questions, snapshot), snapshot); + // Parse the complete batch before accepting any of its evidence. + const parsed = systemone.parseBatchResponse(raw, questions); + const entries = questions.map((question, i) => [question.cacheKey, { + ...parsed[i], backend: snapshot.apiType, version: 2, + model: typeof raw.model === 'string' ? raw.model : snapshot.model, + messageHash: question.cacheKey.split(':')[1] + }]); + for (const [key, entry] of entries) evidence.set(key, entry); + if (epoch === generation) { + for (const [key, entry] of entries) cache.set(key, entry); + await saveCache(); + } + } + yield { rule: candidate.rule, result: withPolicy(evidence.get(candidate.cacheKey), + candidate.rule.decisionThreshold ?? snapshot.decisionThreshold) }; + } +} + export async function classifyText(text, criterion, cacheKey = null) { return (await classify(text, criterion, { cacheKey })).matched; } export async function removeCacheEntries(keys = []) { await init(); generation++; diff --git a/modules/backends/systemOne.js b/modules/backends/systemOne.js index 0de2472..70627f1 100644 --- a/modules/backends/systemOne.js +++ b/modules/backends/systemOne.js @@ -1,17 +1,22 @@ export const path = '/v1/systemone'; export function buildRequest(text, criterion, config) { - const payload = { - state: text, - questions: { match: { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` } } - }; + return buildBatchRequest(text, [{ id: 'match', criterion }], config); +} +export function buildBatchRequest(text, questions, config) { + const payload = { state: text, questions: Object.fromEntries(questions.map(({ id, criterion }) => + [id, { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` }])) }; if (config.model) payload.model = config.model; return payload; } -export function parseResponse(result) { - const answer = result?.answers?.match; +export function parseResponse(result) { return parseAnswer(result, 'match'); } +export function parseBatchResponse(result, questions) { + return questions.map(({ id }) => parseAnswer(result, id)); +} +function parseAnswer(result, id) { + const answer = result?.answers?.[id]; if (answer?.type !== 'noul' || typeof answer.noul !== 'number' || !Number.isFinite(answer.noul) || answer.noul < 0 || answer.noul > 1) { - throw new Error('SystemOne response must contain answers.match with type noul and a probability from 0 to 1.'); + throw new Error(`SystemOne response must contain answers.${id} with type noul and a probability from 0 to 1.`); } return { matched: null, reason: null, probability: answer.noul }; } diff --git a/modules/classificationConfig.js b/modules/classificationConfig.js index 3abe7cc..d6dbb5a 100644 --- a/modules/classificationConfig.js +++ b/modules/classificationConfig.js @@ -2,6 +2,7 @@ import { DEFAULT_AI_PARAMS } from './defaultParams.js'; export const CONFIG_DEFAULTS = { apiType: 'chat-completions', endpoint: 'http://127.0.0.1:5000', model: '', apiKey: '', openaiOrganization: '', openaiProject: '', customSystemPrompt: '', + systemOneProcessAll: true, systemOneBatchSize: 8, decisionThreshold: 0.5, inputCharacterLimit: 3686, aiParams: DEFAULT_AI_PARAMS, htmlToMarkdown: false, stripUrlParams: false, altTextImages: false, collapseWhitespace: false, tokenReduction: false, debugLogging: false @@ -13,6 +14,12 @@ export function validateThreshold(value) { return value; } export function validateSettings(data) { + if (data.systemOneProcessAll !== undefined && typeof data.systemOneProcessAll !== 'boolean') { + throw new Error('Process all must be a boolean.'); + } + if (data.systemOneBatchSize !== undefined && (!Number.isSafeInteger(data.systemOneBatchSize) || data.systemOneBatchSize < 1)) { + throw new Error('Rules per request must be a positive whole number.'); + } if (data.apiType !== undefined && !['chat-completions', 'systemone'].includes(data.apiType)) throw new Error('Unknown API type.'); if (data.decisionThreshold !== undefined) validateThreshold(data.decisionThreshold); if (data.inputCharacterLimit !== undefined && (!Number.isSafeInteger(data.inputCharacterLimit) || data.inputCharacterLimit < 1)) { diff --git a/options/dataTransfer.js b/options/dataTransfer.js index ef2b237..298974c 100644 --- a/options/dataTransfer.js +++ b/options/dataTransfer.js @@ -4,6 +4,7 @@ import { validCacheEntry } from '../modules/AiClassifier.js'; const storage = (globalThis.messenger ?? browser).storage; const KEY_GROUPS = { settings: [ + 'systemOneProcessAll', 'systemOneBatchSize', 'apiType', 'decisionThreshold', 'inputCharacterLimit', 'tokenReduction', 'endpoint', 'model', diff --git a/options/options.html b/options/options.html index 8091df5..2ab0c8a 100644 --- a/options/options.html +++ b/options/options.html @@ -78,6 +78,15 @@ +
+ +

Group all eligible, uncached rules for each email into one request. Actions still follow rule order and stop-after-match.

+
+
+ + +

Turn off Process all to limit each request. Use 1 for one rule at a time.

+
diff --git a/options/options.js b/options/options.js index d1f71a2..124d163 100644 --- a/options/options.js +++ b/options/options.js @@ -7,6 +7,7 @@ document.addEventListener('DOMContentLoaded', async () => { const { DEFAULT_AI_PARAMS } = await import(browser.runtime.getURL('modules/defaultParams.js')); const { validateSettings, migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js')); const defaults = await storage.local.get([ + 'systemOneProcessAll', 'systemOneBatchSize', 'apiType', 'decisionThreshold', 'inputCharacterLimit', 'endpoint', 'customSystemPrompt', @@ -90,6 +91,15 @@ document.addEventListener('DOMContentLoaded', async () => { }); const apiTypeInput = document.getElementById('api-type'); apiTypeInput.value = defaults.apiType || 'chat-completions'; + const processAllInput = document.getElementById('systemone-process-all'); + const batchSizeInput = document.getElementById('systemone-batch-size'); + processAllInput.checked = defaults.systemOneProcessAll ?? true; + let savedBatchSize = defaults.systemOneBatchSize ?? 8; + batchSizeInput.value = savedBatchSize; + function updateBatchControls() { + batchSizeInput.disabled = processAllInput.checked || apiTypeInput.value !== 'systemone'; + } + processAllInput.addEventListener('change', updateBatchControls); const thresholdInput = document.getElementById('decision-threshold'); thresholdInput.value = (defaults.decisionThreshold ?? 0.5) * 100; const inputLimit = document.getElementById('input-character-limit'); @@ -97,6 +107,7 @@ document.addEventListener('DOMContentLoaded', async () => { const settingsError = document.getElementById('settings-error'); function updateApiControls() { const decision = apiTypeInput.value === 'systemone'; + updateBatchControls(); document.querySelectorAll('.systemone-only').forEach(el => el.classList.toggle('is-hidden', !decision)); for (const id of ['system-instructions', 'openai-organization', 'openai-project', ...Object.keys(DEFAULT_AI_PARAMS)]) { document.getElementById(id)?.closest('.field')?.classList.toggle('is-hidden', decision); @@ -1097,6 +1108,14 @@ document.addEventListener('DOMContentLoaded', async () => { document.getElementById('save').addEventListener('click', async () => { settingsError.textContent = ''; const apiType = apiTypeInput.value; + const systemOneProcessAll = processAllInput.checked; + const enteredBatchSize = Number(batchSizeInput.value); + const validBatchSize = Number.isSafeInteger(enteredBatchSize) && enteredBatchSize > 0; + if (!batchSizeInput.disabled && !validBatchSize) { + settingsError.textContent = 'Rules per request must be a positive whole number.'; + return; + } + const systemOneBatchSize = validBatchSize ? enteredBatchSize : savedBatchSize; const decisionThreshold = thresholdInput.value.trim() === '' ? NaN : Number(thresholdInput.value) / 100; const inputCharacterLimit = Number(inputLimit.value); const endpoint = endpointInput.value.trim(); @@ -1127,13 +1146,15 @@ document.addEventListener('DOMContentLoaded', async () => { const tokenReduction = tokenReductionToggle.checked; const showDebugTab = debugTabToggle.checked; const theme = themeSelect.value; - try { validateSettings({ apiType, decisionThreshold, inputCharacterLimit, aiRules: rules }); } + try { validateSettings({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, aiRules: rules }); } catch (error) { settingsError.textContent = error.message; return; } - await storage.local.set({ apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, aiRules: rules, theme, showDebugTab }); + await storage.local.set({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, aiRules: rules, theme, showDebugTab }); + savedBatchSize = systemOneBatchSize; + batchSizeInput.value = savedBatchSize; await storage.local.remove(['templateName', 'customTemplate']); await applyTheme(theme); try { - await AiClassifier.setConfig({ apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging }); + await AiClassifier.setConfig({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging }); logger.setDebug(debugLogging); } catch (e) { logger.aiLog('[options] failed to apply config', {level: 'error'}, e); diff --git a/tests/batching.test.mjs b/tests/batching.test.mjs new file mode 100644 index 0000000..000ebe3 --- /dev/null +++ b/tests/batching.test.mjs @@ -0,0 +1,219 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import vm from 'node:vm'; +import { readFile } from 'node:fs/promises'; +import { CONFIG_DEFAULTS, validateSettings } from '../modules/classificationConfig.js'; +import * as systemone from '../modules/backends/systemOne.js'; + +const data = {}; +let cacheWrites = 0; +globalThis.messenger = { storage: { local: { + async get(keys) { return Object.fromEntries((Array.isArray(keys) ? keys : [keys]).map(key => [key, data[key]])); }, + async set(values) { Object.assign(data, structuredClone(values)); if ('aiCache' in values) cacheWrites++; }, + async remove() {} +} }, messages: { async get(id) { return { headerMessageId: `mail-${id}` }; } } }; +const classifier = await import('../modules/AiClassifier.js'); +const transfer = await import('../options/dataTransfer.js'); +let requests = []; +async function setup(update = {}) { + await classifier.clearCache(); + classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone', ...update }); + requests = []; cacheWrites = 0; + globalThis.fetch = async (url, request) => { + const body = JSON.parse(request.body); + requests.push({ url, body, headers: request.headers }); + // Intentionally reverse response order and include an unsolicited answer. + return { ok: true, json: async () => ({ model: 'served', answers: { + unsolicited: { type: 'noul', noul: 1 }, + ...Object.fromEntries(Object.keys(body.questions).reverse().map(id => [id, { type: 'noul', noul: 0.75 }])) + } }) }; + }; + return classifier.getConfig(); +} +async function candidates(count, snapshot, criteria = []) { + return Promise.all(Array.from({ length: count }, async (_, i) => { + const criterion = criteria[i] ?? `criterion ${i}`; + return { id: `rule_${i}`, rule: { criterion, actions: [] }, cacheKey: await classifier.buildCacheKey(1, criterion, snapshot) }; + })); +} +async function collect(items, snapshot) { + const results = []; + for await (const value of classifier.classifyRules('email state', items, snapshot)) results.push(value); + return results; +} + +test('Process all, explicit batch sizes, partial batches, empty input and cache reuse', async () => { + for (const [settings, expected] of [ + [{}, [10]], [{ systemOneProcessAll: false, systemOneBatchSize: 8 }, [8, 2]], + [{ systemOneProcessAll: false, systemOneBatchSize: 1 }, Array(10).fill(1)] + ]) { + const snapshot = await setup(settings); + const items = await candidates(10, snapshot); + const results = await collect(items, snapshot); + assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), expected); + assert.equal(cacheWrites, expected.length); + assert.deepEqual(results.map(r => r.rule.criterion), items.map(r => r.rule.criterion)); + assert.ok(results.every(r => r.result.probability === 0.75)); + assert.ok(requests.every(r => r.body.state === 'email state' && !('messages' in r.body))); + await collect(items, snapshot); + assert.equal(requests.length, expected.length); + await collect([], snapshot); + assert.equal(requests.length, expected.length); + } +}); + +test('cached rules consume no capacity and duplicate criteria use independent thresholds', async () => { + const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 }); + const items = await candidates(5, snapshot, ['cached', 'same', 'cached', 'other', 'same']); + await collect([items[0]], snapshot); + requests = []; cacheWrites = 0; + items[1].rule.decisionThreshold = 0.8; + items[4].rule.decisionThreshold = 0.7; + const results = await collect(items, snapshot); + assert.equal(requests.length, 1); + assert.deepEqual(Object.keys(requests[0].body.questions), ['rule_1', 'rule_3']); + assert.equal(results[1].result.matched, false); + assert.equal(results[4].result.matched, true); + assert.equal(cacheWrites, 1); + assert.equal(await classifier.buildCacheKey(1, 'same', { ...snapshot, systemOneProcessAll: true, systemOneBatchSize: 100 }), items[1].cacheKey); +}); + +test('answer identity determines probability, not answer order', () => { + const result = { answers: { rule_2: { type: 'noul', noul: 0 }, rule_1: { type: 'noul', noul: 1 } } }; + assert.deepEqual(systemone.parseBatchResponse(result, [{ id: 'rule_1' }, { id: 'rule_2' }]).map(r => r.probability), [1, 0]); +}); + +test('failed batches neither yield nor cache partial results and do not retry', async () => { + for (const failure of ['missing', 'invalid', 'http']) { + const snapshot = await setup(); + const items = await candidates(2, snapshot); + let calls = 0, yielded = 0; + globalThis.fetch = async () => { + calls++; + return { ok: failure !== 'http', status: 413, json: async () => ({ answers: { + rule_0: { type: 'noul', noul: 0.9 }, + ...(failure === 'invalid' ? { rule_1: { type: 'noul', noul: '0.9' } } : {}) + } }) }; + }; + await assert.rejects(async () => { + for await (const value of classifier.classifyRules('state', items, snapshot)) yielded++; + }); + assert.equal(yielded, 0); + assert.equal(calls, 1); + assert.equal(cacheWrites, 0); + assert.equal(await classifier.getCacheSize(), 0); + } +}); + +test('clearing or importing during a batch prevents stale cache writes', async () => { + for (const replace of [() => classifier.clearCache(), () => classifier.replaceCache({})]) { + const snapshot = await setup(); + const items = await candidates(2, snapshot); + const fetch = globalThis.fetch; + globalThis.fetch = async (...args) => { await replace(); return fetch(...args); }; + const results = await collect(items, snapshot); + assert.equal(results.length, 2); + assert.equal(requests.length, 1); + assert.equal(await classifier.getCacheSize(), 0); + } +}); + +test('Chat Completions remains lazy and sequential', async () => { + const snapshot = await setup({ apiType: 'chat-completions' }); + const items = await candidates(3, snapshot); + globalThis.fetch = async (url, request) => { + requests.push(JSON.parse(request.body)); + return { ok: true, json: async () => ({ choices: [{ message: { content: '{"match":true}' } }] }) }; + }; + for await (const value of classifier.classifyRules('state', items, snapshot)) break; + assert.equal(requests.length, 1); + assert.ok(Array.isArray(requests[0].messages)); +}); + +test('batch settings validate and survive import', async () => { + for (const value of [0, -1, 1.2, Infinity, '8', null]) assert.throws(() => validateSettings({ systemOneBatchSize: value })); + assert.throws(() => validateSettings({ systemOneProcessAll: 'true' })); + await transfer.importData({ text: async () => JSON.stringify({ systemOneProcessAll: false, systemOneBatchSize: 3 }) }, ['settings']); + assert.equal(data.systemOneProcessAll, false); + assert.equal(data.systemOneBatchSize, 3); + await assert.rejects(transfer.importData({ text: async () => '{"systemOneBatchSize":0}' }, ['settings'])); +}); + +// Exercise the action runner with the real lazy classifier and mocked mail APIs. +const source = await readFile(new URL('../background.js', import.meta.url), 'utf8'); +const functions = source.slice(source.indexOf('async function processMessage('), source.indexOf('async function clearCacheForMessages(')); +test('stop after match skips later actions and batches; all-mode retains later evidence', async () => { + for (const processAll of [true, false]) { + const snapshot = await setup({ systemOneProcessAll: processAll, systemOneBatchSize: 2 }); + const actions = [], errors = []; + const rules = Array.from({ length: 5 }, (_, i) => ({ criterion: `rule ${i}`, actions: [{ type: 'flag', flagged: true }], stopProcessing: i === 1 })); + const context = vm.createContext({ + structuredClone, Date, Error, aiRules: rules, AiClassifier: classifier, + queue: Promise.resolve(), queuedCount: 0, processing: false, currentStart: 0, + tokenReduction: false, showDebugTab: false, timingStats: {}, + updateActionIcon() {}, updateTimingStats() {}, showTransientIcon() {}, + buildEmailText() { return 'state'; }, ICONS: { circle: 'circle' }, + logger: { aiLog() {} }, recordError(...args) { errors.push(args); }, ERROR_NOTIFICATION_ID: 'error', + storage: messenger.storage, + messenger: { messages: { + async getFull() { return {}; }, async get() { return { tags: [], folder: { accountId: 'account' } }; }, + async update(id, value) { actions.push({ id, value }); } + }, identities: { async list() { return []; } } }, + browser: { runtime: { getURL: x => x }, notifications: { async create() {} } } + }); + vm.runInContext(functions, context); + await vm.runInContext('applyAiRules([1])', context); + assert.equal(errors.length, 0); + assert.equal(actions.length, 2); + assert.equal(requests.length, 1); + assert.equal(Object.keys(requests[0].body.questions).length, processAll ? 5 : 2); + assert.equal(await classifier.getCacheSize(), processAll ? 5 : 2); + } +}); + +test('batch controls default to all and retain numeric input across toggles and API switches', async () => { + const optionsSource = await readFile(new URL('../options/options.js', import.meta.url), 'utf8'); + const section = optionsSource.slice(optionsSource.indexOf(' const apiTypeInput ='), optionsSource.indexOf(' const endpointInput =')); + const fields = new Map(); + const field = id => { + if (!fields.has(id)) fields.set(id, { + value: '', checked: false, disabled: false, listeners: {}, + addEventListener(event, fn) { this.listeners[event] = fn; }, + classList: { toggle() {} }, closest() { return this; } + }); + return fields.get(id); + }; + const context = vm.createContext({ + defaults: { apiType: 'systemone' }, DEFAULT_AI_PARAMS: {}, migratedInputLimit() { return 3686; }, + updateEndpointPreview() {}, document: { getElementById: field, querySelectorAll() { return []; } } + }); + vm.runInContext(`${section}\nupdateApiControls();`, context); + const all = field('systemone-process-all'), size = field('systemone-batch-size'), api = field('api-type'); + assert.equal(all.checked, true); + assert.equal(size.value, 8); + assert.equal(size.disabled, true); + all.checked = false; all.listeners.change(); + assert.equal(size.disabled, false); + size.value = '3'; + all.checked = true; all.listeners.change(); + assert.equal(size.disabled, true); + all.checked = false; all.listeners.change(); + assert.equal(size.value, '3'); + api.value = 'chat-completions'; api.listeners.change(); + assert.equal(size.disabled, true); + api.value = 'systemone'; api.listeners.change(); + assert.equal(size.disabled, false); + assert.equal(size.value, '3'); +}); + +test('batch settings remain fixed while a per-email snapshot is running', async () => { + const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 }); + const items = await candidates(5, snapshot); + const fetch = globalThis.fetch; + globalThis.fetch = async (...args) => { + classifier.setConfig({ systemOneProcessAll: true, systemOneBatchSize: 1 }); + return fetch(...args); + }; + await collect(items, snapshot); + assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), [2, 2, 1]); +}); diff --git a/tests/cacheRestore.test.mjs b/tests/cacheRestore.test.mjs index f5ca74d..91a8515 100644 --- a/tests/cacheRestore.test.mjs +++ b/tests/cacheRestore.test.mjs @@ -51,7 +51,7 @@ const messageStart = source.indexOf('browser.runtime.onMessage.addListener('); vm.runInContext(source.slice(messageStart, source.indexOf('\n\n // Automatically classify', messageStart)), context); test('export/reinstall/import restores Details immediately with changed Thunderbird IDs', async () => { - const settings = { ...CONFIG_DEFAULTS, apiType: 'systemone', endpoint: 'http://decision-server:8080', tokenReduction: true, inputCharacterLimit: 5000 }; + const settings = { ...CONFIG_DEFAULTS, systemOneProcessAll: false, systemOneBatchSize: 3, apiType: 'systemone', endpoint: 'http://decision-server:8080', tokenReduction: true, inputCharacterLimit: 5000 }; const rules = [{ criterion: 'invoice', decisionThreshold: 0.8, actions: [] }]; await storage.local.set({ ...settings, aiRules: rules }); globalThis.fetch = async () => ({ ok: true, json: async () => ({ answers: { match: { type: 'noul', noul: 0.9 } } }) }); @@ -73,6 +73,8 @@ test('export/reinstall/import restores Details immediately with changed Thunderb assert.equal((await onMessage({ type: 'sortana:getDetails', id: 99 })).results.length, 0); await transfer.importData(exported, ['settings', 'rules', 'cache']); assert.equal((await onMessage({ type: 'sortana:getCacheSize' })).count, 1); + assert.equal(classifier.getConfig().systemOneProcessAll, false); + assert.equal(classifier.getConfig().systemOneBatchSize, 3); const details = await onMessage({ type: 'sortana:getDetails', id: 99 }); assert.equal(details.results.length, 1); assert.equal(details.results[0].criterion, 'invoice'); diff --git a/tests/ruleExecution.test.mjs b/tests/ruleExecution.test.mjs index 153a050..327a14d 100644 --- a/tests/ruleExecution.test.mjs +++ b/tests/ruleExecution.test.mjs @@ -39,10 +39,13 @@ function runner({ fail = false, changeSettings = false } = {}) { AiClassifier: { getConfig() { return structuredClone(currentConfig); }, async buildCacheKey(id, criterion) { return `${id}:${criterion}`; }, - async classify(text, criterion, options) { - evaluated.push({ text, criterion, ...options }); - if (fail && options.cacheKey.startsWith('1:')) throw new Error('inference failed'); - return { matched: true }; + async *classifyRules(text, candidates, snapshot) { + for (const { rule, cacheKey } of candidates) { + evaluated.push({ text, criterion: rule.criterion, cacheKey, snapshot, + threshold: rule.decisionThreshold ?? snapshot.decisionThreshold }); + if (fail && cacheKey.startsWith('1:')) throw new Error('inference failed'); + yield { rule, result: { matched: true } }; + } } } });