Compare commits

..
32 changed files with 602 additions and 4265 deletions

0
.codex
View file

View file

@ -8,12 +8,10 @@ This file provides guidelines for codex agents contributing to the Sortana proje
- `modules/`: Contains reusable JavaScript modules such as `AiClassifier.js`, - `modules/`: Contains reusable JavaScript modules such as `AiClassifier.js`,
`defaultParams.js` and `themeUtils.js`. `defaultParams.js` and `themeUtils.js`.
- `options/`: The options page HTML, JavaScript and bundled Bulma CSS (v1.0.3). - `options/`: The options page HTML, JavaScript and bundled Bulma CSS (v1.0.3).
- `details.html` and `details.js`: View classification evidence and clear cache for a message. - `details.html` and `details.js`: View AI reasoning and clear cache for a message.
- `resources/`: Images and other static files. - `resources/`: Images and other static files.
- `prompt_templates/`: Prompt template files for the AI service.
- `build-xpi.ps1`: PowerShell script to package the extension. - `build-xpi.ps1`: PowerShell script to package the extension.
- `build-xpi.sh`: Bash script to package the extension.
- `resources/svg2img.ps1`: PowerShell script to regenerate themed PNG icons from SVGs.
- `resources/svg2img.sh`: Bash script to regenerate themed PNG icons from SVGs.
## Coding Style ## Coding Style
@ -30,65 +28,7 @@ This file provides guidelines for codex agents contributing to the Sortana proje
## Testing ## Testing
Run the dependency-free suite with Node.js 22 or newer: `node --test tests/*.test.mjs`. There are currently no automated tests for this project. If you add tests in the future, specify the commands to run them here. For now, verification must happen manually in Thunderbird. Do **not** run the `ps1` build script or the SVG processing script.
The tests mock extension storage, HTTP responses, and Thunderbird rule execution.
Manual verification in Thunderbird is still required for UI and real-server integration.
Do **not** run the `ps1` build script or the SVG processing script.
## Endpoint Notes
Sortana supports `apiType: "chat-completions"` (default) and `"systemone"`.
Plain adapter modules under `modules/backends/` own request construction and response
validation; `AiClassifier` owns shared transport, configuration snapshots, threshold
policy, and caching. The stored endpoint is a base URL. Normalization accepts `/v1`
and recognized full API paths, preserves proxy prefixes, and defaults to HTTPS
when no scheme is supplied. Model discovery uses `/v1/models`; **None** omits `model`.
Bearer authentication is optional for both APIs. Organization/project headers are
sent only for Chat Completions.
Chat Completions sends system/user messages and extracts the last JSON object from
`choices[0].message.content`, requiring a `match` or `matched` boolean. SystemOne
sends prepared email text as `state` with `noul` questions identified by stable
`rule_<snapshot index>` IDs. The single-question API still uses `match` for backward
compatibility and cache fingerprinting. Every requested answer must have type
`noul` and a finite numeric probability in `[0, 1]`; response order is irrelevant.
No chat generation parameters or system prompt are sent to SystemOne.
The global `decisionThreshold` defaults to 0.5; optional rule-level
`decisionThreshold` overrides it. UI percentages are stored as numbers in `[0, 1]`.
Matching uses `probability >= threshold` before display rounding. Details displays
probabilities in a progress bar with the percentage inside; threshold and model
appear in its tooltip. Results use the current threshold, never fabricated reasoning or action
history. No secondary model calls are made.
`classify()` returns structured evidence; `classifyText()` remains a boolean wrapper.
Failures throw and are never cached. HTTP errors include the request URL and
JSON server error message when available; SystemOne 404 errors suggest checking
the running server build and decision-model support. The rule runner stops that email, reports the
error, and continues the next queued email. Snapshot settings and rules per email.
`inputCharacterLimit` replaces coupling input truncation to output `max_tokens`;
it applies with aggressive reduction and migrates from the old effective limit.
### SystemOne batching
`systemOneProcessAll` defaults to true; `systemOneBatchSize` defaults to 8 and must
be a positive safe integer. Only SystemOne shows these controls. The numeric input
is disabled while Process all is enabled, retaining its value. Include both in
settings import/export, startup loading, live updates, and per-email snapshots.
`classifyRules()` is a lazy ordered async iterator over precomputed candidate keys.
Chat requests stay sequential. SystemOne batches the next N unique uncached keys
(or all), sends state once, validates the whole response, and persists one batch
of evidence before yielding ordered results. Per-rule thresholds remain independent.
Do not change existing cache keys or include batch settings in their fingerprint.
Prepare eligibility and keys before actions, using the existing initial mail metadata.
Stop-after-match ends action iteration and prevents further requests; already
computed later answers remain cached. A malformed answer fails its entire batch:
no partial cache writes or new actions. Keep cache-generation guards for imports
and clearing during inference. No automatic splitting or retries. Tests in
`tests/batching.test.mjs` cover batching, cache reuse, failure atomicity, and actual
rule-runner stop behavior with the real classifier.
## Documentation ## Documentation
@ -122,26 +62,15 @@ text extracted from all text parts.
### Cache Strategy ### Cache Strategy
`aiCache` persists version 2 inference evidence. Keys are `aiCache` persists classification results. Each key is the SHA‑256 hex of
`v2:<SHA-256 Message-ID>:<SHA-256 stable configuration/criterion fingerprint>`. `"<message Message-ID>|<criterion>"` and maps to an object with `matched` and `reason`
Include backend, endpoint, model, effective request parameters and preprocessing; properties. Any legacy `aiReasonCache` data is merged into `aiCache` the first
exclude secrets, actions and thresholds. Recompute SystemOne matches from cached time the add-on loads after an update.
probability and current thresholds without automatically executing actions.
Entries retain a hashed message identifier for clearing across configurations.
Version 4.0.0 intentionally breaks legacy cache compatibility: legacy caches
(including `aiReasonCache`) are invalidated because their inference
configuration is unknown. Import/export validates versioned evidence; storage
changes refresh the running cache. Maintenance uses `sortana:getCacheSize` to
count validated entries in the background classifier, rather than raw storage keys.
Full restoration requires matching settings and rules as well as cache evidence;
`tests/cacheRestore.test.mjs` exercises export/reinstall/import through the actual
storage and Details handlers, including changed numeric Thunderbird message IDs. Clearing/importing during inference prevents
that in-flight result from repopulating the replaced cache.
### Icon Set Usage ### Icon Set Usage
Toolbar and menu icons reside under `resources/img` and are provided in 16, 32 Toolbar and menu icons reside under `resources/img` and are provided in 16, 32
and 64 pixel variants. When changing these icons, pass a dictionary mapping the and 64 pixel variants. When changing these icons, pass a dictionary mapping the
sizes to the paths in `browserAction.setIcon` or `messageDisplayAction.setIcon`. sizes to the paths in `browserAction.setIcon` or `messageDisplayAction.setIcon`.
Use `resources/svg2img.ps1` on Windows or `resources/svg2img.sh` on Linux to Use `resources/svg2img.ps1` to regenerate PNGs from the SVG sources.
regenerate PNGs from the SVG sources.

157
README.md
View file

@ -4,145 +4,47 @@
Sortana is an experimental Thunderbird add-on that integrates an AI-powered filter rule. Sortana is an experimental Thunderbird add-on that integrates an AI-powered filter rule.
It allows you to classify email messages by sending their contents to a configurable It allows you to classify email messages by sending their contents to a configurable
HTTP endpoint. Choose **Chat Completions** for an OpenAI-compatible HTTP endpoint. The endpoint should respond with JSON indicating whether the
`/v1/chat/completions` service or **SystemOne** for decision models served through message meets a specified criterion.
`/v1/systemone`. Chat Completions returns a match boolean and textual explanation;
SystemOne returns a match probability, compared against a configurable threshold.
The options page stores a base URL and shows the resolved request URL.
## Features ## Features
- **Configurable endpoint** – set the classification service base URL on the options page. - **Configurable endpoint** – set the classification service URL on the options page.
- **Model selection** – load available models from the endpoint and choose one (or omit the model field). - **Prompt templates** – choose between several model formats or provide your own custom template.
- **Optional OpenAI auth headers** – provide an API key plus optional organization/project headers when needed. - **Custom system prompts** – tailor the instructions sent to the model for more precise results.
- **Custom system prompts** – tailor the instructions sent as the system message for more precise results.
- **Persistent result caching** – classification results and reasoning are saved to disk so messages aren't re-evaluated across restarts. - **Persistent result caching** – classification results and reasoning are saved to disk so messages aren't re-evaluated across restarts.
- **Advanced parameters (Chat Completions)** – tune generation settings like temperature, top‑p and more from the options page. - **Advanced parameters** – tune generation settings like temperature, top‑p and more from the options page.
Provider-specific advanced fields are forwarded as configured and may be rejected by strict OpenAI endpoints.
- **Markdown conversion** – optionally convert HTML bodies to Markdown before sending them to the AI service. - **Markdown conversion** – optionally convert HTML bodies to Markdown before sending them to the AI service.
- **Debug logging** – optional colorized logs help troubleshoot interactions with the AI service. - **Debug logging** – optional colorized logs help troubleshoot interactions with the AI service.
- **Debug tab** – view the last request payload and a diff between the unaltered message text and the final prompt.
- **Light/Dark themes** – automatically match Thunderbird's appearance with optional manual override. - **Light/Dark themes** – automatically match Thunderbird's appearance with optional manual override.
- **Automatic rules** – create rules that tag, move, copy, forward, reply, delete, archive, mark read/unread or flag/unflag messages based on AI classification. Rules can optionally apply only to unread messages and can ignore messages outside a chosen age range. - **Automatic rules** – create rules that tag, move, copy, forward, reply, delete, archive, mark read/unread or flag/unflag messages based on AI classification. Rules can optionally apply only to unread messages and can ignore messages outside a chosen age range.
- **Rule ordering** – drag rules to prioritize them and optionally stop processing after a match. - **Rule ordering** – drag rules to prioritize them and optionally stop processing after a match.
- **Rule enable/disable** – temporarily turn a rule off without removing it. - **Rule enable/disable** – temporarily turn a rule off without removing it.
- **Account & folder filters** – limit rules to specific accounts or folders. - **Account & folder filters** – limit rules to specific accounts or folders.
- **Context menu** – apply AI rules from the message list or the message display action button. - **Context menu** – apply AI rules from the message list or the message display action button.
- **Status icons** – toolbar icons show when classification is in progress and briefly display success states. If a failure occurs the icon turns red briefly before returning to normal. - **Status icons** – toolbar icons show when classification is in progress and briefly display success states. If a failure occurs the icon turns red until you dismiss the notification.
- **Error notification** – failed classification displays a notification in Thunderbird. - **Error notification** – failed classification displays a notification with a button to clear the error and reset the icon.
- **Session error log** – the Errors tab (visible only when errors occur) shows errors recorded since the last add-on start. - **View reasoning** – inspect why rules matched via the Details popup.
- **View results** – inspect chat explanations or SystemOne probabilities and thresholds in the Details popup.
- **Cache management** – clear cached results from the context menu or options page. - **Cache management** – clear cached results from the context menu or options page.
- **Queue & timing stats** – monitor processing time on the Maintenance tab. - **Queue & timing stats** – monitor processing time on the Maintenance tab.
- **Packaging scripts** – `build-xpi.ps1` (PowerShell) or `build-xpi.sh` (bash) build an XPI ready for installation. - **Packaging script** – `build-xpi.ps1` builds an XPI ready for installation.
- **Icon generation scripts** – `resources/svg2img.ps1` (PowerShell) or `resources/svg2img.sh` (bash) regenerate themed PNG icons from the SVG sources.
- **Maintenance tab** – view rule counts, cache entries and clear cached results from the options page. - **Maintenance tab** – view rule counts, cache entries and clear cached results from the options page.
### Cache Storage ### Cache Storage
Classification results are stored under `aiCache` using version 2 entries. Keys Classification results are stored under the `aiCache` key in extension storage.
combine a hashed Message-ID with a fingerprint of the criterion, backend, Each entry maps a SHA‑256 hash of `"<message Message-ID>|<criterion>"` to an object
endpoint, selected model, effective request settings, and preprocessing settings. containing `matched` and `reason` fields. Older installations with a separate
Credentials and thresholds are excluded. SystemOne probabilities are retained at `aiReasonCache` will be migrated automatically on startup.
full precision; changing a threshold reuses the evidence without another request.
Changing inference settings requires a fresh classification. If the model behind
an unchanged server/model alias changes, clear the cache manually.
**Version 4.0.0 upgrade note:** legacy cache entries and `aiReasonCache` are discarded because
they do not identify the inference configuration. Settings and rules are retained.
Imported legacy caches are also discarded. To restore cached Details after reinstalling,
export and import **Settings, Rules, and Cache** together: lookups require the
original criterion and matching inference settings. Import refreshes the running
classifier immediately; Maintenance counts its validated, loaded entries. A cache-only
import will not recreate rules or settings. Clearing a message's cache removes its
entries across rules and configurations.
### SystemOne with llama-server
Use a llama.cpp build supporting `/v1/systemone` and a compatible decision GGUF;
older builds and ordinary chat models cannot serve this API. A 404 with
`File Not Found` from `/v1/systemone` indicates the route is unavailable: check
the running server build and any reverse-proxy routing. Loading a decision GGUF
alone does not add this route to an older server. For example:
```sh
llama-server -hf ggml-org/Laya-GGUF
```
1. Select **SystemOne** in Settings and set the endpoint to `http://127.0.0.1:8080`
(or your server's address). Refresh models and select one, or choose **None**
for a single loaded model. Router deployments require the appropriate model ID.
2. Set the default match probability threshold (initially **50%**). Each rule has
an optional percentage override; leave it blank to inherit the default.
3. Apply rules normally. Each uncached criterion becomes a `noul` question. A rule
matches when the unrounded probability is **greater than or equal to** its
effective threshold. For example, 80% matches an 80% threshold.
4. Open Details to see a probability bar with the percentage inside. Hover over
the bar to see the current threshold and model. This is the
result under current settings, not an action history. Changing thresholds does
not automatically rerun actions. Decision models provide no textual explanation;
no second model is called to invent one. Use Chat Completions for explanations.
**SystemOne batching:** **Process all** is enabled by default and sends all eligible,
uncached criteria for an email in one request, with the email state included once.
Turn it off to set **Rules per request** (initially 8); use 1 for sequential
requests. The number is retained while disabled. These controls apply only to
SystemOne; Chat Completions continues processing one rule per request.
Cached rules consume no batch capacity, and identical criteria share a question
while retaining their individual thresholds. Actions execute in rule order and
honor stop-after-match. A batch can evaluate later rules whose actions are then
skipped; their probabilities remain available in Details. No further batches are
requested after stopping. Changing batch settings preserves existing cached results.
If any requested answer is missing or invalid, the entire batch fails without
caching its new answers or executing their actions. Previously completed actions
remain applied. Sortana does not automatically retry or split rejected batches;
if the server rejects a large request, disable Process all and reduce Rules per
request. Limits and performance depend on the server and model.
Probabilities are model estimates, not guaranteed accuracy. Evaluate thresholds
against representative labeled emails, including long messages and attachments.
SystemOne uses the existing text extraction and attachment summaries; it does not
send attachment binaries or images. Chat prompts and generation settings remain
saved but are inactive for SystemOne. Optional Bearer authentication works with
both APIs; organization/project headers apply only to Chat Completions.
**Input character limit** applies when aggressive reduction is enabled, independently
of maximum output tokens. Existing installations inherit their previous effective
character limit; the default is 3686. Character counts do not guarantee that input
fits a model's token context or llama-server batch limits. Leave room for question
instructions and configure the server for the chosen model.
A request or response-parsing failure stops the remaining rules for that email,
records an error, and allows the queue to continue with the next email. Failed
results are never cached. Actions already completed are not rolled back.
Upstream: [llama-server SystemOne documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#typesafe-compatible-api-endpoints).
### Verification
Run the dependency-free tests with Node.js 22 or newer:
```sh
node --test tests/*.test.mjs
```
Tests cover adapters, thresholds, URLs, settings validation, cache migration and
reuse, request failures, per-email snapshots, and queue behavior with mocked
Thunderbird services. Manual acceptance testing in Thunderbird should verify both
API types, model discovery, rule overrides and ordering, import/export, cache
clearing, Details, and stop-after-match against a running server.
## Architecture Overview ## Architecture Overview
Sortana is implemented entirely with documented MailExtension/WebExtension APIs. No custom experiment code or Thunderbird-internal fallback paths are required: Sortana is implemented entirely with standard WebExtension scripts—no custom experiment code is required:
- `background.js` loads saved settings, manages the classification queue and listens for new messages. - `background.js` loads saved settings, manages the classification queue and listens for new messages.
- `modules/AiClassifier.js` shares transport, threshold policy, and cache handling; - `modules/AiClassifier.js` implements the classification logic and cache handling.
`modules/backends/` contains the Chat Completions and SystemOne adapters.
- `options/` contains the HTML and JavaScript for configuring the endpoint and rules. - `options/` contains the HTML and JavaScript for configuring the endpoint and rules.
- `details.html` / `details.js` present cached classification evidence for a message. - `details.html` / `details.js` present cached reasoning for a message.
- `_locales/` holds localized strings used throughout the UI. - `_locales/` holds localized strings used throughout the UI.
### Key Files ### Key Files
@ -153,29 +55,23 @@ Sortana is implemented entirely with documented MailExtension/WebExtension APIs.
| `background.js` | Startup tasks and classification queue management. | | `background.js` | Startup tasks and classification queue management. |
| `modules/AiClassifier.js` | Core classification logic and cache handling. | | `modules/AiClassifier.js` | Core classification logic and cache handling. |
| `options/options.html` and `options.js` | Endpoint and rule configuration UI. | | `options/options.html` and `options.js` | Endpoint and rule configuration UI. |
| `details.html` and `details.js` | View stored classification evidence for a message. | | `details.html` and `details.js` | View stored reasoning for a message. |
| `logger.js` | Colorized logging with optional debug mode. | | `logger.js` | Colorized logging with optional debug mode. |
## Building ## Building
1. Ensure PowerShell is available on Windows or bash, `python3`, and Inkscape are 1. Ensure PowerShell is available (for Windows) or adapt the script for other
available on Linux. environments.
2. The Bulma stylesheet (v1.0.3) is already included as `options/bulma.css`. 2. The Bulma stylesheet (v1.0.3) is already included as `options/bulma.css`.
3. Run `powershell ./build-xpi.ps1` or `./build-xpi.sh` from the repository root. 3. Run `powershell ./build-xpi.ps1` from the repository root. The script reads
The script reads the version from `manifest.json` and creates an XPI in the the version from `manifest.json` and creates an XPI in the `release` folder.
`release` folder.
4. Install the generated XPI in Thunderbird via the Add-ons Manager. During 4. Install the generated XPI in Thunderbird via the Add-ons Manager. During
development you can also load the directory as a temporary add-on. development you can also load the directory as a temporary add-on.
5. To regenerate PNG icons from the SVG sources, run `resources/svg2img.ps1` on 5. To regenerate PNG icons from the SVG sources, run `resources/svg2img.ps1`.
Windows or `./resources/svg2img.sh` on Linux.
## Usage ## Usage
1. Open the add-on's options and set the base URL of your classification service 1. Open the add-on's options and set the URL of your classification service.
(Sortana appends the path for the selected API type). Use the Model dropdown to load
`/v1/models` and select a model or choose **None** to omit the `model` field.
Advanced settings include optional API key, organization, and project headers
for OpenAI-hosted endpoints.
2. Use the **Classification Rules** section to add a criterion and optional 2. Use the **Classification Rules** section to add a criterion and optional
actions such as tagging, moving, copying, forwarding, replying, actions such as tagging, moving, copying, forwarding, replying,
deleting or archiving a message when it matches. Drag rules to deleting or archiving a message when it matches. Drag rules to
@ -188,7 +84,7 @@ Sortana is implemented entirely with documented MailExtension/WebExtension APIs.
open a compose window using the account that received the message. open a compose window using the account that received the message.
3. Save your settings. New mail will be evaluated automatically using the 3. Save your settings. New mail will be evaluated automatically using the
configured rules. configured rules.
4. If the toolbar icon shows a red X, it will clear after a few seconds. Open the Errors tab in Options to review the latest failures. 4. If the toolbar icon shows a red X, click the notification's **Dismiss** button to clear the error.
### Example Filters ### Example Filters
@ -244,8 +140,6 @@ uses the following third party libraries:
- MIT License - MIT License
- [turndown v7.2.0](https://github.com/mixmark-io/turndown/tree/v7.2.0) - [turndown v7.2.0](https://github.com/mixmark-io/turndown/tree/v7.2.0)
- MIT License - MIT License
- [diff](https://github.com/google/diff-match-patch/blob/62f2e689f498f9c92dbc588c58750addec9b1654/javascript/diff_match_patch_uncompressed.js)
- Apache-2.0 license
## License ## License
@ -261,3 +155,4 @@ how Thunderbird's WebExtension and experiment APIs can be extended. Their code
provided invaluable guidance during development. provided invaluable guidance during development.
- Icons from [cc0-icons.jonh.eu](https://cc0-icons.jonh.eu/) are used under the CC0 license. - Icons from [cc0-icons.jonh.eu](https://cc0-icons.jonh.eu/) are used under the CC0 license.

View file

@ -4,15 +4,23 @@
"doesntMatch": { "message": "doesn't match" }, "doesntMatch": { "message": "doesn't match" },
"options.title": { "message": "AI Filter Options" }, "options.title": { "message": "AI Filter Options" },
"options.endpoint": { "message": "Endpoint" }, "options.endpoint": { "message": "Endpoint" },
"options.template": { "message": "Prompt template" },
"options.customTemplate": { "message": "Custom template" },
"options.systemInstructions": { "message": "System instructions" }, "options.systemInstructions": { "message": "System instructions" },
"options.reset": { "message": "Reset to default" }, "options.reset": { "message": "Reset to default" },
"options.placeholders": { "message": "Placeholders: {{system}}, {{email}}, {{query}}" },
"template.openai": { "message": "OpenAI / ChatML" },
"template.qwen": { "message": "Qwen" },
"template.mistral": { "message": "Mistral" },
"template.custom": { "message": "Custom" },
"options.save": { "message": "Save" }, "options.save": { "message": "Save" },
"options.debugLogging": { "message": "Enable debug logging" }, "options.debugLogging": { "message": "Enable debug logging" },
"options.htmlToMarkdown": { "message": "Convert HTML body to Markdown" }, "options.htmlToMarkdown": { "message": "Convert HTML body to Markdown" },
"options.stripUrlParams": { "message": "Remove URL tracking parameters" }, "options.stripUrlParams": { "message": "Remove URL tracking parameters" },
"options.altTextImages": { "message": "Replace images with alt text" }, "options.altTextImages": { "message": "Replace images with alt text" },
"options.collapseWhitespace": { "message": "Collapse long whitespace" }, "options.collapseWhitespace": { "message": "Collapse long whitespace" },
"options.tokenReduction": { "message": "Aggressive token reduction" } "options.tokenReduction": { "message": "Aggressive token reduction" },
"options.contextLength": { "message": "Context length" }
,"action.read": { "message": "read" } ,"action.read": { "message": "read" }
,"action.flag": { "message": "flag" } ,"action.flag": { "message": "flag" }
,"action.copy": { "message": "copy" } ,"action.copy": { "message": "copy" }

View file

@ -39,6 +39,13 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "en-US", "en-US", "{8BEA7793
_locales\en-US\messages.json = _locales\en-US\messages.json _locales\en-US\messages.json = _locales\en-US\messages.json
EndProjectSection EndProjectSection
EndProject EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "prompt_templates", "prompt_templates", "{86516D53-50D4-4FE2-9D8A-977A8F5EBDBD}"
ProjectSection(SolutionItems) = preProject
prompt_templates\mistral.txt = prompt_templates\mistral.txt
prompt_templates\openai.txt = prompt_templates\openai.txt
prompt_templates\qwen.txt = prompt_templates\qwen.txt
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "resources", "resources", "{68A87938-5C2B-49F5-8AAA-8A34FBBFD854}" Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "resources", "resources", "{68A87938-5C2B-49F5-8AAA-8A34FBBFD854}"
ProjectSection(SolutionItems) = preProject ProjectSection(SolutionItems) = preProject
resources\svg2img.ps1 = resources\svg2img.ps1 resources\svg2img.ps1 = resources\svg2img.ps1
@ -101,7 +108,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "img", "img", "{F266602F-175
EndProject EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "js", "js", "{21D2A42C-3F85-465C-9141-C106AFD92B68}" Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "js", "js", "{21D2A42C-3F85-465C-9141-C106AFD92B68}"
ProjectSection(SolutionItems) = preProject ProjectSection(SolutionItems) = preProject
resources\js\diff_match_patch_uncompressed.js = resources\js\diff_match_patch_uncompressed.js
resources\js\turndown.js = resources\js\turndown.js resources\js\turndown.js = resources\js\turndown.js
EndProjectSection EndProjectSection
EndProject EndProject

View file

@ -19,7 +19,6 @@ let queue = Promise.resolve();
let queuedCount = 0; let queuedCount = 0;
let processing = false; let processing = false;
let iconTimer = null; let iconTimer = null;
let errorTimer = null;
let timingStats = { count: 0, mean: 0, m2: 0, total: 0, last: -1 }; let timingStats = { count: 0, mean: 0, m2: 0, total: 0, last: -1 };
let currentStart = 0; let currentStart = 0;
let logGetTiming = true; let logGetTiming = true;
@ -28,16 +27,13 @@ let stripUrlParams = false;
let altTextImages = false; let altTextImages = false;
let collapseWhitespace = false; let collapseWhitespace = false;
let tokenReduction = false; let tokenReduction = false;
let contextLength = 16384;
let TurndownService = null; let TurndownService = null;
let userTheme = 'auto'; let userTheme = 'auto';
let currentTheme = 'light'; let currentTheme = 'light';
let detectSystemTheme; let detectSystemTheme;
let errorPending = false; let errorPending = false;
let errorLog = [];
let showDebugTab = false;
const ERROR_NOTIFICATION_ID = 'sortana-error'; const ERROR_NOTIFICATION_ID = 'sortana-error';
const ERROR_ICON_TIMEOUT = 4500;
const MAX_ERROR_LOG = 50;
function normalizeRules(rules) { function normalizeRules(rules) {
return Array.isArray(rules) ? rules.map(r => { return Array.isArray(rules) ? rules.map(r => {
@ -52,7 +48,6 @@ function normalizeRules(rules) {
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo }); if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
if (r.copyTarget || r.copyTo) actions.push({ type: 'copy', copyTarget: r.copyTarget || r.copyTo }); if (r.copyTarget || r.copyTo) actions.push({ type: 'copy', copyTarget: r.copyTarget || r.copyTo });
const rule = { criterion: r.criterion, actions }; const rule = { criterion: r.criterion, actions };
if (r.decisionThreshold !== undefined) rule.decisionThreshold = r.decisionThreshold;
if (r.stopProcessing) rule.stopProcessing = true; if (r.stopProcessing) rule.stopProcessing = true;
if (r.unreadOnly) rule.unreadOnly = true; if (r.unreadOnly) rule.unreadOnly = true;
if (typeof r.minAgeDays === 'number') rule.minAgeDays = r.minAgeDays; if (typeof r.minAgeDays === 'number') rule.minAgeDays = r.minAgeDays;
@ -112,42 +107,11 @@ function showTransientIcon(factory, delay = 1500) {
async function clearError() { async function clearError() {
errorPending = false; errorPending = false;
clearTimeout(errorTimer); await storage.local.set({ errorPending: false });
await browser.notifications.clear(ERROR_NOTIFICATION_ID); await browser.notifications.clear(ERROR_NOTIFICATION_ID);
updateActionIcon(); updateActionIcon();
} }
function recordError(context, err) {
let message = 'Unknown error';
let detail = '';
if (err instanceof Error) {
message = err.message;
detail = err.stack || '';
} else if (err && typeof err === 'object') {
message = typeof err.message === 'string' ? err.message : String(err || 'Unknown error');
detail = typeof err.detail === 'string' ? err.detail : '';
} else {
message = String(err || 'Unknown error');
}
errorLog.unshift({
time: Date.now(),
context,
message,
detail
});
if (errorLog.length > MAX_ERROR_LOG) {
errorLog.length = MAX_ERROR_LOG;
}
errorPending = true;
updateActionIcon();
clearTimeout(errorTimer);
errorTimer = setTimeout(() => {
errorPending = false;
updateActionIcon();
}, ERROR_ICON_TIMEOUT);
browser.runtime.sendMessage({ type: 'sortana:errorLogUpdated', count: errorLog.length }).catch(() => {});
}
function refreshMenuIcons() { function refreshMenuIcons() {
browser.menus.update('apply-ai-rules-list', { icons: iconPaths('eye') }); browser.menus.update('apply-ai-rules-list', { icons: iconPaths('eye') });
browser.menus.update('apply-ai-rules-display', { icons: iconPaths('eye') }); browser.menus.update('apply-ai-rules-display', { icons: iconPaths('eye') });
@ -245,38 +209,17 @@ function collectText(part, bodyParts, attachments) {
} }
} }
function collectRawText(part, bodyParts, attachments) { function buildEmailText(full) {
if (part.parts && part.parts.length) {
for (const p of part.parts) collectRawText(p, bodyParts, attachments);
return;
}
const ct = (part.contentType || "text/plain").toLowerCase();
const cd = (part.headers?.["content-disposition"]?.[0] || "").toLowerCase();
const body = String(part.body || "");
if (cd.includes("attachment") || !ct.startsWith("text/")) {
const nameMatch = /filename\s*=\s*"?([^";]+)/i.exec(cd) || /name\s*=\s*"?([^";]+)/i.exec(part.headers?.["content-type"]?.[0] || "");
const name = nameMatch ? nameMatch[1] : "";
attachments.push(`${name} (${ct}, ${part.size || byteSize(body)} bytes)`);
} else if (ct.startsWith("text/html")) {
const doc = new DOMParser().parseFromString(body, 'text/html');
bodyParts.push(doc.body.textContent || "");
} else {
bodyParts.push(body);
}
}
function buildEmailText(full, applyTransforms = true) {
const bodyParts = []; const bodyParts = [];
const attachments = []; const attachments = [];
const collect = applyTransforms ? collectText : collectRawText; collectText(full, bodyParts, attachments);
collect(full, bodyParts, attachments);
const headers = Object.entries(full.headers || {}) const headers = Object.entries(full.headers || {})
.map(([k, v]) => `${k}: ${v.join(' ')}`) .map(([k, v]) => `${k}: ${v.join(' ')}`)
.join('\n'); .join('\n');
const attachInfo = `Attachments: ${attachments.length}` + const attachInfo = `Attachments: ${attachments.length}` +
(attachments.length ? "\n" + attachments.map(a => ` - ${a}`).join('\n') : ""); (attachments.length ? "\n" + attachments.map(a => ` - ${a}`).join('\n') : "");
let combined = `${headers}\n${attachInfo}\n\n${bodyParts.join('\n')}`.trim(); let combined = `${headers}\n${attachInfo}\n\n${bodyParts.join('\n')}`.trim();
if (applyTransforms && tokenReduction) { if (tokenReduction) {
const seen = new Set(); const seen = new Set();
combined = combined.split('\n').filter(l => { combined = combined.split('\n').filter(l => {
if (seen.has(l)) return false; if (seen.has(l)) return false;
@ -284,7 +227,7 @@ function buildEmailText(full, applyTransforms = true) {
return true; return true;
}).join('\n'); }).join('\n');
} }
return applyTransforms ? sanitizeString(combined) : combined; return sanitizeString(combined);
} }
function updateTimingStats(elapsed) { function updateTimingStats(elapsed) {
@ -318,19 +261,13 @@ async function processMessage(id) {
updateActionIcon(); updateActionIcon();
try { try {
const full = await messenger.messages.getFull(id); const full = await messenger.messages.getFull(id);
const snapshot = AiClassifier.getConfig();
const rules = structuredClone(aiRules);
const originalText = buildEmailText(full, false);
let text = buildEmailText(full); let text = buildEmailText(full);
if (tokenReduction) { if (tokenReduction && contextLength > 0) {
const limit = snapshot.inputCharacterLimit; const limit = Math.floor(contextLength * 0.9);
if (text.length > limit) { if (text.length > limit) {
text = text.slice(0, limit); text = text.slice(0, limit);
} }
} }
if (showDebugTab) {
await storage.local.set({ lastFullText: originalText, lastPromptText: text });
}
let hdr; let hdr;
let currentTags = []; let currentTags = [];
let alreadyRead = false; let alreadyRead = false;
@ -347,8 +284,7 @@ async function processMessage(id) {
identityId = null; identityId = null;
} }
const candidates = []; for (const rule of aiRules) {
for (const [index, rule] of rules.entries()) {
if (rule.enabled === false) { if (rule.enabled === false) {
continue; continue;
} }
@ -375,10 +311,8 @@ async function processMessage(id) {
} }
} }
} }
const cacheKey = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot); const cacheKey = await AiClassifier.buildCacheKey(id, rule.criterion);
candidates.push({ id: `rule_${index}`, rule, cacheKey }); const matched = await AiClassifier.classifyText(text, rule.criterion, cacheKey);
}
for await (const { rule, result: { matched } } of AiClassifier.classifyRules(text, candidates, snapshot)) {
if (matched) { if (matched) {
for (const act of (rule.actions || [])) { for (const act of (rule.actions || [])) {
if (act.type === 'tag' && act.tagKey) { if (act.type === 'tag' && act.tagKey) {
@ -422,14 +356,16 @@ async function processMessage(id) {
const elapsed = Date.now() - currentStart; const elapsed = Date.now() - currentStart;
currentStart = 0; currentStart = 0;
updateTimingStats(elapsed); updateTimingStats(elapsed);
await storage.local.set({ classifyStats: timingStats }); await storage.local.set({ classifyStats: timingStats, errorPending: true });
errorPending = true;
logger.aiLog("failed to apply AI rules", { level: 'error' }, e); logger.aiLog("failed to apply AI rules", { level: 'error' }, e);
recordError("Failed to apply AI rules", e); setIcon(ICONS.error());
browser.notifications.create(ERROR_NOTIFICATION_ID, { browser.notifications.create(ERROR_NOTIFICATION_ID, {
type: 'basic', type: 'basic',
iconUrl: browser.runtime.getURL('resources/img/logo.png'), iconUrl: browser.runtime.getURL('resources/img/logo.png'),
title: 'Sortana Error', title: 'Sortana Error',
message: 'Failed to apply AI rules' message: 'Failed to apply AI rules',
buttons: [{ title: 'Dismiss' }]
}); });
} }
} }
@ -461,8 +397,18 @@ async function clearCacheForMessages(idsInput) {
aiRules = normalizeRules(stored); aiRules = normalizeRules(stored);
} }
for (const msg of ids) await AiClassifier.clearCacheForMessage(msg?.id ?? msg); const keys = [];
showTransientIcon(ICONS.circle); for (const msg of ids) {
const id = msg?.id ?? msg;
for (const rule of aiRules) {
const key = await AiClassifier.buildCacheKey(id, rule.criterion);
keys.push(key);
}
}
if (keys.length) {
await AiClassifier.removeCacheEntries(keys);
showTransientIcon(ICONS.circle);
}
} }
(async () => { (async () => {
@ -479,13 +425,8 @@ async function clearCacheForMessages(idsInput) {
} }
try { try {
const store = await storage.local.get(["systemOneProcessAll", "systemOneBatchSize", "apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]); const store = await storage.local.get(["endpoint", "templateName", "customTemplate", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "contextLength", "aiRules", "theme", "errorPending"]);
logger.setDebug(store.debugLogging); logger.setDebug(store.debugLogging);
const { migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js'));
if (store.inputCharacterLimit === undefined) {
store.inputCharacterLimit = migratedInputLimit(store);
await storage.local.set({ inputCharacterLimit: store.inputCharacterLimit });
}
await AiClassifier.setConfig(store); await AiClassifier.setConfig(store);
userTheme = store.theme || 'auto'; userTheme = store.theme || 'auto';
currentTheme = userTheme === 'auto' ? await detectSystemTheme() : userTheme; currentTheme = userTheme === 'auto' ? await detectSystemTheme() : userTheme;
@ -495,7 +436,8 @@ async function clearCacheForMessages(idsInput) {
altTextImages = store.altTextImages === true; altTextImages = store.altTextImages === true;
collapseWhitespace = store.collapseWhitespace === true; collapseWhitespace = store.collapseWhitespace === true;
tokenReduction = store.tokenReduction === true; tokenReduction = store.tokenReduction === true;
showDebugTab = store.showDebugTab === true; contextLength = parseInt(store.contextLength) || contextLength;
errorPending = store.errorPending === true;
const savedStats = await storage.local.get('classifyStats'); const savedStats = await storage.local.get('classifyStats');
if (savedStats.classifyStats && typeof savedStats.classifyStats === 'object') { if (savedStats.classifyStats && typeof savedStats.classifyStats === 'object') {
Object.assign(timingStats, savedStats.classifyStats); Object.assign(timingStats, savedStats.classifyStats);
@ -511,9 +453,20 @@ async function clearCacheForMessages(idsInput) {
aiRules = normalizeRules(newRules); aiRules = normalizeRules(newRules);
logger.aiLog("aiRules updated from storage change", { debug: true }, aiRules); logger.aiLog("aiRules updated from storage change", { debug: true }, aiRules);
} }
const configChanges = Object.fromEntries(Object.entries(changes).map(([key, change]) => [key, change.newValue])); if (changes.endpoint || changes.templateName || changes.customTemplate || changes.customSystemPrompt || changes.aiParams || changes.debugLogging) {
AiClassifier.setConfig(configChanges); const config = {};
if (changes.aiCache) AiClassifier.replaceCache(changes.aiCache.newValue); if (changes.endpoint) config.endpoint = changes.endpoint.newValue;
if (changes.templateName) config.templateName = changes.templateName.newValue;
if (changes.customTemplate) config.customTemplate = changes.customTemplate.newValue;
if (changes.customSystemPrompt) config.customSystemPrompt = changes.customSystemPrompt.newValue;
if (changes.aiParams) config.aiParams = changes.aiParams.newValue;
if (changes.debugLogging) {
config.debugLogging = changes.debugLogging.newValue === true;
logger.setDebug(config.debugLogging);
}
await AiClassifier.setConfig(config);
logger.aiLog("AiClassifier config updated from storage change", { debug: true }, config);
}
if (changes.htmlToMarkdown) { if (changes.htmlToMarkdown) {
htmlToMarkdown = changes.htmlToMarkdown.newValue === true; htmlToMarkdown = changes.htmlToMarkdown.newValue === true;
logger.aiLog("htmlToMarkdown updated from storage change", { debug: true }, htmlToMarkdown); logger.aiLog("htmlToMarkdown updated from storage change", { debug: true }, htmlToMarkdown);
@ -534,8 +487,13 @@ async function clearCacheForMessages(idsInput) {
tokenReduction = changes.tokenReduction.newValue === true; tokenReduction = changes.tokenReduction.newValue === true;
logger.aiLog("tokenReduction updated from storage change", { debug: true }, tokenReduction); logger.aiLog("tokenReduction updated from storage change", { debug: true }, tokenReduction);
} }
if (changes.showDebugTab) { if (changes.contextLength) {
showDebugTab = changes.showDebugTab.newValue === true; contextLength = parseInt(changes.contextLength.newValue) || contextLength;
logger.aiLog("contextLength updated from storage change", { debug: true }, contextLength);
}
if (changes.errorPending) {
errorPending = changes.errorPending.newValue === true;
updateActionIcon();
} }
if (changes.theme) { if (changes.theme) {
userTheme = changes.theme.newValue || 'auto'; userTheme = changes.theme.newValue || 'auto';
@ -645,10 +603,10 @@ async function clearCacheForMessages(idsInput) {
logger.aiLog("sortana:test – criterion", { debug: true }, criterion); logger.aiLog("sortana:test – criterion", { debug: true }, criterion);
try { try {
logger.aiLog("Calling AiClassifier.classify()", { debug: true }); logger.aiLog("Calling AiClassifier.classifyText()", { debug: true });
const result = await AiClassifier.classify(text, criterion, { threshold: msg.decisionThreshold ?? AiClassifier.getConfig().decisionThreshold }); const result = await AiClassifier.classifyText(text, criterion);
logger.aiLog("classify() returned", { debug: true }, result); logger.aiLog("classify() returned", { debug: true }, result);
return { ...result, match: result.matched }; return { match: result };
} }
catch (err) { catch (err) {
logger.aiLog("Error in classify()", { level: 'error' }, err); logger.aiLog("Error in classify()", { level: 'error' }, err);
@ -695,11 +653,13 @@ async function clearCacheForMessages(idsInput) {
aiRules = normalizeRules(stored); aiRules = normalizeRules(stored);
} }
const results = []; const results = [];
const snapshot = AiClassifier.getConfig(); for (const rule of aiRules) {
for (const rule of structuredClone(aiRules)) { const key = await AiClassifier.buildCacheKey(id, rule.criterion);
const key = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot); const matched = AiClassifier.getCachedResult(key);
const detail = AiClassifier.getCachedDetails(key, rule.decisionThreshold ?? snapshot.decisionThreshold); const reason = AiClassifier.getReason(key);
if (detail) results.push({ criterion: rule.criterion, ...detail }); if (matched !== null || reason) {
results.push({ criterion: rule.criterion, matched, reason });
}
} }
return { subject, results }; return { subject, results };
} catch (e) { } catch (e) {
@ -725,24 +685,8 @@ async function clearCacheForMessages(idsInput) {
logger.aiLog("failed to clear cache for message", { level: 'error' }, e); logger.aiLog("failed to clear cache for message", { level: 'error' }, e);
return { ok: false }; return { ok: false };
} }
} else if (msg?.type === "sortana:resetTimingStats") {
const last = timingStats.last;
timingStats.count = 0;
timingStats.mean = 0;
timingStats.m2 = 0;
timingStats.total = 0;
timingStats.last = typeof last === 'number' ? last : -1;
await storage.local.set({ classifyStats: timingStats });
return { ok: true };
} else if (msg?.type === "sortana:getCacheSize") {
return { count: await AiClassifier.getCacheSize() };
} else if (msg?.type === "sortana:getQueueCount") { } else if (msg?.type === "sortana:getQueueCount") {
return { count: queuedCount + (processing ? 1 : 0) }; return { count: queuedCount + (processing ? 1 : 0) };
} else if (msg?.type === "sortana:getErrorLog") {
return { errors: errorLog.slice() };
} else if (msg?.type === "sortana:recordError") {
recordError(msg.context || "Sortana Error", { message: msg.message, detail: msg.detail });
return { ok: true };
} else if (msg?.type === "sortana:getTiming") { } else if (msg?.type === "sortana:getTiming") {
const t = timingStats; const t = timingStats;
const std = t.count > 1 ? Math.sqrt(t.m2 / (t.count - 1)) : 0; const std = t.count > 1 ? Math.sqrt(t.m2 / (t.count - 1)) : 0;
@ -774,7 +718,6 @@ async function clearCacheForMessages(idsInput) {
// Catch any unhandled rejections // Catch any unhandled rejections
window.addEventListener("unhandledrejection", ev => { window.addEventListener("unhandledrejection", ev => {
logger.aiLog("Unhandled promise rejection", { level: 'error' }, ev.reason); logger.aiLog("Unhandled promise rejection", { level: 'error' }, ev.reason);
recordError("Unhandled promise rejection", ev.reason);
}); });
browser.notifications.onClicked.addListener(id => { browser.notifications.onClicked.addListener(id => {

View file

@ -1,77 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
release_dir="$script_dir/release"
manifest="$script_dir/manifest.json"
if [[ ! -f "$manifest" ]]; then
echo "manifest.json not found at $manifest" >&2
exit 1
fi
if ! command -v zip >/dev/null 2>&1; then
echo "zip is required to build the XPI." >&2
exit 1
fi
if command -v jq >/dev/null 2>&1; then
version="$(jq -r '.version // empty' "$manifest")"
else
if ! command -v python3 >/dev/null 2>&1; then
echo "python3 is required to read manifest.json without jq." >&2
exit 1
fi
version="$(python3 - <<'PY'
import json
import sys
with open(sys.argv[1], 'r', encoding='utf-8') as f:
data = json.load(f)
print(data.get('version', '') or '')
PY
"$manifest")"
fi
if [[ -z "$version" ]]; then
echo "No version found in manifest.json" >&2
exit 1
fi
mkdir -p "$release_dir"
xpi_name="sortana-$version.xpi"
zip_path="$release_dir/ai-filter-$version.zip"
xpi_path="$release_dir/$xpi_name"
rm -f "$zip_path" "$xpi_path"
mapfile -d '' files < <(
find "$script_dir" -type f \
! -name '*.sln' \
! -name '*.ps1' \
! -name '*.sh' \
! -path "$release_dir/*" \
! -path "$script_dir/.vs/*" \
! -path "$script_dir/.git/*" \
-printf '%P\0'
)
if [[ ${#files[@]} -eq 0 ]]; then
echo "No files found to package." >&2
exit 0
fi
for rel in "${files[@]}"; do
full="$script_dir/$rel"
size=$(stat -c '%s' "$full")
echo "Zipping: $rel <- $full ($size bytes)"
done
(
cd "$script_dir"
printf '%s\n' "${files[@]}" | zip -q -9 -@ "$zip_path"
)
mv -f "$zip_path" "$xpi_path"
echo "Built XPI at: $xpi_path"

View file

@ -4,33 +4,6 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<title>AI Details</title> <title>AI Details</title>
<link rel="stylesheet" href="options/bulma.css"> <link rel="stylesheet" href="options/bulma.css">
<style>
.probability-bar {
position: relative;
height: 1.75rem;
overflow: hidden;
border: 1px solid var(--bulma-border);
border-radius: var(--bulma-radius-small);
background: var(--bulma-background);
}
.probability-fill {
height: 100%;
background: var(--bulma-success);
opacity: 0.35;
}
.probability-bar.is-danger .probability-fill {
background: var(--bulma-danger);
}
.probability-value {
position: absolute;
inset: 0;
display: flex;
align-items: center;
justify-content: center;
color: var(--bulma-text-strong);
font-weight: 600;
}
</style>
</head> </head>
<body> <body>
<section class="section"> <section class="section">

View file

@ -51,39 +51,15 @@ async function loadMessage(id) {
article.className = `message ${color} mb-4`; article.className = `message ${color} mb-4`;
const header = document.createElement('div'); const header = document.createElement('div');
header.className = 'message-header'; header.className = 'message-header';
const criterion = document.createElement('p'); header.innerHTML = `<p>${r.criterion}</p>`;
criterion.textContent = r.criterion;
header.appendChild(criterion);
const body = document.createElement('div'); const body = document.createElement('div');
body.className = 'message-body'; body.className = 'message-body';
const status = document.createElement('p'); const status = document.createElement('p');
status.textContent = r.matched ? 'Matched' : 'Did not match'; status.textContent = r.matched ? 'Matched' : 'Did not match';
if (r.backend === 'systemone') { const pre = document.createElement('pre');
status.textContent += ' under current threshold'; pre.textContent = r.reason || '';
status.classList.add('mb-2'); body.appendChild(status);
body.appendChild(status); body.appendChild(pre);
const percentage = r.probability * 100;
const bar = document.createElement('div');
bar.className = `probability-bar ${color}`;
bar.setAttribute('role', 'progressbar');
bar.setAttribute('aria-label', 'Match probability');
bar.setAttribute('aria-valuemin', '0');
bar.setAttribute('aria-valuemax', '100');
bar.setAttribute('aria-valuenow', String(percentage));
bar.title = `Threshold: ${r.threshold * 100}%${r.model ? ` · Model: ${r.model}` : ''}`;
const fill = document.createElement('div');
fill.className = 'probability-fill';
fill.style.width = `${percentage}%`;
const value = document.createElement('span');
value.className = 'probability-value';
value.textContent = `${percentage.toFixed(2)}%`;
bar.append(fill, value);
body.appendChild(bar);
} else {
const pre = document.createElement('pre');
pre.textContent = r.reason || '';
body.append(status, pre);
}
article.appendChild(header); article.appendChild(header);
article.appendChild(body); article.appendChild(body);
container.appendChild(article); container.appendChild(article);

View file

@ -1,12 +1,13 @@
{ {
"manifest_version": 2, "manifest_version": 2,
"name": "Sortana", "name": "Sortana",
"version": "4.1.0", "version": "2.1.2",
"default_locale": "en-US", "default_locale": "en-US",
"applications": { "applications": {
"gecko": { "gecko": {
"id": "ai-filter@jordanwages", "id": "ai-filter@jordanwages",
"strict_min_version": "128.0" "strict_min_version": "128.0",
"strict_max_version": "139.*"
} }
}, },
"icons": { "icons": {

View file

@ -1,209 +1,342 @@
"use strict"; "use strict";
import { aiLog, setDebug } from '../logger.js'; import { aiLog, setDebug } from "../logger.js";
import { CONFIG_DEFAULTS, validateSettings, validateThreshold } from './classificationConfig.js'; import { DEFAULT_AI_PARAMS } from "./defaultParams.js";
import * as chat from './backends/chatCompletions.js';
import * as systemone from './backends/systemOne.js';
const storage = (globalThis.messenger ?? globalThis.browser).storage; const storage = (globalThis.messenger ?? globalThis.browser).storage;
const adapters = { 'chat-completions': chat, systemone };
let config = structuredClone(CONFIG_DEFAULTS);
let cache = new Map();
let loaded = false;
let loading;
let generation = 0;
export function normalizeEndpointBase(endpoint) { let Services;
if (typeof endpoint !== 'string' || !endpoint.trim()) return ''; try {
return endpoint.trim().replace(/\/+$/, '').replace(/\/v1(?:\/(?:chat\/completions|completions|models|systemone))?$/i, ''); if (typeof globalThis !== "undefined" && globalThis.Services) {
} Services = globalThis.Services;
function urlFor(endpoint, path) { } else if (typeof ChromeUtils !== "undefined" && ChromeUtils.importESModule) {
const base = normalizeEndpointBase(endpoint); ({ Services } = ChromeUtils.importESModule("resource://gre/modules/Services.sys.mjs"));
return base ? `${/^https?:\/\//i.test(base) ? base : `https://${base}`}${path}` : '';
}
export function buildEndpointUrl(endpoint, apiType = 'chat-completions') {
if (!adapters[apiType]) throw new Error('Unknown API type.');
return urlFor(endpoint, adapters[apiType].path);
}
export function buildModelsUrl(endpoint) { return urlFor(endpoint, '/v1/models'); }
export function getConfig() { return structuredClone(config); }
export function setConfig(update = {}) {
validateSettings(update);
for (const key of Object.keys(CONFIG_DEFAULTS)) {
if (update[key] !== undefined) config[key] = structuredClone(update[key]);
} }
config.aiParams = { ...CONFIG_DEFAULTS.aiParams, ...config.aiParams }; } catch (e) {
for (const key of ['model', 'apiKey', 'openaiOrganization', 'openaiProject']) config[key] = String(config[key]).trim(); Services = undefined;
setDebug(config.debugLogging);
} }
async function hash(value) {
const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(value)); const SYSTEM_PREFIX = `You are an email-classification assistant.
return Array.from(new Uint8Array(digest), b => b.toString(16).padStart(2, '0')).join(''); Read the email below and the classification criterion provided by the user.
} `;
async function messageHash(id) {
const api = globalThis.messenger ?? globalThis.browser; const DEFAULT_CUSTOM_SYSTEM_PROMPT = "Determine whether the email satisfies the user's criterion.";
if (typeof id === 'number' && api.messages?.get) {
const header = await api.messages.get(id); const SYSTEM_SUFFIX = `
id = header.headerMessageId || id; Return ONLY a JSON object on a single line of the form:
{"match": true} - if the email satisfies the criterion
{"match": false} - otherwise
Do not add any other keys, text, or formatting.`;
let gEndpoint = "http://127.0.0.1:5000/v1/classify";
let gTemplateName = "openai";
let gCustomTemplate = "";
let gCustomSystemPrompt = DEFAULT_CUSTOM_SYSTEM_PROMPT;
let gTemplateText = "";
let gAiParams = Object.assign({}, DEFAULT_AI_PARAMS);
let gCache = new Map();
let gCacheLoaded = false;
function sha256HexSync(str) {
try {
const hasher = Cc["@mozilla.org/security/hash;1"].createInstance(Ci.nsICryptoHash);
hasher.init(Ci.nsICryptoHash.SHA256);
const data = new TextEncoder().encode(str);
hasher.update(data, data.length);
const binary = hasher.finish(false);
return Array.from(binary, c => ("0" + c.charCodeAt(0).toString(16)).slice(-2)).join("");
} catch (e) {
aiLog(`sha256HexSync failed`, { level: 'error' }, e);
return "";
} }
return hash(String(id));
} }
function stable(value) {
if (Array.isArray(value)) return value.map(stable); async function sha256Hex(str) {
if (value && typeof value === 'object') return Object.fromEntries(Object.keys(value).sort().map(k => [k, stable(value[k])])); if (typeof crypto?.subtle?.digest === "function") {
return value; const buf = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(str));
} return Array.from(new Uint8Array(buf), b => b.toString(16).padStart(2, "0")).join("");
export async function buildCacheKey(id, criterion, snapshot = getConfig()) {
const fingerprint = {
backend: snapshot.apiType, endpoint: buildEndpointUrl(snapshot.endpoint, snapshot.apiType),
request: adapters[snapshot.apiType].buildRequest('', criterion, snapshot),
organization: snapshot.apiType === 'chat-completions' ? snapshot.openaiOrganization : '',
project: snapshot.apiType === 'chat-completions' ? snapshot.openaiProject : '',
preprocessing: Object.fromEntries(['htmlToMarkdown', 'stripUrlParams', 'altTextImages', 'collapseWhitespace', 'tokenReduction', 'inputCharacterLimit'].map(k => [k, snapshot[k]]))
};
return `v2:${await messageHash(id)}:${await hash(JSON.stringify(stable(fingerprint)))}`;
}
export function validCacheEntry(key, entry) {
return /^v2:[a-f0-9]{64}:[a-f0-9]{64}$/.test(key) && entry?.version === 2 &&
entry.messageHash === key.split(':')[1] && typeof entry.model === 'string' &&
(entry.backend === 'systemone'
? typeof entry.probability === 'number' && Number.isFinite(entry.probability) && entry.probability >= 0 && entry.probability <= 1 && entry.reason === null
: entry.backend === 'chat-completions' && typeof entry.matched === 'boolean' && entry.probability === null && (entry.reason === null || typeof entry.reason === 'string'));
}
export function replaceCache(data = {}) {
cache = new Map(Object.entries(data || {}).filter(([key, entry]) => validCacheEntry(key, entry)));
loaded = true;
generation++;
}
async function saveCache() { await storage.local.set({ aiCache: Object.fromEntries(cache) }); }
export async function init() {
if (loaded) return;
if (!loading) loading = (async () => {
const { aiCache = {}, aiReasonCache } = await storage.local.get(['aiCache', 'aiReasonCache']);
replaceCache(aiCache);
if (Object.keys(aiCache || {}).length !== cache.size) await saveCache();
if (aiReasonCache !== undefined) await storage.local.remove('aiReasonCache');
})().finally(() => { loading = null; });
await loading;
}
function withPolicy(entry, threshold) {
const isDecision = entry.backend === 'systemone';
if (isDecision) validateThreshold(threshold);
return { matched: isDecision ? entry.probability >= threshold : entry.matched,
reason: entry.reason, probability: entry.probability, threshold: isDecision ? threshold : null,
backend: entry.backend, model: entry.model };
}
export function getCachedDetails(key, threshold = config.decisionThreshold) {
const entry = cache.get(key);
return entry ? withPolicy(entry, threshold) : null;
}
export function getCachedResult(key, threshold) { return getCachedDetails(key, threshold)?.matched ?? null; }
export function getReason(key) { return cache.get(key)?.reason ?? null; }
async function sendRequest(payload, snapshot) {
try { await storage.local.set({ lastPayload: payload }); }
catch (error) { aiLog('Failed to save last payload', { level: 'warn' }, error); }
const headers = { 'Content-Type': 'application/json' };
if (snapshot.apiKey) headers.Authorization = `Bearer ${snapshot.apiKey}`;
if (snapshot.apiType === 'chat-completions') {
if (snapshot.openaiOrganization) headers['OpenAI-Organization'] = snapshot.openaiOrganization;
if (snapshot.openaiProject) headers['OpenAI-Project'] = snapshot.openaiProject;
} }
const endpoint = buildEndpointUrl(snapshot.endpoint, snapshot.apiType); return sha256HexSync(str);
const response = await fetch(endpoint, { method: 'POST', headers, body: JSON.stringify(payload) }); }
if (!response.ok) {
let detail = ''; async function resolveHeaderId(id) {
if (typeof id === "number" && typeof messenger?.messages?.get === "function") {
try { try {
const body = await response.json(); const hdr = await messenger.messages.get(id);
const message = body?.error?.message; if (hdr?.headerMessageId) {
if (typeof message === 'string') detail = ` Server response: ${message.slice(0, 300)}.`; return hdr.headerMessageId;
} catch { /* Non-JSON error responses still report the URL and status. */ } }
const hint = response.status === 404 && snapshot.apiType === 'systemone' } catch (e) {
? ' Verify that the running server build supports /v1/systemone and the selected decision model.' : ''; aiLog(`Failed to resolve headerMessageId for ${id}`, { level: 'warn' }, e);
throw new Error(`Classification request failed: HTTP ${response.status} at ${endpoint}.${detail}${hint}`); }
} }
return response.json(); return String(id);
} }
export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) { async function buildCacheKey(id, criterion) {
await init(); const resolvedId = await resolveHeaderId(id);
if (snapshot.apiType === 'systemone') validateThreshold(threshold); if (Services) {
const cached = getCachedDetails(cacheKey, threshold); return sha256HexSync(`${resolvedId}|${criterion}`);
if (cached) return cached;
const epoch = generation;
const adapter = adapters[snapshot.apiType];
const payload = adapter.buildRequest(text, criterion, snapshot);
const raw = await sendRequest(payload, snapshot);
const entry = { ...adapter.parseResponse(raw), backend: snapshot.apiType,
model: typeof raw.model === 'string' ? raw.model : snapshot.model, version: 2,
messageHash: cacheKey?.split(':')[1] };
if (cacheKey && epoch === generation) {
cache.set(cacheKey, entry);
await saveCache();
} }
return withPolicy(entry, threshold); return sha256Hex(`${resolvedId}|${criterion}`);
} }
// Yield lazily so the action runner can stop without starting the next request.
// Candidates have precomputed keys, allowing actions to move/delete the message. async function loadCache() {
export async function* classifyRules(text, candidates, snapshot = getConfig()) { if (gCacheLoaded) {
await init();
validateSettings(snapshot);
if (snapshot.apiType !== 'systemone') {
for (const candidate of candidates) {
const result = await classify(text, candidate.rule.criterion, {
cacheKey: candidate.cacheKey, snapshot,
threshold: candidate.rule.decisionThreshold ?? snapshot.decisionThreshold
});
yield { rule: candidate.rule, result };
}
return; return;
} }
const limit = snapshot.systemOneProcessAll ? Infinity : snapshot.systemOneBatchSize; aiLog(`[AiClassifier] Loading cache`, {debug: true});
// Keep received evidence locally even if persistence is cleared during this run. try {
const evidence = new Map(); const { aiCache, aiReasonCache } = await storage.local.get(["aiCache", "aiReasonCache"]);
for (const candidate of candidates) { if (aiCache) {
validateThreshold(candidate.rule.decisionThreshold ?? snapshot.decisionThreshold); for (let [k, v] of Object.entries(aiCache)) {
const entry = cache.get(candidate.cacheKey); if (v && typeof v === "object") {
if (entry) evidence.set(candidate.cacheKey, entry); gCache.set(k, { matched: v.matched ?? null, reason: v.reason || "" });
} } else {
for (let index = 0; index < candidates.length; index++) { gCache.set(k, { matched: v, reason: "" });
const candidate = candidates[index];
if (!evidence.has(candidate.cacheKey)) {
const pending = new Map();
for (let next = index; next < candidates.length && pending.size < limit; next++) {
const item = candidates[next];
if (!evidence.has(item.cacheKey) && !pending.has(item.cacheKey)) {
pending.set(item.cacheKey, { ...item, criterion: item.rule.criterion });
} }
} }
const questions = [...pending.values()]; aiLog(`[AiClassifier] Loaded ${gCache.size} cache entries`, {debug: true});
const epoch = generation; } else {
const raw = await sendRequest(systemone.buildBatchRequest(text, questions, snapshot), snapshot); aiLog(`[AiClassifier] Cache is empty`, {debug: true});
// Parse the complete batch before accepting any of its evidence.
const parsed = systemone.parseBatchResponse(raw, questions);
const entries = questions.map((question, i) => [question.cacheKey, {
...parsed[i], backend: snapshot.apiType, version: 2,
model: typeof raw.model === 'string' ? raw.model : snapshot.model,
messageHash: question.cacheKey.split(':')[1]
}]);
for (const [key, entry] of entries) evidence.set(key, entry);
if (epoch === generation) {
for (const [key, entry] of entries) cache.set(key, entry);
await saveCache();
}
} }
yield { rule: candidate.rule, result: withPolicy(evidence.get(candidate.cacheKey), if (aiReasonCache) {
candidate.rule.decisionThreshold ?? snapshot.decisionThreshold) }; aiLog(`[AiClassifier] Migrating ${Object.keys(aiReasonCache).length} reason entries`, {debug: true});
for (let [k, reason] of Object.entries(aiReasonCache)) {
let entry = gCache.get(k) || { matched: null, reason: "" };
entry.reason = reason;
gCache.set(k, entry);
}
await storage.local.remove("aiReasonCache");
await storage.local.set({ aiCache: Object.fromEntries(gCache) });
}
} catch (e) {
aiLog(`Failed to load cache`, {level: 'error'}, e);
}
gCacheLoaded = true;
}
async function saveCache(updatedKey, updatedValue) {
if (typeof updatedKey !== "undefined") {
aiLog(`[AiClassifier] ⮡ Persisting entry '${updatedKey}' → ${updatedValue}`, {debug: true});
}
try {
await storage.local.set({ aiCache: Object.fromEntries(gCache) });
} catch (e) {
aiLog(`Failed to save cache`, {level: 'error'}, e);
} }
} }
export async function classifyText(text, criterion, cacheKey = null) { return (await classify(text, criterion, { cacheKey })).matched; }
export async function removeCacheEntries(keys = []) { async function loadTemplate(name) {
await init(); generation++; try {
for (const key of Array.isArray(keys) ? keys : [keys]) cache.delete(key); const url = typeof browser !== "undefined" && browser.runtime?.getURL
await saveCache(); ? browser.runtime.getURL(`prompt_templates/${name}.txt`)
: `resource://aifilter/prompt_templates/${name}.txt`;
const res = await fetch(url);
if (res.ok) {
return await res.text();
}
} catch (e) {
aiLog(`Failed to load template '${name}':`, {level: 'error'}, e);
}
return "";
} }
export async function clearCacheForMessage(id) {
await init(); function loadTemplateSync(name) {
const identifier = await messageHash(id); if (!Services?.tm?.spinEventLoopUntil) {
await removeCacheEntries([...cache].filter(([, entry]) => entry.messageHash === identifier).map(([key]) => key)); throw new Error("loadTemplateSync requires Services");
}
let text = "";
let done = false;
loadTemplate(name).then(t => { text = t; }).catch(() => {}).finally(() => { done = true; });
Services.tm.spinEventLoopUntil(() => done);
return text;
} }
export async function clearCache() { await init(); generation++; cache.clear(); await saveCache(); }
export async function getCacheSize() { await init(); return cache.size; } async function setConfig(config = {}) {
if (config.endpoint) {
gEndpoint = config.endpoint;
}
if (config.templateName) {
gTemplateName = config.templateName;
}
if (typeof config.customTemplate === "string") {
gCustomTemplate = config.customTemplate;
}
if (typeof config.customSystemPrompt === "string") {
gCustomSystemPrompt = config.customSystemPrompt;
}
if (config.aiParams && typeof config.aiParams === "object") {
for (let [k, v] of Object.entries(config.aiParams)) {
if (k in gAiParams && typeof v !== "undefined") {
gAiParams[k] = v;
}
}
}
if (typeof config.debugLogging === "boolean") {
setDebug(config.debugLogging);
}
if (gTemplateName === "custom") {
gTemplateText = gCustomTemplate;
} else if (Services?.tm?.spinEventLoopUntil) {
gTemplateText = loadTemplateSync(gTemplateName);
} else {
gTemplateText = await loadTemplate(gTemplateName);
}
aiLog(`[AiClassifier] Endpoint set to ${gEndpoint}`, {debug: true});
aiLog(`[AiClassifier] Template set to ${gTemplateName}`, {debug: true});
}
function buildSystemPrompt() {
return SYSTEM_PREFIX + (gCustomSystemPrompt || DEFAULT_CUSTOM_SYSTEM_PROMPT) + SYSTEM_SUFFIX;
}
function buildPrompt(body, criterion) {
aiLog(`[AiClassifier] Building prompt with criterion: "${criterion}"`, {debug: true});
const data = {
system: buildSystemPrompt(),
email: body,
query: criterion,
};
let template = gTemplateText || loadTemplateSync(gTemplateName);
return template.replace(/{{\s*(\w+)\s*}}/g, (m, key) => data[key] || "");
}
function getCachedResult(cacheKey) {
if (!gCacheLoaded) {
return null;
}
if (cacheKey && gCache.has(cacheKey)) {
aiLog(`[AiClassifier] Cache hit for key: ${cacheKey}`, {debug: true});
const entry = gCache.get(cacheKey);
return entry?.matched ?? null;
}
return null;
}
function getReason(cacheKey) {
if (!gCacheLoaded) {
return null;
}
const entry = gCache.get(cacheKey);
return cacheKey && entry ? entry.reason || null : null;
}
function buildPayload(text, criterion) {
let payloadObj = Object.assign({
prompt: buildPrompt(text, criterion)
}, gAiParams);
return JSON.stringify(payloadObj);
}
function parseMatch(result) {
const rawText = result.choices?.[0]?.text || "";
const thinkText = rawText.match(/<think>[\s\S]*?<\/think>/gi)?.join('') || '';
aiLog('[AiClassifier] ⮡ Reasoning:', {debug: true}, thinkText);
const cleanedText = rawText.replace(/<think>[\s\S]*?<\/think>/gi, "").trim();
aiLog('[AiClassifier] ⮡ Cleaned Response Text:', {debug: true}, cleanedText);
const obj = JSON.parse(cleanedText);
const matched = obj.matched === true || obj.match === true;
return { matched, reason: thinkText };
}
function cacheEntry(cacheKey, matched, reason) {
if (!cacheKey) {
return;
}
aiLog(`[AiClassifier] Caching entry '${cacheKey}'`, {debug: true});
const entry = gCache.get(cacheKey) || { matched: null, reason: "" };
if (typeof matched === "boolean") {
entry.matched = matched;
}
if (typeof reason === "string") {
entry.reason = reason;
}
gCache.set(cacheKey, entry);
saveCache(cacheKey, entry);
}
async function removeCacheEntries(keys = []) {
if (!Array.isArray(keys)) {
keys = [keys];
}
if (!gCacheLoaded) {
await loadCache();
}
let removed = false;
for (let key of keys) {
if (gCache.delete(key)) {
removed = true;
aiLog(`[AiClassifier] Removed cache entry '${key}'`, {debug: true});
}
}
if (removed) {
await saveCache();
}
}
async function clearCache() {
if (!gCacheLoaded) {
await loadCache();
}
if (gCache.size > 0) {
gCache.clear();
await saveCache();
aiLog(`[AiClassifier] Cleared cache`, {debug: true});
}
}
async function getCacheSize() {
if (!gCacheLoaded) {
await loadCache();
}
return gCache.size;
}
async function classifyText(text, criterion, cacheKey = null) {
if (!gCacheLoaded) {
await loadCache();
}
const cached = getCachedResult(cacheKey);
if (cached !== null) {
return cached;
}
const payload = buildPayload(text, criterion);
aiLog(`[AiClassifier] Sending classification request to ${gEndpoint}`, {debug: true});
aiLog(`[AiClassifier] Classification request payload:`, { debug: true }, payload);
try {
const response = await fetch(gEndpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: payload,
});
if (!response.ok) {
aiLog(`HTTP status ${response.status}`, {level: 'warn'});
return false;
}
const result = await response.json();
aiLog(`[AiClassifier] Received response:`, {debug: true}, result);
const parsed = parseMatch(result);
cacheEntry(cacheKey, parsed.matched, parsed.reason);
return parsed.matched;
} catch (e) {
aiLog(`HTTP request failed`, {level: 'error'}, e);
return false;
}
}
async function init() {
await loadCache();
}
export { classifyText, setConfig, removeCacheEntries, clearCache, getReason, getCachedResult, buildCacheKey, getCacheSize, init };

View file

@ -1,81 +0,0 @@
const SYSTEM_PREFIX = `You are an email-classification assistant.
Read the email below and the classification criterion provided by the user.
`;
const DEFAULT_CUSTOM_SYSTEM_PROMPT = "Determine whether the email satisfies the user's criterion.";
const SYSTEM_SUFFIX = `
Return ONLY a JSON object on a single line of the form:
{"match": true, "reason": "<short explanation>"} - if the email satisfies the criterion
{"match": false, "reason": "<short explanation>"} - otherwise
Do not add any other keys, text, or formatting.`;
function extractLastJsonObject(text) {
let last = null;
let start = -1;
let depth = 0;
let inString = false;
let escape = false;
for (let i = 0; i < text.length; i += 1) {
const ch = text[i];
if (inString) {
if (escape) {
escape = false;
continue;
}
if (ch === "\\") {
escape = true;
continue;
}
if (ch === "\"") {
inString = false;
}
continue;
}
if (ch === "\"") {
inString = true;
continue;
}
if (ch === "{") {
if (depth === 0) {
start = i;
}
depth += 1;
continue;
}
if (ch === "}" && depth > 0) {
depth -= 1;
if (depth === 0 && start !== -1) {
last = text.slice(start, i + 1);
start = -1;
}
}
}
return last;
}
export const path = '/v1/chat/completions';
export function buildRequest(text, criterion, config) {
const payload = {
messages: [
{ role: 'system', content: SYSTEM_PREFIX + (config.customSystemPrompt || DEFAULT_CUSTOM_SYSTEM_PROMPT) + SYSTEM_SUFFIX },
{ role: 'user', content: `**Email Contents**\n\`\`\`\n${text}\n\`\`\`\nClassification Criterion: ${criterion}` }
], ...config.aiParams
};
if (config.model) payload.model = config.model;
return payload;
}
export function parseResponse(result) {
const content = result?.choices?.[0]?.message?.content;
if (typeof content !== 'string') throw new Error('Chat response missing text content.');
const candidate = extractLastJsonObject(content);
if (!candidate) throw new Error('No JSON object found in AI response.');
const obj = JSON.parse(candidate);
const matched = Object.hasOwn(obj, 'match') ? obj.match : obj.matched;
if (typeof matched !== 'boolean') throw new Error('AI response missing valid match boolean.');
const reason = obj.reason ?? obj.reasoning ?? obj.explaination;
return { matched, reason: typeof reason === 'string' ? reason : null, probability: null };
}

View file

@ -1,22 +0,0 @@
export const path = '/v1/systemone';
export function buildRequest(text, criterion, config) {
return buildBatchRequest(text, [{ id: 'match', criterion }], config);
}
export function buildBatchRequest(text, questions, config) {
const payload = { state: text, questions: Object.fromEntries(questions.map(({ id, criterion }) =>
[id, { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` }])) };
if (config.model) payload.model = config.model;
return payload;
}
export function parseResponse(result) { return parseAnswer(result, 'match'); }
export function parseBatchResponse(result, questions) {
return questions.map(({ id }) => parseAnswer(result, id));
}
function parseAnswer(result, id) {
const answer = result?.answers?.[id];
if (answer?.type !== 'noul' || typeof answer.noul !== 'number' ||
!Number.isFinite(answer.noul) || answer.noul < 0 || answer.noul > 1) {
throw new Error(`SystemOne response must contain answers.${id} with type noul and a probability from 0 to 1.`);
}
return { matched: null, reason: null, probability: answer.noul };
}

View file

@ -1,39 +0,0 @@
import { DEFAULT_AI_PARAMS } from './defaultParams.js';
export const CONFIG_DEFAULTS = {
apiType: 'chat-completions', endpoint: 'http://127.0.0.1:5000', model: '',
apiKey: '', openaiOrganization: '', openaiProject: '', customSystemPrompt: '',
systemOneProcessAll: true, systemOneBatchSize: 8,
decisionThreshold: 0.5, inputCharacterLimit: 3686, aiParams: DEFAULT_AI_PARAMS,
htmlToMarkdown: false, stripUrlParams: false, altTextImages: false,
collapseWhitespace: false, tokenReduction: false, debugLogging: false
};
export function validateThreshold(value) {
if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 || value > 1) {
throw new Error('Match threshold must be between 0% and 100%.');
}
return value;
}
export function validateSettings(data) {
if (data.systemOneProcessAll !== undefined && typeof data.systemOneProcessAll !== 'boolean') {
throw new Error('Process all must be a boolean.');
}
if (data.systemOneBatchSize !== undefined && (!Number.isSafeInteger(data.systemOneBatchSize) || data.systemOneBatchSize < 1)) {
throw new Error('Rules per request must be a positive whole number.');
}
if (data.apiType !== undefined && !['chat-completions', 'systemone'].includes(data.apiType)) throw new Error('Unknown API type.');
if (data.decisionThreshold !== undefined) validateThreshold(data.decisionThreshold);
if (data.inputCharacterLimit !== undefined && (!Number.isSafeInteger(data.inputCharacterLimit) || data.inputCharacterLimit < 1)) {
throw new Error('Input character limit must be a positive integer.');
}
if (data.aiRules !== undefined) {
if (!Array.isArray(data.aiRules)) throw new Error('Rules must be an array.');
for (const rule of data.aiRules) {
if (rule.decisionThreshold !== undefined) validateThreshold(rule.decisionThreshold);
}
}
}
export function migratedInputLimit(data) {
if (data.inputCharacterLimit !== undefined) return data.inputCharacterLimit;
const value = Math.floor(Number(data.aiParams?.max_tokens ?? 4096) * 0.9);
return Number.isSafeInteger(value) && value > 0 ? value : 3686;
}

View file

@ -1,16 +1,10 @@
"use strict"; "use strict";
import { validateSettings, migratedInputLimit } from '../modules/classificationConfig.js';
import { validCacheEntry } from '../modules/AiClassifier.js';
const storage = (globalThis.messenger ?? browser).storage; const storage = (globalThis.messenger ?? browser).storage;
const KEY_GROUPS = { const KEY_GROUPS = {
settings: [ settings: [
'systemOneProcessAll', 'systemOneBatchSize',
'apiType', 'decisionThreshold', 'inputCharacterLimit', 'tokenReduction',
'endpoint', 'endpoint',
'model', 'templateName',
'apiKey', 'customTemplate',
'openaiOrganization',
'openaiProject',
'customSystemPrompt', 'customSystemPrompt',
'aiParams', 'aiParams',
'debugLogging', 'debugLogging',
@ -40,15 +34,12 @@ export async function exportData(categories) {
URL.revokeObjectURL(url); URL.revokeObjectURL(url);
} }
export async function importData(file, categories = Object.keys(KEY_GROUPS)) { export async function importData(file, categories) {
const text = await file.text(); const text = await file.text();
const parsed = JSON.parse(text); const parsed = JSON.parse(text);
const data = {}; const data = {};
for (const key of collectKeys(categories)) { for (const key of collectKeys(categories)) {
if (key in parsed) data[key] = parsed[key]; if (key in parsed) data[key] = parsed[key];
} }
if (categories.includes('settings') && data.inputCharacterLimit === undefined) data.inputCharacterLimit = migratedInputLimit(data);
validateSettings(data);
if (data.aiCache !== undefined) data.aiCache = Object.fromEntries(Object.entries(data.aiCache || {}).filter(([key, entry]) => validCacheEntry(key, entry)));
await storage.local.set(data); await storage.local.set(data);
} }

View file

@ -31,10 +31,6 @@
.tag { .tag {
--bulma-tag-h: 318; --bulma-tag-h: 318;
} }
#diff-display {
white-space: pre-wrap;
font-family: monospace;
}
</style> </style>
</head> </head>
<body> <body>
@ -51,8 +47,6 @@
<li class="is-active" data-tab="settings"><a><span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span><span>Settings</span></a></li> <li class="is-active" data-tab="settings"><a><span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span><span>Settings</span></a></li>
<li data-tab="rules"><a><span class="icon is-small"><img data-icon="clipboarddata" data-size="16" src="../resources/img/clipboarddata-light-16.png" alt=""></span><span>Rules</span></a></li> <li data-tab="rules"><a><span class="icon is-small"><img data-icon="clipboarddata" data-size="16" src="../resources/img/clipboarddata-light-16.png" alt=""></span><span>Rules</span></a></li>
<li data-tab="maintenance"><a><span class="icon is-small"><img data-icon="gear" data-size="16" src="../resources/img/gear-light-16.png" alt=""></span><span>Maintenance</span></a></li> <li data-tab="maintenance"><a><span class="icon is-small"><img data-icon="gear" data-size="16" src="../resources/img/gear-light-16.png" alt=""></span><span>Maintenance</span></a></li>
<li id="errors-tab-button" class="is-hidden" data-tab="errors"><a><span class="icon is-small"><img data-icon="x" data-size="16" src="../resources/img/x-light-16.png" alt=""></span><span>Errors</span></a></li>
<li id="debug-tab-button" class="is-hidden" data-tab="debug"><a><span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span><span>Debug</span></a></li>
</ul> </ul>
</div> </div>
</div> </div>
@ -64,55 +58,33 @@
</div> </div>
</div> </div>
<p class="help is-danger mb-3" id="settings-error" role="alert"></p>
<div id="settings-tab" class="tab-content"> <div id="settings-tab" class="tab-content">
<h2 class="title is-4"> <h2 class="title is-4">
<span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span> <span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span>
<span>Settings</span> <span>Settings</span>
</h2> </h2>
<div class="field">
<label class="label" for="api-type">API type</label>
<div class="select"><select id="api-type">
<option value="chat-completions">Chat Completions</option>
<option value="systemone">SystemOne</option>
</select></div>
</div>
<div class="field systemone-only">
<label class="checkbox"><input type="checkbox" id="systemone-process-all"> Process all</label>
<p class="help">Group all eligible, uncached rules for each email into one request. Actions still follow rule order and stop-after-match.</p>
</div>
<div class="field systemone-only">
<label class="label" for="systemone-batch-size">Rules per request</label>
<input class="input" id="systemone-batch-size" type="number" min="1" step="1" required>
<p class="help">Turn off Process all to limit each request. Use 1 for one rule at a time.</p>
</div>
<div class="field systemone-only">
<label class="label" for="decision-threshold">Default match probability threshold (%)</label>
<input class="input" id="decision-threshold" type="number" min="0" max="100" step="any">
<p class="help">Match when probability is at least this value. Rules may override it. Evaluate thresholds on your own email examples.</p>
</div>
<div class="field"> <div class="field">
<label class="label" for="endpoint">Endpoint</label> <label class="label" for="endpoint">Endpoint</label>
<div class="control"> <div class="control">
<input class="input" type="text" id="endpoint" placeholder="https://api.example.com"> <input class="input" type="text" id="endpoint" placeholder="https://api.example.com">
</div> </div>
<p class="help" id="endpoint-preview"></p>
</div> </div>
<div class="field"> <div class="field">
<label class="label" for="model-select">Model</label> <label class="label" for="template">Prompt template</label>
<div class="field has-addons"> <div class="control">
<div class="control is-expanded"> <div class="select is-fullwidth">
<div class="select is-fullwidth"> <select id="template"></select>
<select id="model-select"></select>
</div>
</div>
<div class="control">
<button class="button" id="refresh-models" type="button">Refresh</button>
</div> </div>
</div> </div>
<p class="help" id="model-help"></p> </div>
<div id="custom-template-container" class="field is-hidden">
<label class="label">Custom template</label>
<div class="control">
<textarea class="textarea" id="custom-template" rows="6" placeholder="Enter your custom template here..."></textarea>
</div>
<p class="help">Placeholders: {{system}}, {{email}}, {{query}}</p>
</div> </div>
<div class="field"> <div class="field">
@ -147,32 +119,6 @@
</div> </div>
<div id="advanced-options" class="mt-4 is-hidden"> <div id="advanced-options" class="mt-4 is-hidden">
<div class="field">
<label class="label" for="api-key">API key</label>
<div class="field has-addons">
<div class="control is-expanded">
<input class="input" type="password" id="api-key" placeholder="sk-...">
</div>
<div class="control">
<button class="button" id="toggle-api-key" type="button">Show</button>
</div>
</div>
<p class="help">Leave blank for unauthenticated endpoints.</p>
</div>
<div class="field">
<label class="label" for="openai-organization">OpenAI Organization</label>
<div class="control">
<input class="input" type="text" id="openai-organization" placeholder="org-...">
</div>
<p class="help">Optional header for OpenAI-hosted endpoints.</p>
</div>
<div class="field">
<label class="label" for="openai-project">OpenAI Project</label>
<div class="control">
<input class="input" type="text" id="openai-project" placeholder="proj_...">
</div>
<p class="help">Optional header for OpenAI-hosted endpoints.</p>
</div>
<div class="field"> <div class="field">
<label class="checkbox"> <label class="checkbox">
<input type="checkbox" id="debug-logging"> Enable debug logging <input type="checkbox" id="debug-logging"> Enable debug logging
@ -204,17 +150,13 @@
</label> </label>
</div> </div>
<div class="field"> <div class="field">
<label class="checkbox"> <label class="label" for="context-length">Context length</label>
<input type="checkbox" id="show-debug-tab"> Show debug information <div class="control">
</label> <input class="input" type="number" id="context-length">
</div>
</div> </div>
<div class="field"> <div class="field">
<label class="label" for="input-character-limit">Input character limit</label> <label class="label" for="max_tokens">Max tokens</label>
<input class="input" id="input-character-limit" type="number" min="1" step="1">
<p class="help">Applied with aggressive reduction. Characters are not tokens; leave room for instructions within your model's context limit.</p>
</div>
<div class="field">
<label class="label" for="max_tokens">Maximum output tokens</label>
<div class="control"> <div class="control">
<input class="input" type="number" id="max_tokens"> <input class="input" type="number" id="max_tokens">
</div> </div>
@ -309,16 +251,10 @@
<tr><th>Messages per day</th><td id="per-day">0</td></tr> <tr><th>Messages per day</th><td id="per-day">0</td></tr>
</tbody> </tbody>
</table> </table>
<div class="buttons"> <button class="button is-danger" id="clear-cache" type="button">
<button class="button is-danger" id="clear-cache" type="button"> <span class="icon is-small"><img data-icon="trash" data-size="16" src="../resources/img/trash-light-16.png" alt=""></span>
<span class="icon is-small"><img data-icon="trash" data-size="16" src="../resources/img/trash-light-16.png" alt=""></span> <span>Clear Cache</span>
<span>Clear Cache</span> </button>
</button>
<button class="button is-warning" id="reset-timing" type="button">
<span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span>
<span>Reset Timing Stats</span>
</button>
</div>
<div class="field mt-4"> <div class="field mt-4">
<label class="label">Data categories</label> <label class="label">Data categories</label>
<div class="control"> <div class="control">
@ -343,50 +279,8 @@
</p> </p>
</div> </div>
</div> </div>
<div id="errors-tab" class="tab-content is-hidden">
<h2 class="title is-4">
<span class="icon is-small"><img data-icon="x" data-size="16" src="../resources/img/x-light-16.png" alt=""></span>
<span>Session Errors</span>
</h2>
<div id="errors-empty" class="notification is-success is-light">
No errors have been recorded since the last start.
</div>
<div id="errors-panel" class="is-hidden">
<div class="box mb-4">
<div class="level">
<div class="level-left">
<div>
<p class="title is-5 mb-1">Error Log</p>
<p class="subtitle is-6">Visible only for this session.</p>
</div>
</div>
<div class="level-right">
<span class="tag is-danger is-light" id="errors-count">0</span>
</div>
</div>
</div>
<div id="errors-list"></div>
</div>
</div>
<div id="debug-tab" class="tab-content is-hidden">
<h2 class="title is-4">
<span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span>
<span>Debug</span>
</h2>
<pre id="payload-display"></pre>
<div id="diff-container" class="mt-4 is-hidden">
<div class="is-flex is-align-items-center is-justify-content-space-between">
<label class="label mb-0">Prompt diff</label>
<span id="prompt-reduction" class="tag is-info is-light is-hidden">Prompt Token Reduction: 0%</span>
</div>
<div id="diff-display" class="box content is-family-monospace"></div>
</div>
</div>
</div> </div>
</section> </section>
<script src="../resources/js/diff_match_patch_uncompressed.js"></script>
<script src="options.js"></script> <script src="options.js"></script>
</body> </body>
</html> </html>

View file

@ -5,16 +5,11 @@ document.addEventListener('DOMContentLoaded', async () => {
const dataTransfer = await import(browser.runtime.getURL('options/dataTransfer.js')); const dataTransfer = await import(browser.runtime.getURL('options/dataTransfer.js'));
const { detectSystemTheme } = await import(browser.runtime.getURL('modules/themeUtils.js')); const { detectSystemTheme } = await import(browser.runtime.getURL('modules/themeUtils.js'));
const { DEFAULT_AI_PARAMS } = await import(browser.runtime.getURL('modules/defaultParams.js')); const { DEFAULT_AI_PARAMS } = await import(browser.runtime.getURL('modules/defaultParams.js'));
const { validateSettings, migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js'));
const defaults = await storage.local.get([ const defaults = await storage.local.get([
'systemOneProcessAll', 'systemOneBatchSize',
'apiType', 'decisionThreshold', 'inputCharacterLimit',
'endpoint', 'endpoint',
'templateName',
'customTemplate',
'customSystemPrompt', 'customSystemPrompt',
'model',
'apiKey',
'openaiOrganization',
'openaiProject',
'aiParams', 'aiParams',
'debugLogging', 'debugLogging',
'htmlToMarkdown', 'htmlToMarkdown',
@ -22,13 +17,10 @@ document.addEventListener('DOMContentLoaded', async () => {
'altTextImages', 'altTextImages',
'collapseWhitespace', 'collapseWhitespace',
'tokenReduction', 'tokenReduction',
'contextLength',
'aiRules', 'aiRules',
'aiCache', 'aiCache',
'theme', 'theme'
'showDebugTab',
'lastPayload',
'lastFullText',
'lastPromptText'
]); ]);
const tabButtons = document.querySelectorAll('#main-tabs li'); const tabButtons = document.querySelectorAll('#main-tabs li');
const tabs = document.querySelectorAll('.tab-content'); const tabs = document.querySelectorAll('.tab-content');
@ -73,201 +65,36 @@ document.addEventListener('DOMContentLoaded', async () => {
} }
await applyTheme(themeSelect.value); await applyTheme(themeSelect.value);
const payloadDisplay = document.getElementById('payload-display');
const diffDisplay = document.getElementById('diff-display');
const diffContainer = document.getElementById('diff-container');
const promptReductionLabel = document.getElementById('prompt-reduction');
let lastFullText = defaults.lastFullText || '';
let lastPromptText = defaults.lastPromptText || '';
let lastPayload = defaults.lastPayload ? JSON.stringify(defaults.lastPayload, null, 2) : '';
if (lastPayload) {
payloadDisplay.textContent = lastPayload;
}
themeSelect.addEventListener('change', async () => { themeSelect.addEventListener('change', async () => {
markDirty(); markDirty();
await applyTheme(themeSelect.value); await applyTheme(themeSelect.value);
}); });
const apiTypeInput = document.getElementById('api-type'); document.getElementById('endpoint').value = defaults.endpoint || 'http://127.0.0.1:5000/v1/completions';
apiTypeInput.value = defaults.apiType || 'chat-completions';
const processAllInput = document.getElementById('systemone-process-all'); const templates = {
const batchSizeInput = document.getElementById('systemone-batch-size'); openai: browser.i18n.getMessage('template.openai'),
processAllInput.checked = defaults.systemOneProcessAll ?? true; qwen: browser.i18n.getMessage('template.qwen'),
let savedBatchSize = defaults.systemOneBatchSize ?? 8; mistral: browser.i18n.getMessage('template.mistral'),
batchSizeInput.value = savedBatchSize; custom: browser.i18n.getMessage('template.custom')
function updateBatchControls() { };
batchSizeInput.disabled = processAllInput.checked || apiTypeInput.value !== 'systemone'; const templateSelect = document.getElementById('template');
for (const [value, label] of Object.entries(templates)) {
const opt = document.createElement('option');
opt.value = value;
opt.textContent = label;
templateSelect.appendChild(opt);
} }
processAllInput.addEventListener('change', updateBatchControls); templateSelect.value = defaults.templateName || 'openai';
const thresholdInput = document.getElementById('decision-threshold');
thresholdInput.value = (defaults.decisionThreshold ?? 0.5) * 100; const customBox = document.getElementById('custom-template-container');
const inputLimit = document.getElementById('input-character-limit'); const customTemplate = document.getElementById('custom-template');
inputLimit.value = migratedInputLimit(defaults); customTemplate.value = defaults.customTemplate || '';
const settingsError = document.getElementById('settings-error');
function updateApiControls() { function updateVisibility() {
const decision = apiTypeInput.value === 'systemone'; customBox.classList.toggle('is-hidden', templateSelect.value !== 'custom');
updateBatchControls();
document.querySelectorAll('.systemone-only').forEach(el => el.classList.toggle('is-hidden', !decision));
for (const id of ['system-instructions', 'openai-organization', 'openai-project', ...Object.keys(DEFAULT_AI_PARAMS)]) {
document.getElementById(id)?.closest('.field')?.classList.toggle('is-hidden', decision);
}
document.getElementById('reset-system').classList.toggle('is-hidden', decision);
updateEndpointPreview();
} }
apiTypeInput.addEventListener('change', updateApiControls); templateSelect.addEventListener('change', updateVisibility);
const endpointInput = document.getElementById('endpoint'); updateVisibility();
const endpointPreview = document.getElementById('endpoint-preview');
const fallbackEndpoint = 'http://127.0.0.1:5000';
const storedEndpoint = defaults.endpoint || fallbackEndpoint;
const endpointBase = AiClassifier.normalizeEndpointBase(storedEndpoint) || storedEndpoint;
endpointInput.value = endpointBase;
function updateEndpointPreview() {
const resolved = AiClassifier.buildEndpointUrl(endpointInput.value, apiTypeInput.value);
endpointPreview.textContent = resolved
? `Resolved endpoint: ${resolved}`
: 'Resolved endpoint: (invalid)';
}
endpointInput.addEventListener('input', updateEndpointPreview);
updateApiControls();
const modelSelect = document.getElementById('model-select');
const refreshModelsBtn = document.getElementById('refresh-models');
const modelHelp = document.getElementById('model-help');
const storedModel = typeof defaults.model === 'string' ? defaults.model : '';
function setModelHelp(message = '', isError = false) {
if (!modelHelp) return;
modelHelp.textContent = message;
modelHelp.classList.toggle('is-danger', isError);
}
function populateModelOptions(models = [], selectedModel = '') {
if (!modelSelect) return;
const modelIds = Array.isArray(models) ? models.filter(Boolean) : [];
modelSelect.innerHTML = '';
const noneOpt = document.createElement('option');
noneOpt.value = '';
noneOpt.textContent = 'None (omit model)';
modelSelect.appendChild(noneOpt);
if (selectedModel && !modelIds.includes(selectedModel)) {
const storedOpt = document.createElement('option');
storedOpt.value = selectedModel;
storedOpt.textContent = `Stored: ${selectedModel}`;
modelSelect.appendChild(storedOpt);
}
for (const id of modelIds) {
const opt = document.createElement('option');
opt.value = id;
opt.textContent = id;
modelSelect.appendChild(opt);
}
const hasSelected = [...modelSelect.options].some(opt => opt.value === selectedModel);
modelSelect.value = hasSelected ? selectedModel : '';
}
function buildAuthHeaders() {
const headers = {};
const apiKey = apiKeyInput?.value.trim();
if (apiKey) {
headers.Authorization = `Bearer ${apiKey}`;
}
const organization = openaiOrgInput?.value.trim();
if (organization && apiTypeInput.value === 'chat-completions') {
headers["OpenAI-Organization"] = organization;
}
const project = openaiProjectInput?.value.trim();
if (project && apiTypeInput.value === 'chat-completions') {
headers["OpenAI-Project"] = project;
}
return headers;
}
async function fetchModels(preferredModel = '') {
if (!modelSelect || !refreshModelsBtn) return;
const modelsUrl = AiClassifier.buildModelsUrl(endpointInput.value);
const selectedModel = preferredModel || modelSelect.value;
if (!modelsUrl) {
logger.aiLog('[options] model refresh skipped: invalid endpoint', { level: 'warn' }, {
endpoint: endpointInput.value
});
setModelHelp('Set a valid endpoint to load models.', true);
populateModelOptions([], selectedModel);
return;
}
refreshModelsBtn.disabled = true;
setModelHelp('Loading models...');
const headers = buildAuthHeaders();
logger.aiLog('[options] loading models', {}, {
endpoint: endpointInput.value,
modelsUrl,
selectedModel,
hasAuthorization: typeof headers.Authorization === 'string',
hasOrganization: typeof headers['OpenAI-Organization'] === 'string',
hasProject: typeof headers['OpenAI-Project'] === 'string'
});
try {
const response = await fetch(modelsUrl, { method: 'GET', headers });
logger.aiLog('[options] model refresh response received', {}, {
status: response.status,
ok: response.ok,
contentType: response.headers.get('content-type') || ''
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const data = await response.json();
logger.aiLog('[options] model refresh payload parsed', {}, {
payloadType: Array.isArray(data) ? 'array' : typeof data,
hasDataArray: Array.isArray(data?.data),
hasModelsArray: Array.isArray(data?.models),
topLevelKeys: data && typeof data === 'object' && !Array.isArray(data)
? Object.keys(data)
: []
});
let models = [];
if (Array.isArray(data?.data)) {
models = data.data.map(model => model?.id ?? model?.name ?? model?.model ?? '').filter(Boolean);
} else if (Array.isArray(data?.models)) {
models = data.models.map(model => model?.id ?? model?.name ?? model?.model ?? '').filter(Boolean);
} else if (Array.isArray(data)) {
models = data.map(model => model?.id ?? model?.name ?? model?.model ?? model).filter(Boolean);
}
models = [...new Set(models)];
logger.aiLog('[options] model refresh parsed models', {}, {
count: models.length,
models
});
populateModelOptions(models, selectedModel);
setModelHelp(models.length ? `Loaded ${models.length} model${models.length === 1 ? '' : 's'}.` : 'No models returned.');
} catch (e) {
logger.aiLog('[options] failed to load models', { level: 'warn' }, e);
const message = String(e?.message || e || '');
const isLikelyCors = e instanceof TypeError || /Failed to fetch|NetworkError|Load failed/i.test(message);
setModelHelp(
isLikelyCors
? 'Failed to load models. The server responded, but Thunderbird may have blocked access to the response. Check CORS on the AI server and verify the endpoint is reachable from the add-on.'
: 'Failed to load models. Check the endpoint, HTTP status, and network.',
true
);
populateModelOptions([], selectedModel);
} finally {
refreshModelsBtn.disabled = false;
}
}
populateModelOptions([], storedModel);
refreshModelsBtn?.addEventListener('click', () => {
logger.aiLog('[options] model refresh button clicked');
fetchModels(modelSelect.value);
});
const advancedBox = document.getElementById('advanced-options'); const advancedBox = document.getElementById('advanced-options');
const advancedBtn = document.getElementById('toggle-advanced'); const advancedBtn = document.getElementById('toggle-advanced');
@ -275,26 +102,6 @@ document.addEventListener('DOMContentLoaded', async () => {
advancedBox.classList.toggle('is-hidden'); advancedBox.classList.toggle('is-hidden');
}); });
const apiKeyInput = document.getElementById('api-key');
const apiKeyToggle = document.getElementById('toggle-api-key');
const openaiOrgInput = document.getElementById('openai-organization');
const openaiProjectInput = document.getElementById('openai-project');
if (apiKeyInput) {
apiKeyInput.value = typeof defaults.apiKey === 'string' ? defaults.apiKey : '';
}
if (openaiOrgInput) {
openaiOrgInput.value = typeof defaults.openaiOrganization === 'string' ? defaults.openaiOrganization : '';
}
if (openaiProjectInput) {
openaiProjectInput.value = typeof defaults.openaiProject === 'string' ? defaults.openaiProject : '';
}
apiKeyToggle?.addEventListener('click', () => {
if (!apiKeyInput) return;
const show = apiKeyInput.type === 'password';
apiKeyInput.type = show ? 'text' : 'password';
apiKeyToggle.textContent = show ? 'Hide' : 'Show';
});
const debugToggle = document.getElementById('debug-logging'); const debugToggle = document.getElementById('debug-logging');
debugToggle.checked = defaults.debugLogging === true; debugToggle.checked = defaults.debugLogging === true;
@ -313,140 +120,8 @@ document.addEventListener('DOMContentLoaded', async () => {
const tokenReductionToggle = document.getElementById('token-reduction'); const tokenReductionToggle = document.getElementById('token-reduction');
tokenReductionToggle.checked = defaults.tokenReduction === true; tokenReductionToggle.checked = defaults.tokenReduction === true;
function tokenSavingEnabled() { const contextLengthInput = document.getElementById('context-length');
return htmlToggle.checked contextLengthInput.value = defaults.contextLength || 16384;
|| stripUrlToggle.checked
|| altTextToggle.checked
|| collapseWhitespaceToggle.checked
|| tokenReductionToggle.checked;
}
function updatePromptReductionLabel(hasDiff) {
if (!promptReductionLabel) return;
if (!hasDiff || !tokenSavingEnabled() || !lastFullText || !lastPromptText) {
promptReductionLabel.classList.add('is-hidden');
return;
}
const baseLength = lastFullText.length;
const promptLength = lastPromptText.length;
const percentSaved = baseLength > 0
? Math.max(0, Math.round((1 - (promptLength / baseLength)) * 100))
: 0;
promptReductionLabel.textContent = `Prompt Token Reduction: ${percentSaved}%`;
promptReductionLabel.classList.remove('is-hidden');
}
function updateDiffDisplay() {
if (lastFullText && lastPromptText && diff_match_patch) {
const dmp = new diff_match_patch();
dmp.Diff_EditCost = 4;
const diffs = dmp.diff_main(lastFullText, lastPromptText);
dmp.diff_cleanupEfficiency(diffs);
const hasDiff = diffs.some(d => d[0] !== 0);
if (hasDiff) {
diffDisplay.innerHTML = dmp.diff_prettyHtml(diffs);
diffContainer.classList.remove('is-hidden');
} else {
diffDisplay.innerHTML = '';
diffContainer.classList.add('is-hidden');
}
updatePromptReductionLabel(hasDiff);
} else {
diffDisplay.innerHTML = '';
diffContainer.classList.add('is-hidden');
updatePromptReductionLabel(false);
}
}
const debugTabToggle = document.getElementById('show-debug-tab');
const debugTabBtn = document.getElementById('debug-tab-button');
const errorTabBtn = document.getElementById('errors-tab-button');
const errorsEmpty = document.getElementById('errors-empty');
const errorsPanel = document.getElementById('errors-panel');
const errorsList = document.getElementById('errors-list');
const errorsCount = document.getElementById('errors-count');
function updateDebugTab() {
const visible = debugTabToggle.checked;
debugTabBtn.classList.toggle('is-hidden', !visible);
}
debugTabToggle.checked = defaults.showDebugTab === true;
debugTabToggle.addEventListener('change', () => { updateDebugTab(); markDirty(); });
updateDebugTab();
function formatErrorTime(value) {
try {
return new Date(value).toLocaleString();
} catch (e) {
return '';
}
}
function renderErrors(entries = []) {
const hasErrors = entries.length > 0;
errorTabBtn.classList.toggle('is-hidden', !hasErrors);
errorsEmpty.classList.toggle('is-hidden', hasErrors);
errorsPanel.classList.toggle('is-hidden', !hasErrors);
errorsList.innerHTML = '';
errorsCount.textContent = String(entries.length);
if (!hasErrors) {
return;
}
entries.forEach(entry => {
const card = document.createElement('article');
card.className = 'message is-danger is-light mb-4';
const header = document.createElement('div');
header.className = 'message-header';
const title = document.createElement('p');
title.textContent = entry.context || 'Error';
const time = document.createElement('span');
time.className = 'is-size-7 has-text-weight-normal';
time.textContent = formatErrorTime(entry.time);
header.appendChild(title);
header.appendChild(time);
const body = document.createElement('div');
body.className = 'message-body';
const summary = document.createElement('p');
summary.className = 'mb-2';
summary.textContent = entry.message || 'Unknown error';
body.appendChild(summary);
if (entry.detail) {
const detail = document.createElement('pre');
detail.className = 'is-family-monospace is-size-7';
detail.textContent = entry.detail;
body.appendChild(detail);
}
card.appendChild(header);
card.appendChild(body);
errorsList.appendChild(card);
});
}
async function loadErrors() {
try {
const response = await browser.runtime.sendMessage({ type: 'sortana:getErrorLog' });
renderErrors(response?.errors || []);
} catch (e) {
renderErrors([]);
}
}
browser.runtime.onMessage.addListener((msg) => {
if (msg?.type === 'sortana:errorLogUpdated') {
loadErrors();
}
});
await loadErrors();
updateDiffDisplay();
await fetchModels(storedModel);
[htmlToggle, stripUrlToggle, altTextToggle, collapseWhitespaceToggle, tokenReductionToggle].forEach(toggle => {
toggle.addEventListener('change', () => {
updatePromptReductionLabel(!diffContainer.classList.contains('is-hidden'));
});
});
const aiParams = Object.assign({}, DEFAULT_AI_PARAMS, defaults.aiParams || {}); const aiParams = Object.assign({}, DEFAULT_AI_PARAMS, defaults.aiParams || {});
for (const [key, val] of Object.entries(aiParams)) { for (const [key, val] of Object.entries(aiParams)) {
@ -497,7 +172,7 @@ document.addEventListener('DOMContentLoaded', async () => {
const perDayEl = document.getElementById('per-day'); const perDayEl = document.getElementById('per-day');
let timingLogged = false; let timingLogged = false;
ruleCountEl.textContent = (defaults.aiRules || []).length; ruleCountEl.textContent = (defaults.aiRules || []).length;
cacheCountEl.textContent = '…'; cacheCountEl.textContent = defaults.aiCache ? Object.keys(defaults.aiCache).length : 0;
function createActionRow(action = {type: 'tag'}) { function createActionRow(action = {type: 'tag'}) {
const row = document.createElement('div'); const row = document.createElement('div');
@ -891,25 +566,14 @@ document.addEventListener('DOMContentLoaded', async () => {
body.appendChild(folderBox); body.appendChild(folderBox);
article.appendChild(header); article.appendChild(header);
const thresholdField = document.createElement('label');
thresholdField.className = 'field systemone-only';
thresholdField.classList.toggle('is-hidden', apiTypeInput.value !== 'systemone');
thresholdField.textContent = 'Match probability threshold (%) — blank uses default';
const override = document.createElement('input');
override.className = 'input decision-threshold';
override.type = 'number';
override.min = '0'; override.max = '100'; override.step = 'any';
override.value = rule.decisionThreshold === undefined ? '' : rule.decisionThreshold * 100;
thresholdField.appendChild(override);
body.appendChild(thresholdField);
article.appendChild(body); article.appendChild(body);
rulesContainer.appendChild(article); rulesContainer.appendChild(article);
} }
} }
function readRules() { addRuleBtn.addEventListener('click', () => {
return [...rulesContainer.querySelectorAll('.rule')].map(ruleEl => { const data = [...rulesContainer.querySelectorAll('.rule')].map(ruleEl => {
const criterion = ruleEl.querySelector('.criterion').value; const criterion = ruleEl.querySelector('.criterion').value;
const actions = [...ruleEl.querySelectorAll('.action-row')].map(row => { const actions = [...ruleEl.querySelectorAll('.action-row')].map(row => {
const type = row.querySelector('select').value; const type = row.querySelector('select').value;
@ -931,11 +595,8 @@ document.addEventListener('DOMContentLoaded', async () => {
if (type === 'flag') { if (type === 'flag') {
return { type, flagged: row.querySelector('.flag-select').value === 'true' }; return { type, flagged: row.querySelector('.flag-select').value === 'true' };
} }
if (type === 'forward') { if (type === 'delete' || type === 'archive') {
return { type, address: row.querySelector('.forward-input').value.trim() }; return { type };
}
if (type === 'reply') {
return { type, replyType: row.querySelector('.reply-select').value };
} }
return { type }; return { type };
}); });
@ -947,8 +608,6 @@ document.addEventListener('DOMContentLoaded', async () => {
const accounts = [...(ruleEl.querySelector('.account-select')?.selectedOptions || [])].map(o => o.value); const accounts = [...(ruleEl.querySelector('.account-select')?.selectedOptions || [])].map(o => o.value);
const folders = [...(ruleEl.querySelector('.folder-filter-select')?.selectedOptions || [])].map(o => o.value); const folders = [...(ruleEl.querySelector('.folder-filter-select')?.selectedOptions || [])].map(o => o.value);
const rule = { criterion, actions, enabled }; const rule = { criterion, actions, enabled };
const override = ruleEl.querySelector('.decision-threshold').value;
if (override.trim() !== '') rule.decisionThreshold = Number(override) / 100;
if (unreadOnly) rule.unreadOnly = true; if (unreadOnly) rule.unreadOnly = true;
if (stopProcessing) rule.stopProcessing = true; if (stopProcessing) rule.stopProcessing = true;
if (!isNaN(minAgeDays)) rule.minAgeDays = minAgeDays; if (!isNaN(minAgeDays)) rule.minAgeDays = minAgeDays;
@ -957,9 +616,6 @@ document.addEventListener('DOMContentLoaded', async () => {
if (folders.length) rule.folders = folders; if (folders.length) rule.folders = folders;
return rule; return rule;
}); });
}
addRuleBtn.addEventListener('click', () => {
const data = readRules();
data.push({ criterion: '', actions: [], unreadOnly: false, stopProcessing: false, enabled: true, accounts: [], folders: [] }); data.push({ criterion: '', actions: [], unreadOnly: false, stopProcessing: false, enabled: true, accounts: [], folders: [] });
renderRules(data); renderRules(data);
}); });
@ -976,7 +632,6 @@ document.addEventListener('DOMContentLoaded', async () => {
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo }); if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
if (r.copyTarget || r.copyTo) actions.push({ type: 'copy', copyTarget: r.copyTarget || r.copyTo }); if (r.copyTarget || r.copyTo) actions.push({ type: 'copy', copyTarget: r.copyTarget || r.copyTo });
const rule = { criterion: r.criterion, actions }; const rule = { criterion: r.criterion, actions };
if (r.decisionThreshold !== undefined) rule.decisionThreshold = r.decisionThreshold;
if (r.stopProcessing) rule.stopProcessing = true; if (r.stopProcessing) rule.stopProcessing = true;
if (r.unreadOnly) rule.unreadOnly = true; if (r.unreadOnly) rule.unreadOnly = true;
if (typeof r.minAgeDays === 'number') rule.minAgeDays = r.minAgeDays; if (typeof r.minAgeDays === 'number') rule.minAgeDays = r.minAgeDays;
@ -1046,27 +701,11 @@ document.addEventListener('DOMContentLoaded', async () => {
} }
try { try {
const { count } = await browser.runtime.sendMessage({ type: 'sortana:getCacheSize' }); const { aiCache } = await storage.local.get('aiCache');
cacheCountEl.textContent = count; cacheCountEl.textContent = aiCache ? Object.keys(aiCache).length : 0;
} catch { } catch {
cacheCountEl.textContent = '?'; cacheCountEl.textContent = '?';
} }
try {
if (debugTabToggle.checked) {
const latest = await storage.local.get(['lastPayload', 'lastFullText', 'lastPromptText']);
const payloadStr = latest.lastPayload ? JSON.stringify(latest.lastPayload, null, 2) : '';
if (payloadStr !== lastPayload) {
lastPayload = payloadStr;
payloadDisplay.textContent = payloadStr;
}
if (latest.lastFullText !== lastFullText || latest.lastPromptText !== lastPromptText) {
lastFullText = latest.lastFullText || '';
lastPromptText = latest.lastPromptText || '';
updateDiffDisplay();
}
}
} catch {}
} }
refreshMaintenance(); refreshMaintenance();
@ -1077,11 +716,6 @@ document.addEventListener('DOMContentLoaded', async () => {
cacheCountEl.textContent = '0'; cacheCountEl.textContent = '0';
}); });
document.getElementById('reset-timing').addEventListener('click', async () => {
await browser.runtime.sendMessage({ type: 'sortana:resetTimingStats' });
await refreshMaintenance();
});
function selectedCategories() { function selectedCategories() {
return [...document.querySelectorAll('.transfer-category:checked')].map(el => el.value); return [...document.querySelectorAll('.transfer-category:checked')].map(el => el.value);
} }
@ -1094,35 +728,17 @@ document.addEventListener('DOMContentLoaded', async () => {
document.getElementById('import-data').addEventListener('click', () => importInput.click()); document.getElementById('import-data').addEventListener('click', () => importInput.click());
importInput.addEventListener('change', async () => { importInput.addEventListener('change', async () => {
if (importInput.files.length) { if (importInput.files.length) {
try { await dataTransfer.importData(importInput.files[0], selectedCategories());
await dataTransfer.importData(importInput.files[0], selectedCategories()); location.reload();
location.reload();
} catch (error) {
window.alert(`Import failed: ${error.message}`);
}
} }
}); });
initialized = true; initialized = true;
document.getElementById('save').addEventListener('click', async () => { document.getElementById('save').addEventListener('click', async () => {
settingsError.textContent = ''; const endpoint = document.getElementById('endpoint').value;
const apiType = apiTypeInput.value; const templateName = templateSelect.value;
const systemOneProcessAll = processAllInput.checked; const customTemplateText = customTemplate.value;
const enteredBatchSize = Number(batchSizeInput.value);
const validBatchSize = Number.isSafeInteger(enteredBatchSize) && enteredBatchSize > 0;
if (!batchSizeInput.disabled && !validBatchSize) {
settingsError.textContent = 'Rules per request must be a positive whole number.';
return;
}
const systemOneBatchSize = validBatchSize ? enteredBatchSize : savedBatchSize;
const decisionThreshold = thresholdInput.value.trim() === '' ? NaN : Number(thresholdInput.value) / 100;
const inputCharacterLimit = Number(inputLimit.value);
const endpoint = endpointInput.value.trim();
const model = modelSelect?.value || '';
const apiKey = apiKeyInput?.value.trim() || '';
const openaiOrganization = openaiOrgInput?.value.trim() || '';
const openaiProject = openaiProjectInput?.value.trim() || '';
const customSystemPrompt = systemBox.value; const customSystemPrompt = systemBox.value;
const aiParamsSave = {}; const aiParamsSave = {};
for (const key of Object.keys(DEFAULT_AI_PARAMS)) { for (const key of Object.keys(DEFAULT_AI_PARAMS)) {
@ -1134,27 +750,62 @@ document.addEventListener('DOMContentLoaded', async () => {
} }
const debugLogging = debugToggle.checked; const debugLogging = debugToggle.checked;
const htmlToMarkdown = htmlToggle.checked; const htmlToMarkdown = htmlToggle.checked;
const invalidThreshold = [...document.querySelectorAll('.decision-threshold'), thresholdInput, inputLimit].find(el => !el.checkValidity()); const rules = [...rulesContainer.querySelectorAll('.rule')].map(ruleEl => {
if (invalidThreshold) { const criterion = ruleEl.querySelector('.criterion').value;
settingsError.textContent = 'Enter thresholds from 0% to 100% and a positive whole-number character limit.'; const actions = [...ruleEl.querySelectorAll('.action-row')].map(row => {
return; const type = row.querySelector('select').value;
} if (type === 'tag') {
const rules = readRules().filter(r => r.criterion); return { type, tagKey: row.querySelector('.tag-select').value };
}
if (type === 'move') {
return { type, folder: row.querySelector('.folder-select').value };
}
if (type === 'copy') {
return { type, copyTarget: row.querySelector('.folder-select').value };
}
if (type === 'junk') {
return { type, junk: row.querySelector('.junk-select').value === 'true' };
}
if (type === 'read') {
return { type, read: row.querySelector('.read-select').value === 'true' };
}
if (type === 'flag') {
return { type, flagged: row.querySelector('.flag-select').value === 'true' };
}
if (type === 'forward') {
return { type, address: row.querySelector('.forward-input').value.trim() };
}
if (type === 'reply') {
return { type, replyType: row.querySelector('.reply-select').value };
}
return { type };
});
const stopProcessing = ruleEl.querySelector('.stop-processing')?.checked;
const unreadOnly = ruleEl.querySelector('.unread-only')?.checked;
const enabled = ruleEl.dataset.enabled !== 'false';
const minAgeDays = parseFloat(ruleEl.querySelector('.min-age')?.value);
const maxAgeDays = parseFloat(ruleEl.querySelector('.max-age')?.value);
const accounts = [...(ruleEl.querySelector('.account-select')?.selectedOptions || [])].map(o => o.value);
const folders = [...(ruleEl.querySelector('.folder-filter-select')?.selectedOptions || [])].map(o => o.value);
const rule = { criterion, actions, enabled };
if (unreadOnly) rule.unreadOnly = true;
if (stopProcessing) rule.stopProcessing = true;
if (!isNaN(minAgeDays)) rule.minAgeDays = minAgeDays;
if (!isNaN(maxAgeDays)) rule.maxAgeDays = maxAgeDays;
if (accounts.length) rule.accounts = accounts;
if (folders.length) rule.folders = folders;
return rule;
}).filter(r => r.criterion);
const stripUrlParams = stripUrlToggle.checked; const stripUrlParams = stripUrlToggle.checked;
const altTextImages = altTextToggle.checked; const altTextImages = altTextToggle.checked;
const collapseWhitespace = collapseWhitespaceToggle.checked; const collapseWhitespace = collapseWhitespaceToggle.checked;
const tokenReduction = tokenReductionToggle.checked; const tokenReduction = tokenReductionToggle.checked;
const showDebugTab = debugTabToggle.checked; const contextLength = parseInt(contextLengthInput.value) || 0;
const theme = themeSelect.value; const theme = themeSelect.value;
try { validateSettings({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, aiRules: rules }); } await storage.local.set({ endpoint, templateName, customTemplate: customTemplateText, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, contextLength, aiRules: rules, theme });
catch (error) { settingsError.textContent = error.message; return; }
await storage.local.set({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging, htmlToMarkdown, stripUrlParams, altTextImages, collapseWhitespace, tokenReduction, aiRules: rules, theme, showDebugTab });
savedBatchSize = systemOneBatchSize;
batchSizeInput.value = savedBatchSize;
await storage.local.remove(['templateName', 'customTemplate']);
await applyTheme(theme); await applyTheme(theme);
try { try {
await AiClassifier.setConfig({ systemOneProcessAll, systemOneBatchSize, apiType, decisionThreshold, inputCharacterLimit, endpoint, model, apiKey, openaiOrganization, openaiProject, customSystemPrompt, aiParams: aiParamsSave, debugLogging }); await AiClassifier.setConfig({ endpoint, templateName, customTemplate: customTemplateText, customSystemPrompt, aiParams: aiParamsSave, debugLogging });
logger.setDebug(debugLogging); logger.setDebug(debugLogging);
} catch (e) { } catch (e) {
logger.aiLog('[options] failed to apply config', {level: 'error'}, e); logger.aiLog('[options] failed to apply config', {level: 'error'}, e);

View file

@ -0,0 +1,12 @@
[INST] {{system}}
Email:
{{email}}
Criterion: {{query}}
Remember, return ONLY a JSON object on a single line of the form:
{"match": true} - if the email satisfies the criterion
{"match": false} - otherwise
Do not add any other keys, text, or formatting.
[/INST]

View file

@ -0,0 +1,14 @@
<|im_start|>system
{{system}}<|im_end|>
<|im_start|>user
**Email Contents**
```
{{email}}
```
Classification Criterion: {{query}}
Remember, return ONLY a JSON object on a single line of the form:
{"match": true} - if the email satisfies the criterion
{"match": false} - otherwise
Do not add any other keys, text, or formatting.<|im_end|>
<|im_start|>assistant

15
prompt_templates/qwen.txt Normal file
View file

@ -0,0 +1,15 @@
<|im_start|>system
{{system}}
<|im_end|>
<|im_start|>user
Email:
{{email}}
Criterion: {{query}}
Remember, return ONLY a JSON object on a single line of the form:
{"match": true} - if the email satisfies the criterion
{"match": false} - otherwise
Do not add any other keys, text, or formatting.
<|im_end|>
<|im_start|>assistant

Binary file not shown.

Before

Width:  |  Height:  |  Size: 413 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 828 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 408 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 813 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.7 KiB

File diff suppressed because it is too large Load diff

View file

@ -1,94 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
svg_dir="$script_dir/svg"
out_dir="$script_dir/img"
sizes=(16 32 64)
themes=(light dark)
get_theme_color() {
case "$1" in
light) printf '%s\n' '#000000' ;;
dark) printf '%s\n' '#ffffff' ;;
*)
printf 'Unknown theme: %s\n' "$1" >&2
exit 1
;;
esac
}
ensure_temp_svg() {
local input_svg="$1"
local color="$2"
local temp_svg="$3"
python3 - "$input_svg" "$color" "$temp_svg" <<'PY'
import pathlib
import re
import sys
input_path = pathlib.Path(sys.argv[1])
color = sys.argv[2]
output_path = pathlib.Path(sys.argv[3])
content = input_path.read_text(encoding="utf-8")
patched, count = re.subn(r"<svg([^>]*?)>", f'<svg\\1 style="color: {color}">', content, count=1)
if count != 1:
raise SystemExit("Couldn't find <svg> tag to patch.")
output_path.write_text(patched, encoding="utf-8")
PY
}
if [[ ! -d "$svg_dir" ]]; then
echo "SVG source directory not found: $svg_dir" >&2
exit 1
fi
mkdir -p "$out_dir"
if ! command -v inkscape >/dev/null 2>&1; then
echo "Inkscape CLI is not installed or not in PATH. Please install it from https://inkscape.org/" >&2
exit 1
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "python3 is required to patch the SVG color attribute." >&2
exit 1
fi
shopt -s nullglob
svg_files=("$svg_dir"/*.svg)
shopt -u nullglob
if [[ ${#svg_files[@]} -eq 0 ]]; then
echo "No SVG files found in $svg_dir" >&2
exit 0
fi
temp_svg="$(mktemp "${TMPDIR:-/tmp}/sortana-svg2img.XXXXXX.svg")"
cleanup() {
rm -f "$temp_svg"
}
trap cleanup EXIT
for svg_path in "${svg_files[@]}"; do
base_name="$(basename "${svg_path%.svg}")"
for theme in "${themes[@]}"; do
color="$(get_theme_color "$theme")"
ensure_temp_svg "$svg_path" "$color" "$temp_svg"
for size in "${sizes[@]}"; do
out_file="$out_dir/$base_name-$theme-$size.png"
echo "Exporting $out_file (color $color)..."
inkscape "$temp_svg" \
--export-type=png \
--export-filename="$out_file" \
--export-width="$size" \
--export-height="$size" \
--actions=export-do
done
done
done
echo "Done generating light/dark themed PNGs."

View file

@ -1,219 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import vm from 'node:vm';
import { readFile } from 'node:fs/promises';
import { CONFIG_DEFAULTS, validateSettings } from '../modules/classificationConfig.js';
import * as systemone from '../modules/backends/systemOne.js';
const data = {};
let cacheWrites = 0;
globalThis.messenger = { storage: { local: {
async get(keys) { return Object.fromEntries((Array.isArray(keys) ? keys : [keys]).map(key => [key, data[key]])); },
async set(values) { Object.assign(data, structuredClone(values)); if ('aiCache' in values) cacheWrites++; },
async remove() {}
} }, messages: { async get(id) { return { headerMessageId: `mail-${id}` }; } } };
const classifier = await import('../modules/AiClassifier.js');
const transfer = await import('../options/dataTransfer.js');
let requests = [];
async function setup(update = {}) {
await classifier.clearCache();
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone', ...update });
requests = []; cacheWrites = 0;
globalThis.fetch = async (url, request) => {
const body = JSON.parse(request.body);
requests.push({ url, body, headers: request.headers });
// Intentionally reverse response order and include an unsolicited answer.
return { ok: true, json: async () => ({ model: 'served', answers: {
unsolicited: { type: 'noul', noul: 1 },
...Object.fromEntries(Object.keys(body.questions).reverse().map(id => [id, { type: 'noul', noul: 0.75 }]))
} }) };
};
return classifier.getConfig();
}
async function candidates(count, snapshot, criteria = []) {
return Promise.all(Array.from({ length: count }, async (_, i) => {
const criterion = criteria[i] ?? `criterion ${i}`;
return { id: `rule_${i}`, rule: { criterion, actions: [] }, cacheKey: await classifier.buildCacheKey(1, criterion, snapshot) };
}));
}
async function collect(items, snapshot) {
const results = [];
for await (const value of classifier.classifyRules('email state', items, snapshot)) results.push(value);
return results;
}
test('Process all, explicit batch sizes, partial batches, empty input and cache reuse', async () => {
for (const [settings, expected] of [
[{}, [10]], [{ systemOneProcessAll: false, systemOneBatchSize: 8 }, [8, 2]],
[{ systemOneProcessAll: false, systemOneBatchSize: 1 }, Array(10).fill(1)]
]) {
const snapshot = await setup(settings);
const items = await candidates(10, snapshot);
const results = await collect(items, snapshot);
assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), expected);
assert.equal(cacheWrites, expected.length);
assert.deepEqual(results.map(r => r.rule.criterion), items.map(r => r.rule.criterion));
assert.ok(results.every(r => r.result.probability === 0.75));
assert.ok(requests.every(r => r.body.state === 'email state' && !('messages' in r.body)));
await collect(items, snapshot);
assert.equal(requests.length, expected.length);
await collect([], snapshot);
assert.equal(requests.length, expected.length);
}
});
test('cached rules consume no capacity and duplicate criteria use independent thresholds', async () => {
const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 });
const items = await candidates(5, snapshot, ['cached', 'same', 'cached', 'other', 'same']);
await collect([items[0]], snapshot);
requests = []; cacheWrites = 0;
items[1].rule.decisionThreshold = 0.8;
items[4].rule.decisionThreshold = 0.7;
const results = await collect(items, snapshot);
assert.equal(requests.length, 1);
assert.deepEqual(Object.keys(requests[0].body.questions), ['rule_1', 'rule_3']);
assert.equal(results[1].result.matched, false);
assert.equal(results[4].result.matched, true);
assert.equal(cacheWrites, 1);
assert.equal(await classifier.buildCacheKey(1, 'same', { ...snapshot, systemOneProcessAll: true, systemOneBatchSize: 100 }), items[1].cacheKey);
});
test('answer identity determines probability, not answer order', () => {
const result = { answers: { rule_2: { type: 'noul', noul: 0 }, rule_1: { type: 'noul', noul: 1 } } };
assert.deepEqual(systemone.parseBatchResponse(result, [{ id: 'rule_1' }, { id: 'rule_2' }]).map(r => r.probability), [1, 0]);
});
test('failed batches neither yield nor cache partial results and do not retry', async () => {
for (const failure of ['missing', 'invalid', 'http']) {
const snapshot = await setup();
const items = await candidates(2, snapshot);
let calls = 0, yielded = 0;
globalThis.fetch = async () => {
calls++;
return { ok: failure !== 'http', status: 413, json: async () => ({ answers: {
rule_0: { type: 'noul', noul: 0.9 },
...(failure === 'invalid' ? { rule_1: { type: 'noul', noul: '0.9' } } : {})
} }) };
};
await assert.rejects(async () => {
for await (const value of classifier.classifyRules('state', items, snapshot)) yielded++;
});
assert.equal(yielded, 0);
assert.equal(calls, 1);
assert.equal(cacheWrites, 0);
assert.equal(await classifier.getCacheSize(), 0);
}
});
test('clearing or importing during a batch prevents stale cache writes', async () => {
for (const replace of [() => classifier.clearCache(), () => classifier.replaceCache({})]) {
const snapshot = await setup();
const items = await candidates(2, snapshot);
const fetch = globalThis.fetch;
globalThis.fetch = async (...args) => { await replace(); return fetch(...args); };
const results = await collect(items, snapshot);
assert.equal(results.length, 2);
assert.equal(requests.length, 1);
assert.equal(await classifier.getCacheSize(), 0);
}
});
test('Chat Completions remains lazy and sequential', async () => {
const snapshot = await setup({ apiType: 'chat-completions' });
const items = await candidates(3, snapshot);
globalThis.fetch = async (url, request) => {
requests.push(JSON.parse(request.body));
return { ok: true, json: async () => ({ choices: [{ message: { content: '{"match":true}' } }] }) };
};
for await (const value of classifier.classifyRules('state', items, snapshot)) break;
assert.equal(requests.length, 1);
assert.ok(Array.isArray(requests[0].messages));
});
test('batch settings validate and survive import', async () => {
for (const value of [0, -1, 1.2, Infinity, '8', null]) assert.throws(() => validateSettings({ systemOneBatchSize: value }));
assert.throws(() => validateSettings({ systemOneProcessAll: 'true' }));
await transfer.importData({ text: async () => JSON.stringify({ systemOneProcessAll: false, systemOneBatchSize: 3 }) }, ['settings']);
assert.equal(data.systemOneProcessAll, false);
assert.equal(data.systemOneBatchSize, 3);
await assert.rejects(transfer.importData({ text: async () => '{"systemOneBatchSize":0}' }, ['settings']));
});
// Exercise the action runner with the real lazy classifier and mocked mail APIs.
const source = await readFile(new URL('../background.js', import.meta.url), 'utf8');
const functions = source.slice(source.indexOf('async function processMessage('), source.indexOf('async function clearCacheForMessages('));
test('stop after match skips later actions and batches; all-mode retains later evidence', async () => {
for (const processAll of [true, false]) {
const snapshot = await setup({ systemOneProcessAll: processAll, systemOneBatchSize: 2 });
const actions = [], errors = [];
const rules = Array.from({ length: 5 }, (_, i) => ({ criterion: `rule ${i}`, actions: [{ type: 'flag', flagged: true }], stopProcessing: i === 1 }));
const context = vm.createContext({
structuredClone, Date, Error, aiRules: rules, AiClassifier: classifier,
queue: Promise.resolve(), queuedCount: 0, processing: false, currentStart: 0,
tokenReduction: false, showDebugTab: false, timingStats: {},
updateActionIcon() {}, updateTimingStats() {}, showTransientIcon() {},
buildEmailText() { return 'state'; }, ICONS: { circle: 'circle' },
logger: { aiLog() {} }, recordError(...args) { errors.push(args); }, ERROR_NOTIFICATION_ID: 'error',
storage: messenger.storage,
messenger: { messages: {
async getFull() { return {}; }, async get() { return { tags: [], folder: { accountId: 'account' } }; },
async update(id, value) { actions.push({ id, value }); }
}, identities: { async list() { return []; } } },
browser: { runtime: { getURL: x => x }, notifications: { async create() {} } }
});
vm.runInContext(functions, context);
await vm.runInContext('applyAiRules([1])', context);
assert.equal(errors.length, 0);
assert.equal(actions.length, 2);
assert.equal(requests.length, 1);
assert.equal(Object.keys(requests[0].body.questions).length, processAll ? 5 : 2);
assert.equal(await classifier.getCacheSize(), processAll ? 5 : 2);
}
});
test('batch controls default to all and retain numeric input across toggles and API switches', async () => {
const optionsSource = await readFile(new URL('../options/options.js', import.meta.url), 'utf8');
const section = optionsSource.slice(optionsSource.indexOf(' const apiTypeInput ='), optionsSource.indexOf(' const endpointInput ='));
const fields = new Map();
const field = id => {
if (!fields.has(id)) fields.set(id, {
value: '', checked: false, disabled: false, listeners: {},
addEventListener(event, fn) { this.listeners[event] = fn; },
classList: { toggle() {} }, closest() { return this; }
});
return fields.get(id);
};
const context = vm.createContext({
defaults: { apiType: 'systemone' }, DEFAULT_AI_PARAMS: {}, migratedInputLimit() { return 3686; },
updateEndpointPreview() {}, document: { getElementById: field, querySelectorAll() { return []; } }
});
vm.runInContext(`${section}\nupdateApiControls();`, context);
const all = field('systemone-process-all'), size = field('systemone-batch-size'), api = field('api-type');
assert.equal(all.checked, true);
assert.equal(size.value, 8);
assert.equal(size.disabled, true);
all.checked = false; all.listeners.change();
assert.equal(size.disabled, false);
size.value = '3';
all.checked = true; all.listeners.change();
assert.equal(size.disabled, true);
all.checked = false; all.listeners.change();
assert.equal(size.value, '3');
api.value = 'chat-completions'; api.listeners.change();
assert.equal(size.disabled, true);
api.value = 'systemone'; api.listeners.change();
assert.equal(size.disabled, false);
assert.equal(size.value, '3');
});
test('batch settings remain fixed while a per-email snapshot is running', async () => {
const snapshot = await setup({ systemOneProcessAll: false, systemOneBatchSize: 2 });
const items = await candidates(5, snapshot);
const fetch = globalThis.fetch;
globalThis.fetch = async (...args) => {
classifier.setConfig({ systemOneProcessAll: true, systemOneBatchSize: 1 });
return fetch(...args);
};
await collect(items, snapshot);
assert.deepEqual(requests.map(r => Object.keys(r.body.questions).length), [2, 2, 1]);
});

View file

@ -1,86 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import vm from 'node:vm';
import { readFile } from 'node:fs/promises';
import { CONFIG_DEFAULTS } from '../modules/classificationConfig.js';
const persisted = {};
const listeners = [];
const storage = {
local: {
async get(keys) {
return Object.fromEntries((Array.isArray(keys) ? keys : [keys])
.filter(key => key in persisted).map(key => [key, structuredClone(persisted[key])]));
},
async set(values) {
const changes = {};
for (const [key, value] of Object.entries(values)) {
changes[key] = { oldValue: persisted[key], newValue: structuredClone(value) };
persisted[key] = structuredClone(value);
}
for (const listener of listeners) await listener(changes, 'local');
},
async remove(keys) {
for (const key of Array.isArray(keys) ? keys : [keys]) delete persisted[key];
}
},
onChanged: { addListener(listener) { listeners.push(listener); } }
};
const messenger = { storage, messages: { async get(id) {
// Numeric Thunderbird IDs may change after reinstall; RFC Message-ID does not.
return { subject: 'Invoice', headerMessageId: 'persistent-message@example.com' };
} } };
globalThis.messenger = messenger;
const classifier = await import('../modules/AiClassifier.js');
const transfer = await import('../options/dataTransfer.js');
const source = await readFile(new URL('../background.js', import.meta.url), 'utf8');
let onMessage;
const context = vm.createContext({
storage, messenger, structuredClone, AiClassifier: classifier, aiRules: [],
logger: { aiLog() {} }, logGetTiming: false,
htmlToMarkdown: false, stripUrlParams: false, altTextImages: false,
collapseWhitespace: false, tokenReduction: false, showDebugTab: false,
browser: { runtime: { onMessage: { addListener(listener) { onMessage = listener; } } } }
});
// Exercise the actual background storage and Details handlers, not a manual cache reload.
const normalizeStart = source.indexOf('function normalizeRules(');
vm.runInContext(source.slice(normalizeStart, source.indexOf('function iconPaths(', normalizeStart)), context);
const changesStart = source.indexOf('storage.onChanged.addListener(');
vm.runInContext(source.slice(changesStart, source.indexOf('\n\n if (browser.theme', changesStart)), context);
const messageStart = source.indexOf('browser.runtime.onMessage.addListener(');
vm.runInContext(source.slice(messageStart, source.indexOf('\n\n // Automatically classify', messageStart)), context);
test('export/reinstall/import restores Details immediately with changed Thunderbird IDs', async () => {
const settings = { ...CONFIG_DEFAULTS, systemOneProcessAll: false, systemOneBatchSize: 3, apiType: 'systemone', endpoint: 'http://decision-server:8080', tokenReduction: true, inputCharacterLimit: 5000 };
const rules = [{ criterion: 'invoice', decisionThreshold: 0.8, actions: [] }];
await storage.local.set({ ...settings, aiRules: rules });
globalThis.fetch = async () => ({ ok: true, json: async () => ({ answers: { match: { type: 'noul', noul: 0.9 } } }) });
const key = await classifier.buildCacheKey(1, 'invoice');
await classifier.classify('Invoice attached', 'invoice', { cacheKey: key, threshold: 0.8 });
let exported;
const originalDocument = globalThis.document;
const createURL = URL.createObjectURL;
const revokeURL = URL.revokeObjectURL;
URL.createObjectURL = blob => { exported = blob; return 'blob:export'; };
URL.revokeObjectURL = () => {};
globalThis.document = { body: { appendChild() {} }, createElement() { return { click() {}, remove() {} }; } };
try { await transfer.exportData(['settings', 'rules', 'cache']); }
finally { globalThis.document = originalDocument; URL.createObjectURL = createURL; URL.revokeObjectURL = revokeURL; }
await classifier.clearCache();
for (const key of Object.keys(persisted)) delete persisted[key];
classifier.setConfig(CONFIG_DEFAULTS);
context.aiRules = [];
assert.equal((await onMessage({ type: 'sortana:getDetails', id: 99 })).results.length, 0);
await transfer.importData(exported, ['settings', 'rules', 'cache']);
assert.equal((await onMessage({ type: 'sortana:getCacheSize' })).count, 1);
assert.equal(classifier.getConfig().systemOneProcessAll, false);
assert.equal(classifier.getConfig().systemOneBatchSize, 3);
const details = await onMessage({ type: 'sortana:getDetails', id: 99 });
assert.equal(details.results.length, 1);
assert.equal(details.results[0].criterion, 'invoice');
assert.equal(details.results[0].probability, 0.9);
assert.equal(details.results[0].threshold, 0.8);
assert.equal(details.results[0].matched, true);
globalThis.fetch = () => { throw new Error('Restored cache must avoid inference'); };
assert.equal((await classifier.classify('Invoice attached', 'invoice', { cacheKey: await classifier.buildCacheKey(99, 'invoice') })).matched, true);
});

View file

@ -1,167 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import * as chat from '../modules/backends/chatCompletions.js';
import * as decision from '../modules/backends/systemOne.js';
import { CONFIG_DEFAULTS, migratedInputLimit, validateSettings } from '../modules/classificationConfig.js';
const data = { aiCache: { legacy: { matched: true, reason: 'old' } }, aiReasonCache: { legacy: 'old' } };
globalThis.messenger = { storage: { local: {
async get(keys) { return Object.fromEntries((Array.isArray(keys) ? keys : [keys]).map(key => [key, data[key]])); },
async set(values) { Object.assign(data, structuredClone(values)); },
async remove(keys) { for (const key of Array.isArray(keys) ? keys : [keys]) delete data[key]; }
} }, messages: { async get(id) { return { headerMessageId: `message-${id}` }; } } };
const classifier = await import('../modules/AiClassifier.js');
const transfer = await import('../options/dataTransfer.js');
const answer = probability => ({ model: 'served-model', answers: { match: { type: 'noul', noul: probability } } });
let calls = [];
function serve(body, status = 200) {
globalThis.fetch = async (url, request) => { calls.push({ url, ...request }); return { ok: status === 200, status, json: async () => body }; };
}
test('adapters preserve chat prompts and isolate SystemOne payloads', () => {
const config = { ...CONFIG_DEFAULTS, model: 'chosen', customSystemPrompt: 'Custom guidance' };
const payload = chat.buildRequest('email', 'invoice', config);
assert.match(payload.messages[0].content, /Custom guidance/);
assert.match(payload.messages[1].content, /Classification Criterion: invoice/);
assert.equal(payload.max_tokens, 4096);
assert.deepEqual(decision.buildRequest('email', 'invoice', config), {
state: 'email', model: 'chosen', questions: { match: { type: 'noul', instructions: 'Does this email satisfy the following criterion: invoice' } }
});
assert.equal('model' in decision.buildRequest('email', 'invoice', CONFIG_DEFAULTS), false);
assert.equal('model' in chat.buildRequest('email', 'invoice', CONFIG_DEFAULTS), false);
});
test('parsers validate evidence including zero, one, and malformed answers', () => {
for (const probability of [0, 0.5, 1]) assert.equal(decision.parseResponse(answer(probability)).probability, probability);
for (const probability of [null, '0.5', -0.1, 1.1, Infinity, NaN]) assert.throws(() => decision.parseResponse(answer(probability)));
for (const raw of [null, {}, { answers: { other: { type: 'noul', noul: 0.5 } } }, { answers: { match: { type: 'choice', noul: 0.5 } } }]) assert.throws(() => decision.parseResponse(raw));
assert.deepEqual(chat.parseResponse({ choices: [{ message: { content: 'text {"match":false} then {"matched":true,"reason":"brace } inside"}' } }] }), { matched: true, reason: 'brace } inside', probability: null });
for (const content of ['', '{broken}', '{"match":"true"}']) assert.throws(() => chat.parseResponse({ choices: [{ message: { content } }] }));
assert.throws(() => chat.parseResponse(null));
});
test('URLs preserve proxy prefixes and normalize API suffixes', () => {
for (const suffix of ['', '/', '/v1', '/v1/', '/v1/models', '/v1/chat/completions/', '/v1/systemone']) {
assert.equal(classifier.buildEndpointUrl(`http://localhost:8080/proxy${suffix}`, 'systemone'), 'http://localhost:8080/proxy/v1/systemone');
assert.equal(classifier.buildModelsUrl(`http://localhost:8080/proxy${suffix}`), 'http://localhost:8080/proxy/v1/models');
}
assert.equal(classifier.buildEndpointUrl('example.com'), 'https://example.com/v1/chat/completions');
assert.equal(classifier.buildEndpointUrl(''), '');
});
test('settings validation and independent character-limit migration', () => {
assert.equal(migratedInputLimit({}), 3686);
assert.equal(migratedInputLimit({ aiParams: { max_tokens: 1000 } }), 900);
assert.equal(migratedInputLimit({ inputCharacterLimit: 500, aiParams: { max_tokens: 1000 } }), 500);
validateSettings({ decisionThreshold: 0, aiRules: [{ decisionThreshold: 1 }] });
for (const value of [-1, 2, null, '0.5']) assert.throws(() => validateSettings({ aiRules: [{ decisionThreshold: value }] }));
for (const value of [0, 1.2, NaN]) assert.throws(() => validateSettings({ inputCharacterLimit: value }));
});
test('cache migration, inference reuse, fingerprints, clearing and import', async () => {
await classifier.init();
assert.equal(await classifier.getCacheSize(), 0);
assert.equal(data.aiReasonCache, undefined);
assert.deepEqual(data.aiCache, {});
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone', endpoint: 'http://localhost:8080', apiKey: 'secret', openaiOrganization: 'org', openaiProject: 'project' });
const snapshot = classifier.getConfig();
const key = await classifier.buildCacheKey(1, 'invoice', snapshot);
serve(answer(0.75)); calls = [];
const first = await classifier.classify('email', 'invoice', { snapshot, cacheKey: key, threshold: 0.75 });
assert.equal(first.matched, true);
assert.equal(first.model, 'served-model');
assert.deepEqual(calls[0].headers, { 'Content-Type': 'application/json', Authorization: 'Bearer secret' });
const second = await classifier.classify('email', 'invoice', { snapshot, cacheKey: key, threshold: 0.8 });
assert.equal(second.matched, false);
assert.equal(second.probability, 0.75);
assert.equal(calls.length, 1);
classifier.replaceCache(data.aiCache);
assert.equal(classifier.getCachedDetails(key, 0.75).matched, true);
assert.equal(await classifier.buildCacheKey(1, 'invoice', { ...snapshot, decisionThreshold: 0.9, apiKey: 'new secret' }), key);
for (const change of [{ apiType: 'chat-completions' }, { endpoint: 'http://other' }, { model: 'other' }, { tokenReduction: true }, { inputCharacterLimit: 500 }]) {
assert.notEqual(await classifier.buildCacheKey(1, 'invoice', { ...snapshot, ...change }), key);
}
const chatSnapshot = { ...snapshot, apiType: 'chat-completions' };
const chatKey = await classifier.buildCacheKey(1, 'invoice', chatSnapshot);
assert.notEqual(await classifier.buildCacheKey(1, 'invoice', { ...chatSnapshot, customSystemPrompt: 'other' }), chatKey);
assert.notEqual(await classifier.buildCacheKey(1, 'invoice', { ...chatSnapshot, aiParams: { ...snapshot.aiParams, temperature: 1 } }), chatKey);
serve({ choices: [{ message: { content: '{"match":false,"reason":"not invoice"}' } }] });
await classifier.classify('email', 'invoice', { cacheKey: chatKey, snapshot: chatSnapshot });
assert.equal(calls.at(-1).headers['OpenAI-Organization'], 'org');
assert.equal(await classifier.getCacheSize(), 2);
const exported = structuredClone(data.aiCache);
await classifier.clearCacheForMessage(1);
assert.equal(await classifier.getCacheSize(), 0);
await transfer.importData({ text: async () => JSON.stringify({ aiCache: { ...exported, legacy: true } }) }, ['cache']);
classifier.replaceCache(data.aiCache); // background storage listener consumes imported replacement
assert.equal(await classifier.getCacheSize(), 2);
await assert.rejects(transfer.importData({ text: async () => JSON.stringify({ decisionThreshold: 2 }) }, ['settings']));
await assert.rejects(transfer.importData({ text: async () => JSON.stringify({ aiRules: [{ decisionThreshold: -1 }] }) }, ['rules']));
await classifier.clearCache();
});
test('failures remain uncached and snapshots survive configuration changes', async () => {
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone' });
const snapshot = classifier.getConfig();
const key = await classifier.buildCacheKey(2, 'invoice', snapshot);
for (const raw of [null, answer('0.9'), {}]) {
serve(raw);
await assert.rejects(classifier.classify('email', 'invoice', { cacheKey: key, snapshot }));
assert.equal(classifier.getCachedDetails(key), null);
}
serve({}, 503);
await assert.rejects(classifier.classify('email', 'invoice', { cacheKey: key, snapshot }), /503/);
globalThis.fetch = async () => { throw new Error('network'); };
await assert.rejects(classifier.classify('email', 'invoice', { cacheKey: key, snapshot }), /network/);
globalThis.fetch = async () => ({ ok: true, json: async () => { throw new SyntaxError('bad JSON'); } });
await assert.rejects(classifier.classify('email', 'invoice', { cacheKey: key, snapshot }), /bad JSON/);
classifier.setConfig({ apiType: 'chat-completions', endpoint: 'http://changed' });
serve(answer(1)); calls = [];
const result = await classifier.classify('email', 'invoice', { cacheKey: key, snapshot });
assert.equal(result.backend, 'systemone');
assert.equal(calls[0].url, 'http://127.0.0.1:5000/v1/systemone');
assert.equal('Authorization' in calls[0].headers, false);
await classifier.clearCache();
});
test('clearing or replacing cache during inference prevents stale repopulation', async () => {
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone' });
const snapshot = classifier.getConfig();
const key = await classifier.buildCacheKey(3, 'invoice', snapshot);
for (const replace of [() => classifier.clearCache(), () => classifier.replaceCache({})]) {
let release, started;
const ready = new Promise(resolve => { started = resolve; });
globalThis.fetch = async () => {
started();
await new Promise(resolve => { release = resolve; });
return { ok: true, json: async () => answer(0.9) };
};
const pending = classifier.classify('email', 'invoice', { cacheKey: key, snapshot });
await ready;
await replace();
release();
assert.equal((await pending).matched, true);
assert.equal(classifier.getCachedDetails(key), null);
}
});
test('settings and rule imports retain thresholds and reject invalid caches', async () => {
const settings = { apiType: 'systemone', decisionThreshold: 0.75, inputCharacterLimit: 5000, tokenReduction: true };
const rules = [{ criterion: 'invoice', decisionThreshold: 0, actions: [{ type: 'flag', flagged: true }] }, { criterion: 'urgent', actions: [] }];
await transfer.importData({ text: async () => JSON.stringify({ ...settings, aiRules: rules }) }, ['settings', 'rules']);
for (const [key, value] of Object.entries(settings)) assert.equal(data[key], value);
assert.deepEqual(data.aiRules, rules);
const key = await classifier.buildCacheKey(4, 'invoice');
assert.equal(classifier.validCacheEntry(key, { version: 2, backend: 'systemone', probability: '0.8', reason: null, messageHash: key.split(':')[1], model: '' }), false);
});
test('SystemOne 404 reports endpoint, server message, and compatibility guidance', async () => {
classifier.setConfig({ ...CONFIG_DEFAULTS, apiType: 'systemone', endpoint: 'http://seinei.wageshouse:5003' });
serve({ error: { message: 'File Not Found' } }, 404);
await assert.rejects(classifier.classify('synthetic email', 'invoice'), error => {
assert.match(error.message, /HTTP 404 at http:\/\/seinei\.wageshouse:5003\/v1\/systemone/);
assert.match(error.message, /Server response: File Not Found/);
assert.match(error.message, /running server build supports/);
return true;
});
});

View file

@ -1,83 +0,0 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import vm from 'node:vm';
import { readFile } from 'node:fs/promises';
// Run the actual background rule/queue functions with Thunderbird services mocked.
const source = await readFile(new URL('../background.js', import.meta.url), 'utf8');
const functions = source.slice(source.indexOf('async function processMessage('), source.indexOf('async function clearCacheForMessages('));
function runner({ fail = false, changeSettings = false } = {}) {
const actions = [], evaluated = [], errors = [];
const rules = [
{ criterion: 'first', decisionThreshold: 0.8, actions: [{ type: 'flag', flagged: true }], stopProcessing: true },
{ criterion: 'second', actions: [{ type: 'read', read: true }] }
];
let currentConfig = { apiType: 'systemone', inputCharacterLimit: 5, decisionThreshold: 0.5 };
const context = vm.createContext({
structuredClone, Date, Error,
aiRules: rules, queue: Promise.resolve(), queuedCount: 0, processing: false,
currentStart: 0, tokenReduction: true, showDebugTab: false, timingStats: {},
updateActionIcon() {}, updateTimingStats() {}, showTransientIcon() {},
buildEmailText() { return 'email contents'; }, ICONS: { circle: 'circle' },
logger: { aiLog() {} }, recordError(...args) { errors.push(args); },
ERROR_NOTIFICATION_ID: 'error',
storage: { local: { async set() {} } },
messenger: {
messages: {
async getFull() { return {}; },
async get() {
if (changeSettings) {
rules[0].decisionThreshold = 0.1;
currentConfig = { apiType: 'chat-completions', decisionThreshold: 0.1 };
}
return { tags: [], folder: { accountId: 'account' } };
},
async update(id, action) { actions.push({ id, ...action }); }
}, identities: { async list() { return []; } }
},
browser: { runtime: { getURL: x => x }, notifications: { async create() {} } },
AiClassifier: {
getConfig() { return structuredClone(currentConfig); },
async buildCacheKey(id, criterion) { return `${id}:${criterion}`; },
async *classifyRules(text, candidates, snapshot) {
for (const { rule, cacheKey } of candidates) {
evaluated.push({ text, criterion: rule.criterion, cacheKey, snapshot,
threshold: rule.decisionThreshold ?? snapshot.decisionThreshold });
if (fail && cacheKey.startsWith('1:')) throw new Error('inference failed');
yield { rule, result: { matched: true } };
}
}
}
});
vm.runInContext(functions, context);
return { context, actions, evaluated, errors };
}
test('failure stops remaining rules for one email, next queued email still acts', async () => {
const { context, actions, evaluated, errors } = runner({ fail: true });
await vm.runInContext('applyAiRules([1, 2])', context);
assert.deepEqual(evaluated.map(x => x.cacheKey), ['1:first', '2:first']);
assert.deepEqual(actions, [{ id: 2, flagged: true }]);
assert.equal(errors.length, 1);
assert.equal(context.processing, false);
assert.equal(context.queuedCount, 0);
});
test('settings and rules remain fixed during an email; reduction uses characters', async () => {
const { context, evaluated, actions } = runner({ changeSettings: true });
await vm.runInContext('applyAiRules([1])', context);
assert.equal(evaluated[0].text, 'email');
assert.equal(evaluated[0].threshold, 0.8);
assert.equal(evaluated[0].snapshot.apiType, 'systemone');
assert.equal(evaluated.length, 1); // stop after first match
assert.equal(actions.length, 1);
});
test('inherited thresholds and eligibility preserve ordered rule processing', async () => {
const { context, evaluated, actions } = runner();
context.aiRules[0].enabled = false;
await vm.runInContext('applyAiRules([1])', context);
assert.equal(evaluated[0].criterion, 'second');
assert.equal(evaluated[0].threshold, 0.5);
assert.deepEqual(actions, [{ id: 1, read: true }]);
});