Compare commits
0
.codex
114
AGENTS.md
|
|
@ -5,15 +5,12 @@ This file provides guidelines for codex agents contributing to the Sortana proje
|
|||
## Repository Overview
|
||||
|
||||
- `background.js`: Handles startup tasks and coordinates message passing within the extension.
|
||||
- `modules/`: Contains reusable JavaScript modules such as `AiClassifier.js`,
|
||||
`defaultParams.js` and `themeUtils.js`.
|
||||
- `options/`: The options page HTML, JavaScript and bundled Bulma CSS (v1.0.3).
|
||||
- `details.html` and `details.js`: View classification evidence and clear cache for a message.
|
||||
- `modules/`: Holds reusable JavaScript modules for the extension.
|
||||
- `content/`: Scripts for modifying Thunderbird windows (e.g., the filter editor).
|
||||
- `options/`: The options page HTML and JavaScript.
|
||||
- `resources/`: Images and other static files.
|
||||
- `prompt_templates/`: Prompt template files for the AI service.
|
||||
- `build-xpi.ps1`: PowerShell script to package the extension.
|
||||
- `build-xpi.sh`: Bash script to package the extension.
|
||||
- `resources/svg2img.ps1`: PowerShell script to regenerate themed PNG icons from SVGs.
|
||||
- `resources/svg2img.sh`: Bash script to regenerate themed PNG icons from SVGs.
|
||||
|
||||
## Coding Style
|
||||
|
||||
|
|
@ -30,65 +27,7 @@ This file provides guidelines for codex agents contributing to the Sortana proje
|
|||
|
||||
## Testing
|
||||
|
||||
Run the dependency-free suite with Node.js 22 or newer: `node --test tests/*.test.mjs`.
|
||||
The tests mock extension storage, HTTP responses, and Thunderbird rule execution.
|
||||
Manual verification in Thunderbird is still required for UI and real-server integration.
|
||||
Do **not** run the `ps1` build script or the SVG processing script.
|
||||
|
||||
## Endpoint Notes
|
||||
|
||||
Sortana supports `apiType: "chat-completions"` (default) and `"systemone"`.
|
||||
Plain adapter modules under `modules/backends/` own request construction and response
|
||||
validation; `AiClassifier` owns shared transport, configuration snapshots, threshold
|
||||
policy, and caching. The stored endpoint is a base URL. Normalization accepts `/v1`
|
||||
and recognized full API paths, preserves proxy prefixes, and defaults to HTTPS
|
||||
when no scheme is supplied. Model discovery uses `/v1/models`; **None** omits `model`.
|
||||
Bearer authentication is optional for both APIs. Organization/project headers are
|
||||
sent only for Chat Completions.
|
||||
|
||||
Chat Completions sends system/user messages and extracts the last JSON object from
|
||||
`choices[0].message.content`, requiring a `match` or `matched` boolean. SystemOne
|
||||
sends prepared email text as `state` with `noul` questions identified by stable
|
||||
`rule_<snapshot index>` IDs. The single-question API still uses `match` for backward
|
||||
compatibility and cache fingerprinting. Every requested answer must have type
|
||||
`noul` and a finite numeric probability in `[0, 1]`; response order is irrelevant.
|
||||
No chat generation parameters or system prompt are sent to SystemOne.
|
||||
|
||||
The global `decisionThreshold` defaults to 0.5; optional rule-level
|
||||
`decisionThreshold` overrides it. UI percentages are stored as numbers in `[0, 1]`.
|
||||
Matching uses `probability >= threshold` before display rounding. Details displays
|
||||
probabilities in a progress bar with the percentage inside; threshold and model
|
||||
appear in its tooltip. Results use the current threshold, never fabricated reasoning or action
|
||||
history. No secondary model calls are made.
|
||||
|
||||
`classify()` returns structured evidence; `classifyText()` remains a boolean wrapper.
|
||||
Failures throw and are never cached. HTTP errors include the request URL and
|
||||
JSON server error message when available; SystemOne 404 errors suggest checking
|
||||
the running server build and decision-model support. The rule runner stops that email, reports the
|
||||
error, and continues the next queued email. Snapshot settings and rules per email.
|
||||
`inputCharacterLimit` replaces coupling input truncation to output `max_tokens`;
|
||||
it applies with aggressive reduction and migrates from the old effective limit.
|
||||
|
||||
### SystemOne batching
|
||||
|
||||
`systemOneProcessAll` defaults to true; `systemOneBatchSize` defaults to 8 and must
|
||||
be a positive safe integer. Only SystemOne shows these controls. The numeric input
|
||||
is disabled while Process all is enabled, retaining its value. Include both in
|
||||
settings import/export, startup loading, live updates, and per-email snapshots.
|
||||
|
||||
`classifyRules()` is a lazy ordered async iterator over precomputed candidate keys.
|
||||
Chat requests stay sequential. SystemOne batches the next N unique uncached keys
|
||||
(or all), sends state once, validates the whole response, and persists one batch
|
||||
of evidence before yielding ordered results. Per-rule thresholds remain independent.
|
||||
Do not change existing cache keys or include batch settings in their fingerprint.
|
||||
Prepare eligibility and keys before actions, using the existing initial mail metadata.
|
||||
|
||||
Stop-after-match ends action iteration and prevents further requests; already
|
||||
computed later answers remain cached. A malformed answer fails its entire batch:
|
||||
no partial cache writes or new actions. Keep cache-generation guards for imports
|
||||
and clearing during inference. No automatic splitting or retries. Tests in
|
||||
`tests/batching.test.mjs` cover batching, cache reuse, failure atomicity, and actual
|
||||
rule-runner stop behavior with the real classifier.
|
||||
There are currently no automated tests for this project. If you add tests in the future, specify the commands to run them here. For now, verification must happen manually in Thunderbird.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
|
@ -97,51 +36,10 @@ Additional documentation exists outside this repository.
|
|||
- Development guide: [Webextention-API for Thunderbird](https://webextension-api.thunderbird.net/en/stable/)
|
||||
- [Messages API](https://webextension-api.thunderbird.net/en/stable/messages.html)
|
||||
- [Message Tags API](https://webextension-api.thunderbird.net/en/stable/messages.tags.html)
|
||||
- [messageDisplayAction API](https://webextension-api.thunderbird.net/en/stable/messageDisplayAction.html)
|
||||
- [Storage API](https://webextension-api.thunderbird.net/en/stable/storage.html)
|
||||
- Thunderbird Add-on Store Policies
|
||||
- [Third Party Library Usage](https://extensionworkshop.com/documentation/publish/third-party-library-usage/)
|
||||
- Third Party Libraries
|
||||
- [Bulma.css v1.0.3](https://github.com/jgthms/bulma/blob/1.0.3/css/bulma.css)
|
||||
- [Bulma.css](https://github.com/jgthms/bulma)
|
||||
- Issue tracker: [Thunderbird tracker on Bugzilla](https://bugzilla.mozilla.org/describecomponents.cgi?product=Thunderbird)
|
||||
|
||||
|
||||
### Message Structure Notes
|
||||
|
||||
Messages retrieved with `messenger.messages.getFull` are returned as
|
||||
nested objects. The root contains `headers` and a `parts` array. Each part may
|
||||
itself contain `parts` for multipart messages or a `body` string. Attachments are
|
||||
indicated via the `content-disposition` header.
|
||||
|
||||
When constructing the text sent to the AI service, parse the full message
|
||||
recursively. Include key headers such as `from`, `to`, `subject`, and others, and
|
||||
record attachment summaries rather than raw binary data. Inline or attached
|
||||
base64 data should be replaced with placeholders showing the byte size. The
|
||||
final string should have the headers, a brief attachment section, then the plain
|
||||
text extracted from all text parts.
|
||||
|
||||
### Cache Strategy
|
||||
|
||||
`aiCache` persists version 2 inference evidence. Keys are
|
||||
`v2:<SHA-256 Message-ID>:<SHA-256 stable configuration/criterion fingerprint>`.
|
||||
Include backend, endpoint, model, effective request parameters and preprocessing;
|
||||
exclude secrets, actions and thresholds. Recompute SystemOne matches from cached
|
||||
probability and current thresholds without automatically executing actions.
|
||||
Entries retain a hashed message identifier for clearing across configurations.
|
||||
Version 4.0.0 intentionally breaks legacy cache compatibility: legacy caches
|
||||
(including `aiReasonCache`) are invalidated because their inference
|
||||
configuration is unknown. Import/export validates versioned evidence; storage
|
||||
changes refresh the running cache. Maintenance uses `sortana:getCacheSize` to
|
||||
count validated entries in the background classifier, rather than raw storage keys.
|
||||
Full restoration requires matching settings and rules as well as cache evidence;
|
||||
`tests/cacheRestore.test.mjs` exercises export/reinstall/import through the actual
|
||||
storage and Details handlers, including changed numeric Thunderbird message IDs. Clearing/importing during inference prevents
|
||||
that in-flight result from repopulating the replaced cache.
|
||||
|
||||
### Icon Set Usage
|
||||
|
||||
Toolbar and menu icons reside under `resources/img` and are provided in 16, 32
|
||||
and 64 pixel variants. When changing these icons, pass a dictionary mapping the
|
||||
sizes to the paths in `browserAction.setIcon` or `messageDisplayAction.setIcon`.
|
||||
Use `resources/svg2img.ps1` on Windows or `resources/svg2img.sh` on Linux to
|
||||
regenerate PNGs from the SVG sources.
|
||||
|
|
|
|||
216
README.md
|
|
@ -4,145 +4,36 @@
|
|||
|
||||
Sortana is an experimental Thunderbird add-on that integrates an AI-powered filter rule.
|
||||
It allows you to classify email messages by sending their contents to a configurable
|
||||
HTTP endpoint. Choose **Chat Completions** for an OpenAI-compatible
|
||||
`/v1/chat/completions` service or **SystemOne** for decision models served through
|
||||
`/v1/systemone`. Chat Completions returns a match boolean and textual explanation;
|
||||
SystemOne returns a match probability, compared against a configurable threshold.
|
||||
The options page stores a base URL and shows the resolved request URL.
|
||||
HTTP endpoint. The endpoint should respond with JSON indicating whether the
|
||||
message meets a specified criterion.
|
||||
|
||||
## Features
|
||||
|
||||
- **Configurable endpoint** – set the classification service base URL on the options page.
|
||||
- **Model selection** – load available models from the endpoint and choose one (or omit the model field).
|
||||
- **Optional OpenAI auth headers** – provide an API key plus optional organization/project headers when needed.
|
||||
- **Custom system prompts** – tailor the instructions sent as the system message for more precise results.
|
||||
- **Persistent result caching** – classification results and reasoning are saved to disk so messages aren't re-evaluated across restarts.
|
||||
- **Advanced parameters (Chat Completions)** – tune generation settings like temperature, top‑p and more from the options page.
|
||||
Provider-specific advanced fields are forwarded as configured and may be rejected by strict OpenAI endpoints.
|
||||
- **Markdown conversion** – optionally convert HTML bodies to Markdown before sending them to the AI service.
|
||||
- **AI classification rule** – adds the "AI classification" term with
|
||||
`matches` and `doesn't match` operators.
|
||||
- **Configurable endpoint** – set the classification service URL on the options page.
|
||||
- **Prompt templates** – choose between several model formats or provide your own custom template.
|
||||
- **Custom system prompts** – tailor the instructions sent to the model for more precise results.
|
||||
- **Filter editor integration** – patches Thunderbird's filter editor to accept
|
||||
text criteria for AI classification.
|
||||
- **Persistent result caching** – classification results are saved to disk so messages aren't re-evaluated across restarts.
|
||||
- **Advanced parameters** – tune generation settings like temperature, top‑p and more from the options page.
|
||||
- **Debug logging** – optional colorized logs help troubleshoot interactions with the AI service.
|
||||
- **Debug tab** – view the last request payload and a diff between the unaltered message text and the final prompt.
|
||||
- **Light/Dark themes** – automatically match Thunderbird's appearance with optional manual override.
|
||||
- **Automatic rules** – create rules that tag, move, copy, forward, reply, delete, archive, mark read/unread or flag/unflag messages based on AI classification. Rules can optionally apply only to unread messages and can ignore messages outside a chosen age range.
|
||||
- **Automatic rules** – create rules that tag or move new messages based on AI classification.
|
||||
- **Rule ordering** – drag rules to prioritize them and optionally stop processing after a match.
|
||||
- **Rule enable/disable** – temporarily turn a rule off without removing it.
|
||||
- **Account & folder filters** – limit rules to specific accounts or folders.
|
||||
- **Context menu** – apply AI rules from the message list or the message display action button.
|
||||
- **Status icons** – toolbar icons show when classification is in progress and briefly display success states. If a failure occurs the icon turns red briefly before returning to normal.
|
||||
- **Error notification** – failed classification displays a notification in Thunderbird.
|
||||
- **Session error log** – the Errors tab (visible only when errors occur) shows errors recorded since the last add-on start.
|
||||
- **View results** – inspect chat explanations or SystemOne probabilities and thresholds in the Details popup.
|
||||
- **Cache management** – clear cached results from the context menu or options page.
|
||||
- **Queue & timing stats** – monitor processing time on the Maintenance tab.
|
||||
- **Packaging scripts** – `build-xpi.ps1` (PowerShell) or `build-xpi.sh` (bash) build an XPI ready for installation.
|
||||
- **Icon generation scripts** – `resources/svg2img.ps1` (PowerShell) or `resources/svg2img.sh` (bash) regenerate themed PNG icons from the SVG sources.
|
||||
- **Maintenance tab** – view rule counts, cache entries and clear cached results from the options page.
|
||||
|
||||
### Cache Storage
|
||||
|
||||
Classification results are stored under `aiCache` using version 2 entries. Keys
|
||||
combine a hashed Message-ID with a fingerprint of the criterion, backend,
|
||||
endpoint, selected model, effective request settings, and preprocessing settings.
|
||||
Credentials and thresholds are excluded. SystemOne probabilities are retained at
|
||||
full precision; changing a threshold reuses the evidence without another request.
|
||||
Changing inference settings requires a fresh classification. If the model behind
|
||||
an unchanged server/model alias changes, clear the cache manually.
|
||||
|
||||
**Version 4.0.0 upgrade note:** legacy cache entries and `aiReasonCache` are discarded because
|
||||
they do not identify the inference configuration. Settings and rules are retained.
|
||||
Imported legacy caches are also discarded. To restore cached Details after reinstalling,
|
||||
export and import **Settings, Rules, and Cache** together: lookups require the
|
||||
original criterion and matching inference settings. Import refreshes the running
|
||||
classifier immediately; Maintenance counts its validated, loaded entries. A cache-only
|
||||
import will not recreate rules or settings. Clearing a message's cache removes its
|
||||
entries across rules and configurations.
|
||||
|
||||
### SystemOne with llama-server
|
||||
|
||||
Use a llama.cpp build supporting `/v1/systemone` and a compatible decision GGUF;
|
||||
older builds and ordinary chat models cannot serve this API. A 404 with
|
||||
`File Not Found` from `/v1/systemone` indicates the route is unavailable: check
|
||||
the running server build and any reverse-proxy routing. Loading a decision GGUF
|
||||
alone does not add this route to an older server. For example:
|
||||
|
||||
```sh
|
||||
llama-server -hf ggml-org/Laya-GGUF
|
||||
```
|
||||
|
||||
1. Select **SystemOne** in Settings and set the endpoint to `http://127.0.0.1:8080`
|
||||
(or your server's address). Refresh models and select one, or choose **None**
|
||||
for a single loaded model. Router deployments require the appropriate model ID.
|
||||
2. Set the default match probability threshold (initially **50%**). Each rule has
|
||||
an optional percentage override; leave it blank to inherit the default.
|
||||
3. Apply rules normally. Each uncached criterion becomes a `noul` question. A rule
|
||||
matches when the unrounded probability is **greater than or equal to** its
|
||||
effective threshold. For example, 80% matches an 80% threshold.
|
||||
4. Open Details to see a probability bar with the percentage inside. Hover over
|
||||
the bar to see the current threshold and model. This is the
|
||||
result under current settings, not an action history. Changing thresholds does
|
||||
not automatically rerun actions. Decision models provide no textual explanation;
|
||||
no second model is called to invent one. Use Chat Completions for explanations.
|
||||
|
||||
**SystemOne batching:** **Process all** is enabled by default and sends all eligible,
|
||||
uncached criteria for an email in one request, with the email state included once.
|
||||
Turn it off to set **Rules per request** (initially 8); use 1 for sequential
|
||||
requests. The number is retained while disabled. These controls apply only to
|
||||
SystemOne; Chat Completions continues processing one rule per request.
|
||||
|
||||
Cached rules consume no batch capacity, and identical criteria share a question
|
||||
while retaining their individual thresholds. Actions execute in rule order and
|
||||
honor stop-after-match. A batch can evaluate later rules whose actions are then
|
||||
skipped; their probabilities remain available in Details. No further batches are
|
||||
requested after stopping. Changing batch settings preserves existing cached results.
|
||||
|
||||
If any requested answer is missing or invalid, the entire batch fails without
|
||||
caching its new answers or executing their actions. Previously completed actions
|
||||
remain applied. Sortana does not automatically retry or split rejected batches;
|
||||
if the server rejects a large request, disable Process all and reduce Rules per
|
||||
request. Limits and performance depend on the server and model.
|
||||
|
||||
Probabilities are model estimates, not guaranteed accuracy. Evaluate thresholds
|
||||
against representative labeled emails, including long messages and attachments.
|
||||
SystemOne uses the existing text extraction and attachment summaries; it does not
|
||||
send attachment binaries or images. Chat prompts and generation settings remain
|
||||
saved but are inactive for SystemOne. Optional Bearer authentication works with
|
||||
both APIs; organization/project headers apply only to Chat Completions.
|
||||
|
||||
**Input character limit** applies when aggressive reduction is enabled, independently
|
||||
of maximum output tokens. Existing installations inherit their previous effective
|
||||
character limit; the default is 3686. Character counts do not guarantee that input
|
||||
fits a model's token context or llama-server batch limits. Leave room for question
|
||||
instructions and configure the server for the chosen model.
|
||||
|
||||
A request or response-parsing failure stops the remaining rules for that email,
|
||||
records an error, and allows the queue to continue with the next email. Failed
|
||||
results are never cached. Actions already completed are not rolled back.
|
||||
|
||||
Upstream: [llama-server SystemOne documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#typesafe-compatible-api-endpoints).
|
||||
|
||||
### Verification
|
||||
|
||||
Run the dependency-free tests with Node.js 22 or newer:
|
||||
|
||||
```sh
|
||||
node --test tests/*.test.mjs
|
||||
```
|
||||
|
||||
Tests cover adapters, thresholds, URLs, settings validation, cache migration and
|
||||
reuse, request failures, per-email snapshots, and queue behavior with mocked
|
||||
Thunderbird services. Manual acceptance testing in Thunderbird should verify both
|
||||
API types, model discovery, rule overrides and ordering, import/export, cache
|
||||
clearing, Details, and stop-after-match against a running server.
|
||||
- **Status icons** – toolbar icons indicate when messages are queued or being classified.
|
||||
- **Packaging script** – `build-xpi.ps1` builds an XPI ready for installation.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Sortana is implemented entirely with documented MailExtension/WebExtension APIs. No custom experiment code or Thunderbird-internal fallback paths are required:
|
||||
Sortana is implemented entirely with standard WebExtension scripts—no custom experiment code is required:
|
||||
|
||||
- `background.js` loads saved settings, manages the classification queue and listens for new messages.
|
||||
- `modules/AiClassifier.js` shares transport, threshold policy, and cache handling;
|
||||
`modules/backends/` contains the Chat Completions and SystemOne adapters.
|
||||
- `options/` contains the HTML and JavaScript for configuring the endpoint and rules.
|
||||
- `details.html` / `details.js` present cached classification evidence for a message.
|
||||
- `background.js` loads saved settings and listens for new messages.
|
||||
- `modules/ExpressionSearchFilter.jsm` implements the AI filter and performs the
|
||||
HTTP request.
|
||||
- `options/` contains the HTML and JavaScript for configuring the endpoint and
|
||||
rules.
|
||||
- `_locales/` holds localized strings used throughout the UI.
|
||||
|
||||
### Key Files
|
||||
|
|
@ -150,45 +41,30 @@ Sortana is implemented entirely with documented MailExtension/WebExtension APIs.
|
|||
| Path | Purpose |
|
||||
| --------------------------------------- | ---------------------------------------------- |
|
||||
| `manifest.json` | Extension manifest and entry points. |
|
||||
| `background.js` | Startup tasks and classification queue management. |
|
||||
| `modules/AiClassifier.js` | Core classification logic and cache handling. |
|
||||
| `background.js` | Startup tasks and message handling. |
|
||||
| `modules/ExpressionSearchFilter.jsm` | Custom filter term and AI request logic. |
|
||||
| `options/options.html` and `options.js` | Endpoint and rule configuration UI. |
|
||||
| `details.html` and `details.js` | View stored classification evidence for a message. |
|
||||
| `logger.js` | Colorized logging with optional debug mode. |
|
||||
| `logger.js` and `modules/logger.jsm` | Colorized logging with optional debug mode. |
|
||||
|
||||
## Building
|
||||
|
||||
1. Ensure PowerShell is available on Windows or bash, `python3`, and Inkscape are
|
||||
available on Linux.
|
||||
2. The Bulma stylesheet (v1.0.3) is already included as `options/bulma.css`.
|
||||
3. Run `powershell ./build-xpi.ps1` or `./build-xpi.sh` from the repository root.
|
||||
The script reads the version from `manifest.json` and creates an XPI in the
|
||||
`release` folder.
|
||||
1. Ensure PowerShell is available (for Windows) or adapt the script for other
|
||||
environments.
|
||||
2. Ensure the Bulma stylesheet (v1.0.4) is saved as `options/bulma.css`. You can
|
||||
download it from <https://github.com/jgthms/bulma/blob/1.0.4/css/bulma.css>.
|
||||
3. Run `powershell ./build-xpi.ps1` from the repository root. The script reads
|
||||
the version from `manifest.json` and creates an XPI in the `release` folder.
|
||||
4. Install the generated XPI in Thunderbird via the Add-ons Manager. During
|
||||
development you can also load the directory as a temporary add-on.
|
||||
5. To regenerate PNG icons from the SVG sources, run `resources/svg2img.ps1` on
|
||||
Windows or `./resources/svg2img.sh` on Linux.
|
||||
|
||||
## Usage
|
||||
|
||||
1. Open the add-on's options and set the base URL of your classification service
|
||||
(Sortana appends the path for the selected API type). Use the Model dropdown to load
|
||||
`/v1/models` and select a model or choose **None** to omit the `model` field.
|
||||
Advanced settings include optional API key, organization, and project headers
|
||||
for OpenAI-hosted endpoints.
|
||||
1. Open the add-on's options and set the URL of your classification service.
|
||||
2. Use the **Classification Rules** section to add a criterion and optional
|
||||
actions such as tagging, moving, copying, forwarding, replying,
|
||||
deleting or archiving a message when it matches. Drag rules to
|
||||
reorder them, check *Only apply to unread messages* to skip read mail,
|
||||
set optional minimum or maximum message age limits, select the accounts or
|
||||
folders a rule should apply to. Use the
|
||||
slashed-circle/check button to disable or re-enable a rule. The small
|
||||
circle buttons for optional conditions show a filled dot when active, and
|
||||
check *Stop after match* to halt further processing. Forward and reply actions
|
||||
open a compose window using the account that received the message.
|
||||
actions such as tagging or moving a message when it matches. Drag rules to
|
||||
reorder them and check *Stop after match* to halt further processing.
|
||||
3. Save your settings. New mail will be evaluated automatically using the
|
||||
configured rules.
|
||||
4. If the toolbar icon shows a red X, it will clear after a few seconds. Open the Errors tab in Options to review the latest failures.
|
||||
|
||||
### Example Filters
|
||||
|
||||
|
|
@ -216,7 +92,7 @@ Here are some useful and fun example criteria you can use in your filters. Filte
|
|||
For when you're ready to filter based on vibes.
|
||||
|
||||
You can define as many filters as you'd like, each using a different prompt and
|
||||
triggering tags, moves, copies, deletes, archives, read/unread changes or flag updates based on the model's classification.
|
||||
triggering tags, moves, or actions based on the model's classification.
|
||||
|
||||
## Required Permissions
|
||||
|
||||
|
|
@ -224,33 +100,16 @@ Sortana requests the following Thunderbird permissions:
|
|||
|
||||
- `storage` – store configuration and cached classification results.
|
||||
- `messagesRead` – read message contents for classification.
|
||||
- `messagesMove` – move or copy messages when a rule specifies a target folder.
|
||||
- `messagesUpdate` – change message properties such as tags, junk status, read/unread state and flags.
|
||||
- `messagesMove` – move messages when a rule specifies a target folder.
|
||||
- `messagesUpdate` – change message properties such as tags and junk status.
|
||||
- `messagesTagsList` – retrieve existing message tags for rule actions.
|
||||
- `accountsRead` – list accounts and folders for move or copy actions.
|
||||
- `accountsRead` – list accounts and folders for move actions.
|
||||
- `menus` – add context menu commands.
|
||||
- `tabs` – open new tabs and query the active tab.
|
||||
- `notifications` – display error notifications.
|
||||
- `compose` – create reply and forward compose windows for matching rules.
|
||||
|
||||
## Thunderbird Add-on Store Disclosures
|
||||
|
||||
The [Third Party Library Usage](https://extensionworkshop.com/documentation/publish/third-party-library-usage/) policy
|
||||
requires disclosure of third party libraries that are included in the add-on. Even though
|
||||
the disclosure is only required for add-on review, they'll be listed here as well. Sortana
|
||||
uses the following third party libraries:
|
||||
|
||||
- [Bulma.css v1.0.3](https://github.com/jgthms/bulma/blob/1.0.3/css/bulma.css)
|
||||
- MIT License
|
||||
- [turndown v7.2.0](https://github.com/mixmark-io/turndown/tree/v7.2.0)
|
||||
- MIT License
|
||||
- [diff](https://github.com/google/diff-match-patch/blob/62f2e689f498f9c92dbc588c58750addec9b1654/javascript/diff_match_patch_uncompressed.js)
|
||||
- Apache-2.0 license
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the terms of the GNU General Public License
|
||||
version 3. See `LICENSE` for the full text. Third party libraries are licensed seperately.
|
||||
version 3. See `LICENSE` for the full text.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
|
|
@ -260,4 +119,3 @@ Sortana builds upon knowledge gained from open-source projects. In particular,
|
|||
how Thunderbird's WebExtension and experiment APIs can be extended. Their code
|
||||
provided invaluable guidance during development.
|
||||
|
||||
- Icons from [cc0-icons.jonh.eu](https://cc0-icons.jonh.eu/) are used under the CC0 license.
|
||||
|
|
|
|||
|
|
@ -4,27 +4,15 @@
|
|||
"doesntMatch": { "message": "doesn't match" },
|
||||
"options.title": { "message": "AI Filter Options" },
|
||||
"options.endpoint": { "message": "Endpoint" },
|
||||
"options.template": { "message": "Prompt template" },
|
||||
"options.customTemplate": { "message": "Custom template" },
|
||||
"options.systemInstructions": { "message": "System instructions" },
|
||||
"options.reset": { "message": "Reset to default" },
|
||||
"options.placeholders": { "message": "Placeholders: {{system}}, {{email}}, {{query}}" },
|
||||
"template.openai": { "message": "OpenAI / ChatML" },
|
||||
"template.qwen": { "message": "Qwen" },
|
||||
"template.mistral": { "message": "Mistral" },
|
||||
"template.custom": { "message": "Custom" },
|
||||
"options.save": { "message": "Save" },
|
||||
"options.debugLogging": { "message": "Enable debug logging" },
|
||||
"options.htmlToMarkdown": { "message": "Convert HTML body to Markdown" },
|
||||
"options.stripUrlParams": { "message": "Remove URL tracking parameters" },
|
||||
"options.altTextImages": { "message": "Replace images with alt text" },
|
||||
"options.collapseWhitespace": { "message": "Collapse long whitespace" },
|
||||
"options.tokenReduction": { "message": "Aggressive token reduction" }
|
||||
,"action.read": { "message": "read" }
|
||||
,"action.flag": { "message": "flag" }
|
||||
,"action.copy": { "message": "copy" }
|
||||
,"action.delete": { "message": "delete" }
|
||||
,"action.archive": { "message": "archive" }
|
||||
,"action.forward": { "message": "forward" }
|
||||
,"action.reply": { "message": "reply" }
|
||||
,"param.markRead": { "message": "mark read" }
|
||||
,"param.markUnread": { "message": "mark unread" }
|
||||
,"param.flag": { "message": "flag" }
|
||||
,"param.unflag": { "message": "unflag" }
|
||||
,"param.address": { "message": "address" }
|
||||
,"param.replyAll": { "message": "reply all" }
|
||||
,"param.replySender": { "message": "reply sender" }
|
||||
"options.debugLogging": { "message": "Enable debug logging" }
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,8 +9,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
|
|||
AGENTS.md = AGENTS.md
|
||||
background.js = background.js
|
||||
build-xpi.ps1 = build-xpi.ps1
|
||||
details.html = details.html
|
||||
details.js = details.js
|
||||
LICENSE = LICENSE
|
||||
logger.js = logger.js
|
||||
manifest.json = manifest.json
|
||||
|
|
@ -29,7 +27,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "options", "options", "{7372
|
|||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "modules", "modules", "{75ED3C1E-D3C7-4546-9F2E-AC85859DDF4B}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
modules\AiClassifier.js = modules\AiClassifier.js
|
||||
modules\ExpressionSearchFilter.jsm = modules\ExpressionSearchFilter.jsm
|
||||
modules\logger.jsm = modules\logger.jsm
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "_locales", "_locales", "{D446E5C6-BDDE-4091-BD1A-EC57170003CF}"
|
||||
|
|
@ -39,42 +38,23 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "en-US", "en-US", "{8BEA7793
|
|||
_locales\en-US\messages.json = _locales\en-US\messages.json
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "resources", "resources", "{68A87938-5C2B-49F5-8AAA-8A34FBBFD854}"
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "content", "content", "{028FDA4B-AC3E-4A0E-9291-978E213F9B78}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
resources\svg2img.ps1 = resources\svg2img.ps1
|
||||
content\filterEditor.js = content\filterEditor.js
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "prompt_templates", "prompt_templates", "{86516D53-50D4-4FE2-9D8A-977A8F5EBDBD}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
prompt_templates\mistral.txt = prompt_templates\mistral.txt
|
||||
prompt_templates\openai.txt = prompt_templates\openai.txt
|
||||
prompt_templates\qwen.txt = prompt_templates\qwen.txt
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "resources", "resources", "{68A87938-5C2B-49F5-8AAA-8A34FBBFD854}"
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "img", "img", "{F266602F-1755-4A95-A11B-6C90C701C5BF}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
resources\img\average-16.png = resources\img\average-16.png
|
||||
resources\img\average-32.png = resources\img\average-32.png
|
||||
resources\img\average-64.png = resources\img\average-64.png
|
||||
resources\img\check-16.png = resources\img\check-16.png
|
||||
resources\img\check-32.png = resources\img\check-32.png
|
||||
resources\img\check-64.png = resources\img\check-64.png
|
||||
resources\img\circle-16.png = resources\img\circle-16.png
|
||||
resources\img\circle-32.png = resources\img\circle-32.png
|
||||
resources\img\circle-64.png = resources\img\circle-64.png
|
||||
resources\img\circledots-16.png = resources\img\circledots-16.png
|
||||
resources\img\circledots-32.png = resources\img\circledots-32.png
|
||||
resources\img\circledots-64.png = resources\img\circledots-64.png
|
||||
resources\img\clipboarddata-16.png = resources\img\clipboarddata-16.png
|
||||
resources\img\clipboarddata-32.png = resources\img\clipboarddata-32.png
|
||||
resources\img\clipboarddata-64.png = resources\img\clipboarddata-64.png
|
||||
resources\img\download-16.png = resources\img\download-16.png
|
||||
resources\img\download-32.png = resources\img\download-32.png
|
||||
resources\img\download-64.png = resources\img\download-64.png
|
||||
resources\img\eye-16.png = resources\img\eye-16.png
|
||||
resources\img\eye-32.png = resources\img\eye-32.png
|
||||
resources\img\eye-64.png = resources\img\eye-64.png
|
||||
resources\img\flag-16.png = resources\img\flag-16.png
|
||||
resources\img\flag-32.png = resources\img\flag-32.png
|
||||
resources\img\flag-64.png = resources\img\flag-64.png
|
||||
resources\img\full-logo-white.png = resources\img\full-logo-white.png
|
||||
resources\img\full-logo.png = resources\img\full-logo.png
|
||||
resources\img\gear-16.png = resources\img\gear-16.png
|
||||
resources\img\gear-32.png = resources\img\gear-32.png
|
||||
resources\img\gear-64.png = resources\img\gear-64.png
|
||||
resources\img\logo.png = resources\img\logo.png
|
||||
resources\img\logo128.png = resources\img\logo128.png
|
||||
resources\img\logo16.png = resources\img\logo16.png
|
||||
|
|
@ -82,45 +62,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "img", "img", "{F266602F-175
|
|||
resources\img\logo48.png = resources\img\logo48.png
|
||||
resources\img\logo64.png = resources\img\logo64.png
|
||||
resources\img\logo96.png = resources\img\logo96.png
|
||||
resources\img\reply-16.png = resources\img\reply-16.png
|
||||
resources\img\reply-32.png = resources\img\reply-32.png
|
||||
resources\img\reply-64.png = resources\img\reply-64.png
|
||||
resources\img\settings-16.png = resources\img\settings-16.png
|
||||
resources\img\settings-32.png = resources\img\settings-32.png
|
||||
resources\img\settings-64.png = resources\img\settings-64.png
|
||||
resources\img\trash-16.png = resources\img\trash-16.png
|
||||
resources\img\trash-32.png = resources\img\trash-32.png
|
||||
resources\img\trash-64.png = resources\img\trash-64.png
|
||||
resources\img\upload-16.png = resources\img\upload-16.png
|
||||
resources\img\upload-32.png = resources\img\upload-32.png
|
||||
resources\img\upload-64.png = resources\img\upload-64.png
|
||||
resources\img\x-16.png = resources\img\x-16.png
|
||||
resources\img\x-32.png = resources\img\x-32.png
|
||||
resources\img\x-64.png = resources\img\x-64.png
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "js", "js", "{21D2A42C-3F85-465C-9141-C106AFD92B68}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
resources\js\diff_match_patch_uncompressed.js = resources\js\diff_match_patch_uncompressed.js
|
||||
resources\js\turndown.js = resources\js\turndown.js
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "svg", "svg", "{D4E9C905-4884-488E-B763-5BD39049C1B1}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
resources\svg\average.svg = resources\svg\average.svg
|
||||
resources\svg\check.svg = resources\svg\check.svg
|
||||
resources\svg\circle.svg = resources\svg\circle.svg
|
||||
resources\svg\circledots.svg = resources\svg\circledots.svg
|
||||
resources\svg\clipboarddata.svg = resources\svg\clipboarddata.svg
|
||||
resources\svg\download.svg = resources\svg\download.svg
|
||||
resources\svg\eye.svg = resources\svg\eye.svg
|
||||
resources\svg\flag.svg = resources\svg\flag.svg
|
||||
resources\svg\gear.svg = resources\svg\gear.svg
|
||||
resources\svg\reply.svg = resources\svg\reply.svg
|
||||
resources\svg\settings.svg = resources\svg\settings.svg
|
||||
resources\svg\trash.svg = resources\svg\trash.svg
|
||||
resources\svg\upload.svg = resources\svg\upload.svg
|
||||
resources\svg\x.svg = resources\svg\x.svg
|
||||
EndProjectSection
|
||||
EndProject
|
||||
Global
|
||||
|
|
@ -132,10 +73,9 @@ Global
|
|||
{75ED3C1E-D3C7-4546-9F2E-AC85859DDF4B} = {BCC6E6D2-343B-4C48-854D-5FE3BBC3CB70}
|
||||
{D446E5C6-BDDE-4091-BD1A-EC57170003CF} = {BCC6E6D2-343B-4C48-854D-5FE3BBC3CB70}
|
||||
{8BEA7793-3336-40ED-AB96-7FFB09FEB0F6} = {D446E5C6-BDDE-4091-BD1A-EC57170003CF}
|
||||
{028FDA4B-AC3E-4A0E-9291-978E213F9B78} = {BCC6E6D2-343B-4C48-854D-5FE3BBC3CB70}
|
||||
{86516D53-50D4-4FE2-9D8A-977A8F5EBDBD} = {BCC6E6D2-343B-4C48-854D-5FE3BBC3CB70}
|
||||
{68A87938-5C2B-49F5-8AAA-8A34FBBFD854} = {BCC6E6D2-343B-4C48-854D-5FE3BBC3CB70}
|
||||
{F266602F-1755-4A95-A11B-6C90C701C5BF} = {68A87938-5C2B-49F5-8AAA-8A34FBBFD854}
|
||||
{21D2A42C-3F85-465C-9141-C106AFD92B68} = {68A87938-5C2B-49F5-8AAA-8A34FBBFD854}
|
||||
{D4E9C905-4884-488E-B763-5BD39049C1B1} = {68A87938-5C2B-49F5-8AAA-8A34FBBFD854}
|
||||
EndGlobalSection
|
||||
EndGlobal
|
||||
|
|
|
|||
728
background.js
|
|
@ -18,70 +18,14 @@ let aiRules = [];
|
|||
let queue = Promise.resolve();
|
||||
let queuedCount = 0;
|
||||
let processing = false;
|
||||
let iconTimer = null;
|
||||
let errorTimer = null;
|
||||
let timingStats = { count: 0, mean: 0, m2: 0, total: 0, last: -1 };
|
||||
let currentStart = 0;
|
||||
let logGetTiming = true;
|
||||
let htmlToMarkdown = false;
|
||||
let stripUrlParams = false;
|
||||
let altTextImages = false;
|
||||
let collapseWhitespace = false;
|
||||
let tokenReduction = false;
|
||||
let TurndownService = null;
|
||||
let userTheme = 'auto';
|
||||
let currentTheme = 'light';
|
||||
let detectSystemTheme;
|
||||
let errorPending = false;
|
||||
let errorLog = [];
|
||||
let showDebugTab = false;
|
||||
const ERROR_NOTIFICATION_ID = 'sortana-error';
|
||||
const ERROR_ICON_TIMEOUT = 4500;
|
||||
const MAX_ERROR_LOG = 50;
|
||||
|
||||
function normalizeRules(rules) {
|
||||
return Array.isArray(rules) ? rules.map(r => {
|
||||
if (r.actions) {
|
||||
if (!Array.isArray(r.accounts)) r.accounts = [];
|
||||
if (!Array.isArray(r.folders)) r.folders = [];
|
||||
r.enabled = r.enabled !== false;
|
||||
return r;
|
||||
function updateActionIcon() {
|
||||
let path = "resources/img/logo32.png";
|
||||
if (processing) {
|
||||
path = "resources/img/status-classifying.png";
|
||||
} else if (queuedCount > 0) {
|
||||
path = "resources/img/status-queued.png";
|
||||
}
|
||||
const actions = [];
|
||||
if (r.tag) actions.push({ type: 'tag', tagKey: r.tag });
|
||||
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
|
||||
if (r.copyTarget || r.copyTo) actions.push({ type: 'copy', copyTarget: r.copyTarget || r.copyTo });
|
||||
const rule = { criterion: r.criterion, actions };
|
||||
if (r.decisionThreshold !== undefined) rule.decisionThreshold = r.decisionThreshold;
|
||||
if (r.stopProcessing) rule.stopProcessing = true;
|
||||
if (r.unreadOnly) rule.unreadOnly = true;
|
||||
if (typeof r.minAgeDays === 'number') rule.minAgeDays = r.minAgeDays;
|
||||
if (typeof r.maxAgeDays === 'number') rule.maxAgeDays = r.maxAgeDays;
|
||||
if (Array.isArray(r.accounts)) rule.accounts = r.accounts;
|
||||
if (Array.isArray(r.folders)) rule.folders = r.folders;
|
||||
rule.enabled = r.enabled !== false;
|
||||
return rule;
|
||||
}) : [];
|
||||
}
|
||||
|
||||
function iconPaths(name) {
|
||||
return {
|
||||
16: `resources/img/${name}-${currentTheme}-16.png`,
|
||||
32: `resources/img/${name}-${currentTheme}-32.png`,
|
||||
64: `resources/img/${name}-${currentTheme}-64.png`
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
const ICONS = {
|
||||
logo: () => 'resources/img/logo.png',
|
||||
circledots: () => iconPaths('circledots'),
|
||||
circle: () => iconPaths('circle'),
|
||||
average: () => iconPaths('average'),
|
||||
error: () => iconPaths('x')
|
||||
};
|
||||
|
||||
function setIcon(path) {
|
||||
if (browser.browserAction) {
|
||||
browser.browserAction.setIcon({ path });
|
||||
}
|
||||
|
|
@ -90,320 +34,50 @@ function setIcon(path) {
|
|||
}
|
||||
}
|
||||
|
||||
function updateActionIcon() {
|
||||
let path = ICONS.logo();
|
||||
if (errorPending) {
|
||||
path = ICONS.error();
|
||||
} else if (processing || queuedCount > 0) {
|
||||
path = ICONS.circledots();
|
||||
}
|
||||
setIcon(path);
|
||||
async function sha256Hex(str) {
|
||||
const buf = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(str));
|
||||
return Array.from(new Uint8Array(buf), b => b.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
|
||||
function showTransientIcon(factory, delay = 1500) {
|
||||
if (errorPending) {
|
||||
return;
|
||||
}
|
||||
clearTimeout(iconTimer);
|
||||
const path = typeof factory === 'function' ? factory() : factory;
|
||||
setIcon(path);
|
||||
iconTimer = setTimeout(updateActionIcon, delay);
|
||||
async function applyAiRules(idsInput) {
|
||||
const ids = Array.isArray(idsInput) ? idsInput : [idsInput];
|
||||
if (!ids.length) return queue;
|
||||
|
||||
if (!aiRules.length) {
|
||||
const { aiRules: stored } = await storage.local.get("aiRules");
|
||||
aiRules = Array.isArray(stored) ? stored.map(r => {
|
||||
if (r.actions) return r;
|
||||
const actions = [];
|
||||
if (r.tag) actions.push({ type: 'tag', tagKey: r.tag });
|
||||
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
|
||||
const rule = { criterion: r.criterion, actions };
|
||||
if (r.stopProcessing) rule.stopProcessing = true;
|
||||
return rule;
|
||||
}) : [];
|
||||
}
|
||||
|
||||
async function clearError() {
|
||||
errorPending = false;
|
||||
clearTimeout(errorTimer);
|
||||
await browser.notifications.clear(ERROR_NOTIFICATION_ID);
|
||||
for (const msg of ids) {
|
||||
const id = msg?.id ?? msg;
|
||||
queuedCount++;
|
||||
updateActionIcon();
|
||||
}
|
||||
|
||||
function recordError(context, err) {
|
||||
let message = 'Unknown error';
|
||||
let detail = '';
|
||||
if (err instanceof Error) {
|
||||
message = err.message;
|
||||
detail = err.stack || '';
|
||||
} else if (err && typeof err === 'object') {
|
||||
message = typeof err.message === 'string' ? err.message : String(err || 'Unknown error');
|
||||
detail = typeof err.detail === 'string' ? err.detail : '';
|
||||
} else {
|
||||
message = String(err || 'Unknown error');
|
||||
}
|
||||
errorLog.unshift({
|
||||
time: Date.now(),
|
||||
context,
|
||||
message,
|
||||
detail
|
||||
});
|
||||
if (errorLog.length > MAX_ERROR_LOG) {
|
||||
errorLog.length = MAX_ERROR_LOG;
|
||||
}
|
||||
errorPending = true;
|
||||
updateActionIcon();
|
||||
clearTimeout(errorTimer);
|
||||
errorTimer = setTimeout(() => {
|
||||
errorPending = false;
|
||||
updateActionIcon();
|
||||
}, ERROR_ICON_TIMEOUT);
|
||||
browser.runtime.sendMessage({ type: 'sortana:errorLogUpdated', count: errorLog.length }).catch(() => {});
|
||||
}
|
||||
|
||||
function refreshMenuIcons() {
|
||||
browser.menus.update('apply-ai-rules-list', { icons: iconPaths('eye') });
|
||||
browser.menus.update('apply-ai-rules-display', { icons: iconPaths('eye') });
|
||||
browser.menus.update('clear-ai-cache-list', { icons: iconPaths('trash') });
|
||||
browser.menus.update('clear-ai-cache-display', { icons: iconPaths('trash') });
|
||||
browser.menus.update('view-ai-reason-list', { icons: iconPaths('clipboarddata') });
|
||||
browser.menus.update('view-ai-reason-display', { icons: iconPaths('clipboarddata') });
|
||||
}
|
||||
|
||||
|
||||
function byteSize(str) {
|
||||
return new TextEncoder().encode(str || "").length;
|
||||
}
|
||||
|
||||
function replaceInlineBase64(text) {
|
||||
return text.replace(/(?:data:[^;]+;base64,)?[A-Za-z0-9+/=\r\n]{100,}/g,
|
||||
m => tokenReduction ? '__BASE64__' : `[base64: ${byteSize(m)} bytes]`);
|
||||
}
|
||||
|
||||
function sanitizeString(text) {
|
||||
let t = String(text);
|
||||
if (tokenReduction) {
|
||||
t = t.replace(/<!--.*?-->/gs, '')
|
||||
.replace(/url\([^\)]*\)/gi, 'url(__IMG__)');
|
||||
}
|
||||
if (stripUrlParams) {
|
||||
t = t.replace(/https?:\/\/[^\s)]+/g, m => {
|
||||
const idx = m.indexOf('?');
|
||||
return idx >= 0 ? m.slice(0, idx) : m;
|
||||
});
|
||||
}
|
||||
if (collapseWhitespace) {
|
||||
t = t.replace(/[\u200B-\u200D\u2060\s]{2,}/g, ' ').replace(/\n{3,}/g, '\n\n');
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
function collectText(part, bodyParts, attachments) {
|
||||
if (part.parts && part.parts.length) {
|
||||
for (const p of part.parts) collectText(p, bodyParts, attachments);
|
||||
return;
|
||||
}
|
||||
const ct = (part.contentType || "text/plain").toLowerCase();
|
||||
const cd = (part.headers?.["content-disposition"]?.[0] || "").toLowerCase();
|
||||
const body = String(part.body || "");
|
||||
if (cd.includes("attachment") || !ct.startsWith("text/")) {
|
||||
const nameMatch = /filename\s*=\s*"?([^";]+)/i.exec(cd) || /name\s*=\s*"?([^";]+)/i.exec(part.headers?.["content-type"]?.[0] || "");
|
||||
const name = nameMatch ? nameMatch[1] : "";
|
||||
attachments.push(`${name} (${ct}, ${part.size || byteSize(body)} bytes)`);
|
||||
} else if (ct.startsWith("text/html")) {
|
||||
const doc = new DOMParser().parseFromString(body, 'text/html');
|
||||
if (tokenReduction) {
|
||||
doc.querySelectorAll('script,style').forEach(el => el.remove());
|
||||
const walker = doc.createTreeWalker(doc, NodeFilter.SHOW_COMMENT);
|
||||
let node;
|
||||
while ((node = walker.nextNode())) {
|
||||
node.parentNode.removeChild(node);
|
||||
}
|
||||
doc.querySelectorAll('*').forEach(el => {
|
||||
for (const attr of Array.from(el.attributes)) {
|
||||
if (!['href','src','alt'].includes(attr.name)) {
|
||||
el.removeAttribute(attr.name);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
doc.querySelectorAll('img').forEach(img => {
|
||||
const alt = img.getAttribute('alt') || '';
|
||||
const text = altTextImages ? alt : '__IMG__';
|
||||
img.replaceWith(doc.createTextNode(text));
|
||||
});
|
||||
if (stripUrlParams) {
|
||||
doc.querySelectorAll('[href]').forEach(a => {
|
||||
const href = a.getAttribute('href');
|
||||
if (href) a.setAttribute('href', href.split('?')[0]);
|
||||
});
|
||||
doc.querySelectorAll('[src]').forEach(e => {
|
||||
const src = e.getAttribute('src');
|
||||
if (src) e.setAttribute('src', src.split('?')[0]);
|
||||
});
|
||||
}
|
||||
if (htmlToMarkdown && TurndownService) {
|
||||
try {
|
||||
const td = new TurndownService();
|
||||
const md = sanitizeString(td.turndown(doc.body.innerHTML || body));
|
||||
bodyParts.push(replaceInlineBase64(`[HTML Body converted to Markdown]\n${md}`));
|
||||
} catch (e) {
|
||||
bodyParts.push(replaceInlineBase64(sanitizeString(doc.body.textContent || "")));
|
||||
}
|
||||
} else {
|
||||
bodyParts.push(replaceInlineBase64(sanitizeString(doc.body.textContent || "")));
|
||||
}
|
||||
} else {
|
||||
bodyParts.push(replaceInlineBase64(sanitizeString(body)));
|
||||
}
|
||||
}
|
||||
|
||||
function collectRawText(part, bodyParts, attachments) {
|
||||
if (part.parts && part.parts.length) {
|
||||
for (const p of part.parts) collectRawText(p, bodyParts, attachments);
|
||||
return;
|
||||
}
|
||||
const ct = (part.contentType || "text/plain").toLowerCase();
|
||||
const cd = (part.headers?.["content-disposition"]?.[0] || "").toLowerCase();
|
||||
const body = String(part.body || "");
|
||||
if (cd.includes("attachment") || !ct.startsWith("text/")) {
|
||||
const nameMatch = /filename\s*=\s*"?([^";]+)/i.exec(cd) || /name\s*=\s*"?([^";]+)/i.exec(part.headers?.["content-type"]?.[0] || "");
|
||||
const name = nameMatch ? nameMatch[1] : "";
|
||||
attachments.push(`${name} (${ct}, ${part.size || byteSize(body)} bytes)`);
|
||||
} else if (ct.startsWith("text/html")) {
|
||||
const doc = new DOMParser().parseFromString(body, 'text/html');
|
||||
bodyParts.push(doc.body.textContent || "");
|
||||
} else {
|
||||
bodyParts.push(body);
|
||||
}
|
||||
}
|
||||
|
||||
function buildEmailText(full, applyTransforms = true) {
|
||||
const bodyParts = [];
|
||||
const attachments = [];
|
||||
const collect = applyTransforms ? collectText : collectRawText;
|
||||
collect(full, bodyParts, attachments);
|
||||
const headers = Object.entries(full.headers || {})
|
||||
.map(([k, v]) => `${k}: ${v.join(' ')}`)
|
||||
.join('\n');
|
||||
const attachInfo = `Attachments: ${attachments.length}` +
|
||||
(attachments.length ? "\n" + attachments.map(a => ` - ${a}`).join('\n') : "");
|
||||
let combined = `${headers}\n${attachInfo}\n\n${bodyParts.join('\n')}`.trim();
|
||||
if (applyTransforms && tokenReduction) {
|
||||
const seen = new Set();
|
||||
combined = combined.split('\n').filter(l => {
|
||||
if (seen.has(l)) return false;
|
||||
seen.add(l);
|
||||
return true;
|
||||
}).join('\n');
|
||||
}
|
||||
return applyTransforms ? sanitizeString(combined) : combined;
|
||||
}
|
||||
|
||||
function updateTimingStats(elapsed) {
|
||||
const t = timingStats;
|
||||
t.count += 1;
|
||||
t.total += elapsed;
|
||||
t.last = elapsed;
|
||||
const delta = elapsed - t.mean;
|
||||
t.mean += delta / t.count;
|
||||
t.m2 += delta * (elapsed - t.mean);
|
||||
}
|
||||
|
||||
async function getAllMessageIds(list) {
|
||||
const ids = [];
|
||||
if (!list) {
|
||||
return ids;
|
||||
}
|
||||
let page = list;
|
||||
ids.push(...(page.messages || []).map(m => m.id));
|
||||
while (page.id) {
|
||||
page = await messenger.messages.continueList(page.id);
|
||||
ids.push(...(page.messages || []).map(m => m.id));
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
async function processMessage(id) {
|
||||
queue = queue.then(async () => {
|
||||
processing = true;
|
||||
currentStart = Date.now();
|
||||
queuedCount--;
|
||||
updateActionIcon();
|
||||
try {
|
||||
const full = await messenger.messages.getFull(id);
|
||||
const snapshot = AiClassifier.getConfig();
|
||||
const rules = structuredClone(aiRules);
|
||||
const originalText = buildEmailText(full, false);
|
||||
let text = buildEmailText(full);
|
||||
if (tokenReduction) {
|
||||
const limit = snapshot.inputCharacterLimit;
|
||||
if (text.length > limit) {
|
||||
text = text.slice(0, limit);
|
||||
}
|
||||
}
|
||||
if (showDebugTab) {
|
||||
await storage.local.set({ lastFullText: originalText, lastPromptText: text });
|
||||
}
|
||||
let hdr;
|
||||
let currentTags = [];
|
||||
let alreadyRead = false;
|
||||
let identityId = null;
|
||||
try {
|
||||
hdr = await messenger.messages.get(id);
|
||||
currentTags = Array.isArray(hdr.tags) ? [...hdr.tags] : [];
|
||||
alreadyRead = hdr.read === true;
|
||||
const ids = await messenger.identities.list(hdr.folder.accountId);
|
||||
identityId = ids[0]?.id || null;
|
||||
} catch (e) {
|
||||
currentTags = [];
|
||||
alreadyRead = false;
|
||||
identityId = null;
|
||||
}
|
||||
|
||||
const candidates = [];
|
||||
for (const [index, rule] of rules.entries()) {
|
||||
if (rule.enabled === false) {
|
||||
continue;
|
||||
}
|
||||
if (hdr && Array.isArray(rule.accounts) && rule.accounts.length &&
|
||||
!rule.accounts.includes(hdr.folder.accountId)) {
|
||||
continue;
|
||||
}
|
||||
if (hdr && Array.isArray(rule.folders) && rule.folders.length &&
|
||||
!rule.folders.includes(hdr.folder.path)) {
|
||||
continue;
|
||||
}
|
||||
if (rule.unreadOnly && alreadyRead) {
|
||||
continue;
|
||||
}
|
||||
if (hdr && (typeof rule.minAgeDays === 'number' || typeof rule.maxAgeDays === 'number')) {
|
||||
const msgTime = new Date(hdr.date).getTime();
|
||||
if (!isNaN(msgTime)) {
|
||||
const ageDays = (Date.now() - msgTime) / 86400000;
|
||||
if (typeof rule.minAgeDays === 'number' && ageDays < rule.minAgeDays) {
|
||||
continue;
|
||||
}
|
||||
if (typeof rule.maxAgeDays === 'number' && ageDays > rule.maxAgeDays) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
const cacheKey = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot);
|
||||
candidates.push({ id: `rule_${index}`, rule, cacheKey });
|
||||
}
|
||||
for await (const { rule, result: { matched } } of AiClassifier.classifyRules(text, candidates, snapshot)) {
|
||||
const text = full?.parts?.[0]?.body || "";
|
||||
for (const rule of aiRules) {
|
||||
const cacheKey = await sha256Hex(`${id}|${rule.criterion}`);
|
||||
const matched = await AiClassifier.classifyText(text, rule.criterion, cacheKey);
|
||||
if (matched) {
|
||||
for (const act of (rule.actions || [])) {
|
||||
if (act.type === 'tag' && act.tagKey) {
|
||||
if (!currentTags.includes(act.tagKey)) {
|
||||
currentTags.push(act.tagKey);
|
||||
await messenger.messages.update(id, { tags: currentTags });
|
||||
}
|
||||
await messenger.messages.update(id, { tags: [act.tagKey] });
|
||||
} else if (act.type === 'move' && act.folder) {
|
||||
await messenger.messages.move([id], act.folder);
|
||||
} else if (act.type === 'copy' && act.copyTarget) {
|
||||
await messenger.messages.copy([id], act.copyTarget);
|
||||
} else if (act.type === 'junk') {
|
||||
await messenger.messages.update(id, { junk: !!act.junk });
|
||||
} else if (act.type === 'read') {
|
||||
await messenger.messages.update(id, { read: !!act.read });
|
||||
} else if (act.type === 'flag') {
|
||||
await messenger.messages.update(id, { flagged: !!act.flagged });
|
||||
} else if (act.type === 'delete') {
|
||||
await messenger.messages.delete([id]);
|
||||
} else if (act.type === 'archive') {
|
||||
await messenger.messages.archive([id]);
|
||||
} else if (act.type === 'forward' && act.address && identityId) {
|
||||
await browser.compose.beginForward(id, { to: [act.address], identityId });
|
||||
} else if (act.type === 'reply' && act.replyType && identityId) {
|
||||
await browser.compose.beginReply(id, { replyType: act.replyType, identityId });
|
||||
}
|
||||
}
|
||||
if (rule.stopProcessing) {
|
||||
|
|
@ -411,351 +85,116 @@ async function processMessage(id) {
|
|||
}
|
||||
}
|
||||
}
|
||||
processing = false;
|
||||
const elapsed = Date.now() - currentStart;
|
||||
currentStart = 0;
|
||||
updateTimingStats(elapsed);
|
||||
await storage.local.set({ classifyStats: timingStats });
|
||||
showTransientIcon(ICONS.circle);
|
||||
} catch (e) {
|
||||
processing = false;
|
||||
const elapsed = Date.now() - currentStart;
|
||||
currentStart = 0;
|
||||
updateTimingStats(elapsed);
|
||||
await storage.local.set({ classifyStats: timingStats });
|
||||
logger.aiLog("failed to apply AI rules", { level: 'error' }, e);
|
||||
recordError("Failed to apply AI rules", e);
|
||||
browser.notifications.create(ERROR_NOTIFICATION_ID, {
|
||||
type: 'basic',
|
||||
iconUrl: browser.runtime.getURL('resources/img/logo.png'),
|
||||
title: 'Sortana Error',
|
||||
message: 'Failed to apply AI rules'
|
||||
});
|
||||
}
|
||||
}
|
||||
async function applyAiRules(idsInput) {
|
||||
const ids = Array.isArray(idsInput) ? idsInput : [idsInput];
|
||||
if (!ids.length) return queue;
|
||||
|
||||
if (!aiRules.length) {
|
||||
const { aiRules: stored } = await storage.local.get("aiRules");
|
||||
aiRules = normalizeRules(stored);
|
||||
}
|
||||
|
||||
for (const msg of ids) {
|
||||
const id = msg?.id ?? msg;
|
||||
queuedCount++;
|
||||
} finally {
|
||||
processing = false;
|
||||
updateActionIcon();
|
||||
queue = queue.then(() => processMessage(id));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return queue;
|
||||
}
|
||||
|
||||
async function clearCacheForMessages(idsInput) {
|
||||
const ids = Array.isArray(idsInput) ? idsInput : [idsInput];
|
||||
if (!ids.length) return;
|
||||
|
||||
if (!aiRules.length) {
|
||||
const { aiRules: stored } = await storage.local.get("aiRules");
|
||||
aiRules = normalizeRules(stored);
|
||||
}
|
||||
|
||||
for (const msg of ids) await AiClassifier.clearCacheForMessage(msg?.id ?? msg);
|
||||
showTransientIcon(ICONS.circle);
|
||||
}
|
||||
|
||||
(async () => {
|
||||
logger = await import(browser.runtime.getURL("logger.js"));
|
||||
({ detectSystemTheme } = await import(browser.runtime.getURL('modules/themeUtils.js')));
|
||||
try {
|
||||
AiClassifier = await import(browser.runtime.getURL("modules/AiClassifier.js"));
|
||||
logger.aiLog("AiClassifier imported", {debug: true});
|
||||
const td = await import(browser.runtime.getURL("resources/js/turndown.js"));
|
||||
TurndownService = td.default || td.TurndownService;
|
||||
} catch (e) {
|
||||
console.error("failed to import AiClassifier", e);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const store = await storage.local.get(["systemOneProcessAll", "systemOneBatchSize", "apiType", "decisionThreshold", "inputCharacterLimit", "endpoint", "model", "apiKey", "openaiOrganization", "openaiProject", "customSystemPrompt", "aiParams", "debugLogging", "htmlToMarkdown", "stripUrlParams", "altTextImages", "collapseWhitespace", "tokenReduction", "aiRules", "theme", "showDebugTab"]);
|
||||
const store = await storage.local.get(["endpoint", "templateName", "customTemplate", "customSystemPrompt", "aiParams", "debugLogging", "aiRules"]);
|
||||
logger.setDebug(store.debugLogging);
|
||||
const { migratedInputLimit } = await import(browser.runtime.getURL('modules/classificationConfig.js'));
|
||||
if (store.inputCharacterLimit === undefined) {
|
||||
store.inputCharacterLimit = migratedInputLimit(store);
|
||||
await storage.local.set({ inputCharacterLimit: store.inputCharacterLimit });
|
||||
}
|
||||
await AiClassifier.setConfig(store);
|
||||
userTheme = store.theme || 'auto';
|
||||
currentTheme = userTheme === 'auto' ? await detectSystemTheme() : userTheme;
|
||||
await AiClassifier.init();
|
||||
htmlToMarkdown = store.htmlToMarkdown === true;
|
||||
stripUrlParams = store.stripUrlParams === true;
|
||||
altTextImages = store.altTextImages === true;
|
||||
collapseWhitespace = store.collapseWhitespace === true;
|
||||
tokenReduction = store.tokenReduction === true;
|
||||
showDebugTab = store.showDebugTab === true;
|
||||
const savedStats = await storage.local.get('classifyStats');
|
||||
if (savedStats.classifyStats && typeof savedStats.classifyStats === 'object') {
|
||||
Object.assign(timingStats, savedStats.classifyStats);
|
||||
}
|
||||
if (typeof timingStats.last !== 'number') {
|
||||
timingStats.last = -1;
|
||||
}
|
||||
aiRules = normalizeRules(store.aiRules);
|
||||
aiRules = Array.isArray(store.aiRules) ? store.aiRules.map(r => {
|
||||
if (r.actions) return r;
|
||||
const actions = [];
|
||||
if (r.tag) actions.push({ type: 'tag', tagKey: r.tag });
|
||||
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
|
||||
const rule = { criterion: r.criterion, actions };
|
||||
if (r.stopProcessing) rule.stopProcessing = true;
|
||||
return rule;
|
||||
}) : [];
|
||||
logger.aiLog("configuration loaded", {debug: true}, store);
|
||||
storage.onChanged.addListener(async changes => {
|
||||
if (changes.aiRules) {
|
||||
const newRules = changes.aiRules.newValue || [];
|
||||
aiRules = normalizeRules(newRules);
|
||||
aiRules = newRules.map(r => {
|
||||
if (r.actions) return r;
|
||||
const actions = [];
|
||||
if (r.tag) actions.push({ type: 'tag', tagKey: r.tag });
|
||||
if (r.moveTo) actions.push({ type: 'move', folder: r.moveTo });
|
||||
const rule = { criterion: r.criterion, actions };
|
||||
if (r.stopProcessing) rule.stopProcessing = true;
|
||||
return rule;
|
||||
});
|
||||
logger.aiLog("aiRules updated from storage change", {debug: true}, aiRules);
|
||||
}
|
||||
const configChanges = Object.fromEntries(Object.entries(changes).map(([key, change]) => [key, change.newValue]));
|
||||
AiClassifier.setConfig(configChanges);
|
||||
if (changes.aiCache) AiClassifier.replaceCache(changes.aiCache.newValue);
|
||||
if (changes.htmlToMarkdown) {
|
||||
htmlToMarkdown = changes.htmlToMarkdown.newValue === true;
|
||||
logger.aiLog("htmlToMarkdown updated from storage change", { debug: true }, htmlToMarkdown);
|
||||
}
|
||||
if (changes.stripUrlParams) {
|
||||
stripUrlParams = changes.stripUrlParams.newValue === true;
|
||||
logger.aiLog("stripUrlParams updated from storage change", { debug: true }, stripUrlParams);
|
||||
}
|
||||
if (changes.altTextImages) {
|
||||
altTextImages = changes.altTextImages.newValue === true;
|
||||
logger.aiLog("altTextImages updated from storage change", { debug: true }, altTextImages);
|
||||
}
|
||||
if (changes.collapseWhitespace) {
|
||||
collapseWhitespace = changes.collapseWhitespace.newValue === true;
|
||||
logger.aiLog("collapseWhitespace updated from storage change", { debug: true }, collapseWhitespace);
|
||||
}
|
||||
if (changes.tokenReduction) {
|
||||
tokenReduction = changes.tokenReduction.newValue === true;
|
||||
logger.aiLog("tokenReduction updated from storage change", { debug: true }, tokenReduction);
|
||||
}
|
||||
if (changes.showDebugTab) {
|
||||
showDebugTab = changes.showDebugTab.newValue === true;
|
||||
}
|
||||
if (changes.theme) {
|
||||
userTheme = changes.theme.newValue || 'auto';
|
||||
currentTheme = userTheme === 'auto' ? await detectSystemTheme() : userTheme;
|
||||
updateActionIcon();
|
||||
refreshMenuIcons();
|
||||
}
|
||||
});
|
||||
|
||||
if (browser.theme?.onUpdated) {
|
||||
browser.theme.onUpdated.addListener(async () => {
|
||||
if (userTheme === 'auto') {
|
||||
const theme = await detectSystemTheme();
|
||||
if (theme !== currentTheme) {
|
||||
currentTheme = theme;
|
||||
updateActionIcon();
|
||||
refreshMenuIcons();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
logger.aiLog("failed to load config", {level: 'error'}, err);
|
||||
}
|
||||
|
||||
logger.aiLog("background.js loaded – ready to classify", {debug: true});
|
||||
updateActionIcon();
|
||||
if (browser.messageDisplayAction) {
|
||||
browser.messageDisplayAction.setTitle({ title: "Details" });
|
||||
if (browser.messageDisplayAction.setLabel) {
|
||||
browser.messageDisplayAction.setLabel({ label: "Details" });
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
browser.menus.create({
|
||||
id: "apply-ai-rules-list",
|
||||
title: "Apply AI Rules",
|
||||
contexts: ["message_list"],
|
||||
icons: iconPaths('eye')
|
||||
});
|
||||
browser.menus.create({
|
||||
id: "apply-ai-rules-display",
|
||||
title: "Apply AI Rules",
|
||||
contexts: ["message_display_action"],
|
||||
icons: iconPaths('eye')
|
||||
});
|
||||
browser.menus.create({
|
||||
id: "clear-ai-cache-list",
|
||||
title: "Clear AI Cache",
|
||||
contexts: ["message_list"],
|
||||
icons: iconPaths('trash')
|
||||
});
|
||||
browser.menus.create({
|
||||
id: "clear-ai-cache-display",
|
||||
title: "Clear AI Cache",
|
||||
contexts: ["message_display_action"],
|
||||
icons: iconPaths('trash')
|
||||
});
|
||||
browser.menus.create({
|
||||
id: "view-ai-reason-list",
|
||||
title: "View Reasoning",
|
||||
contexts: ["message_list"],
|
||||
icons: iconPaths('clipboarddata')
|
||||
});
|
||||
browser.menus.create({
|
||||
id: "view-ai-reason-display",
|
||||
title: "View Reasoning",
|
||||
contexts: ["message_display_action"],
|
||||
icons: iconPaths('clipboarddata')
|
||||
});
|
||||
refreshMenuIcons();
|
||||
|
||||
browser.menus.onClicked.addListener(async (info, tab) => {
|
||||
if (info.menuItemId === "apply-ai-rules-list" || info.menuItemId === "apply-ai-rules-display") {
|
||||
let ids = info.messageId ? [info.messageId] : [];
|
||||
if (info.selectedMessages) {
|
||||
ids = await getAllMessageIds(info.selectedMessages);
|
||||
}
|
||||
if (browser.messageDisplayAction) {
|
||||
browser.messageDisplayAction.onClicked.addListener(async (tab) => {
|
||||
try {
|
||||
const msgs = await browser.messageDisplay.getDisplayedMessages(tab.id);
|
||||
const ids = msgs.map(m => m.id);
|
||||
await applyAiRules(ids);
|
||||
} else if (info.menuItemId === "clear-ai-cache-list" || info.menuItemId === "clear-ai-cache-display") {
|
||||
let ids = info.messageId ? [info.messageId] : [];
|
||||
if (info.selectedMessages) {
|
||||
ids = await getAllMessageIds(info.selectedMessages);
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to apply AI rules from action", { level: 'error' }, e);
|
||||
}
|
||||
});
|
||||
}
|
||||
await clearCacheForMessages(ids);
|
||||
} else if (info.menuItemId === "view-ai-reason-list" || info.menuItemId === "view-ai-reason-display") {
|
||||
const [header] = await browser.messageDisplay.getDisplayedMessages(tab.id);
|
||||
if (!header) { return; }
|
||||
|
||||
const url = `${browser.runtime.getURL("details.html")}?mid=${header.id}`;
|
||||
|
||||
await browser.tabs.create({ url });
|
||||
browser.menus.onClicked.addListener(async info => {
|
||||
if (info.menuItemId === "apply-ai-rules-list" || info.menuItemId === "apply-ai-rules-display") {
|
||||
const ids = info.selectedMessages?.messages?.map(m => m.id) ||
|
||||
(info.messageId ? [info.messageId] : []);
|
||||
await applyAiRules(ids);
|
||||
}
|
||||
});
|
||||
|
||||
// Listen for messages from UI/devtools
|
||||
browser.runtime.onMessage.addListener(async (msg) => {
|
||||
if ((msg?.type === "sortana:getTiming" && logGetTiming) || (msg?.type !== "sortana:getTiming")) {
|
||||
logGetTiming = false;
|
||||
logger.aiLog("onMessage received", {debug: true}, msg);
|
||||
}
|
||||
|
||||
if (msg?.type === "sortana:test") {
|
||||
if (msg?.type === "aiFilter:test") {
|
||||
const { text = "", criterion = "" } = msg;
|
||||
logger.aiLog("sortana:test – text", { debug: true }, text);
|
||||
logger.aiLog("sortana:test – criterion", { debug: true }, criterion);
|
||||
logger.aiLog("aiFilter:test – text", {debug: true}, text);
|
||||
logger.aiLog("aiFilter:test – criterion", {debug: true}, criterion);
|
||||
|
||||
try {
|
||||
logger.aiLog("Calling AiClassifier.classify()", { debug: true });
|
||||
const result = await AiClassifier.classify(text, criterion, { threshold: msg.decisionThreshold ?? AiClassifier.getConfig().decisionThreshold });
|
||||
logger.aiLog("Calling AiClassifier.classifyText()", {debug: true});
|
||||
const result = await AiClassifier.classifyText(text, criterion);
|
||||
logger.aiLog("classify() returned", {debug: true}, result);
|
||||
return { ...result, match: result.matched };
|
||||
return { match: result };
|
||||
}
|
||||
catch (err) {
|
||||
logger.aiLog("Error in classify()", {level: 'error'}, err);
|
||||
// rethrow so the caller sees the failure
|
||||
throw err;
|
||||
}
|
||||
} else if (msg?.type === "sortana:clearCacheForDisplayed") {
|
||||
try {
|
||||
const msgs = await browser.messageDisplay.getDisplayedMessages();
|
||||
const ids = msgs.map(m => m.id);
|
||||
await clearCacheForMessages(ids);
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to clear cache from message script", { level: 'error' }, e);
|
||||
}
|
||||
} else if (msg?.type === "sortana:getReasons") {
|
||||
try {
|
||||
const id = msg.id;
|
||||
const hdr = await messenger.messages.get(id);
|
||||
const subject = hdr?.subject || "";
|
||||
if (!aiRules.length) {
|
||||
const { aiRules: stored } = await storage.local.get("aiRules");
|
||||
aiRules = normalizeRules(stored);
|
||||
}
|
||||
const reasons = [];
|
||||
for (const rule of aiRules) {
|
||||
const key = await AiClassifier.buildCacheKey(id, rule.criterion);
|
||||
const reason = AiClassifier.getReason(key);
|
||||
if (reason) {
|
||||
reasons.push({ criterion: rule.criterion, reason });
|
||||
}
|
||||
}
|
||||
return { subject, reasons };
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to collect reasons", { level: 'error' }, e);
|
||||
return { subject: '', reasons: [] };
|
||||
}
|
||||
} else if (msg?.type === "sortana:getDetails") {
|
||||
try {
|
||||
const id = msg.id;
|
||||
const hdr = await messenger.messages.get(id);
|
||||
const subject = hdr?.subject || "";
|
||||
if (!aiRules.length) {
|
||||
const { aiRules: stored } = await storage.local.get("aiRules");
|
||||
aiRules = normalizeRules(stored);
|
||||
}
|
||||
const results = [];
|
||||
const snapshot = AiClassifier.getConfig();
|
||||
for (const rule of structuredClone(aiRules)) {
|
||||
const key = await AiClassifier.buildCacheKey(id, rule.criterion, snapshot);
|
||||
const detail = AiClassifier.getCachedDetails(key, rule.decisionThreshold ?? snapshot.decisionThreshold);
|
||||
if (detail) results.push({ criterion: rule.criterion, ...detail });
|
||||
}
|
||||
return { subject, results };
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to collect details", { level: 'error' }, e);
|
||||
return { subject: '', results: [] };
|
||||
}
|
||||
} else if (msg?.type === "sortana:getDisplayedMessages") {
|
||||
try {
|
||||
const [tab] = await browser.tabs.query({ active: true, currentWindow: true });
|
||||
const messages = await browser.messageDisplay.getDisplayedMessages(tab?.id);
|
||||
const ids = messages.map(hdr => hdr.id);
|
||||
|
||||
return { ids };
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to get displayed messages", { level: 'error' }, e);
|
||||
return { messages: [] };
|
||||
}
|
||||
} else if (msg?.type === "sortana:clearCacheForMessage") {
|
||||
try {
|
||||
await clearCacheForMessages([msg.id]);
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
logger.aiLog("failed to clear cache for message", { level: 'error' }, e);
|
||||
return { ok: false };
|
||||
}
|
||||
} else if (msg?.type === "sortana:resetTimingStats") {
|
||||
const last = timingStats.last;
|
||||
timingStats.count = 0;
|
||||
timingStats.mean = 0;
|
||||
timingStats.m2 = 0;
|
||||
timingStats.total = 0;
|
||||
timingStats.last = typeof last === 'number' ? last : -1;
|
||||
await storage.local.set({ classifyStats: timingStats });
|
||||
return { ok: true };
|
||||
} else if (msg?.type === "sortana:getCacheSize") {
|
||||
return { count: await AiClassifier.getCacheSize() };
|
||||
} else if (msg?.type === "sortana:getQueueCount") {
|
||||
return { count: queuedCount + (processing ? 1 : 0) };
|
||||
} else if (msg?.type === "sortana:getErrorLog") {
|
||||
return { errors: errorLog.slice() };
|
||||
} else if (msg?.type === "sortana:recordError") {
|
||||
recordError(msg.context || "Sortana Error", { message: msg.message, detail: msg.detail });
|
||||
return { ok: true };
|
||||
} else if (msg?.type === "sortana:getTiming") {
|
||||
const t = timingStats;
|
||||
const std = t.count > 1 ? Math.sqrt(t.m2 / (t.count - 1)) : 0;
|
||||
return {
|
||||
count: queuedCount + (processing ? 1 : 0),
|
||||
current: currentStart ? Date.now() - currentStart : -1,
|
||||
last: t.last,
|
||||
runs: t.count,
|
||||
average: t.mean,
|
||||
total: t.total,
|
||||
stddev: std
|
||||
};
|
||||
} else {
|
||||
else {
|
||||
logger.aiLog("Unknown message type, ignoring", {level: 'warn'}, msg?.type);
|
||||
}
|
||||
});
|
||||
|
|
@ -774,19 +213,6 @@ async function clearCacheForMessages(idsInput) {
|
|||
// Catch any unhandled rejections
|
||||
window.addEventListener("unhandledrejection", ev => {
|
||||
logger.aiLog("Unhandled promise rejection", {level: 'error'}, ev.reason);
|
||||
recordError("Unhandled promise rejection", ev.reason);
|
||||
});
|
||||
|
||||
browser.notifications.onClicked.addListener(id => {
|
||||
if (id === ERROR_NOTIFICATION_ID) {
|
||||
clearError();
|
||||
}
|
||||
});
|
||||
|
||||
browser.notifications.onButtonClicked.addListener((id) => {
|
||||
if (id === ERROR_NOTIFICATION_ID) {
|
||||
clearError();
|
||||
}
|
||||
});
|
||||
|
||||
browser.runtime.onInstalled.addListener(async ({ reason }) => {
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ if (-not $version) {
|
|||
}
|
||||
|
||||
# 4) Define output names & clean up
|
||||
$xpiName = "sortana-$version.xpi"
|
||||
$xpiName = "ai-filter-$version.xpi"
|
||||
$zipPath = Join-Path $ReleaseDir "ai-filter-$version.zip"
|
||||
$xpiPath = Join-Path $ReleaseDir $xpiName
|
||||
|
||||
|
|
|
|||
77
build-xpi.sh
|
|
@ -1,77 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
release_dir="$script_dir/release"
|
||||
manifest="$script_dir/manifest.json"
|
||||
|
||||
if [[ ! -f "$manifest" ]]; then
|
||||
echo "manifest.json not found at $manifest" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v zip >/dev/null 2>&1; then
|
||||
echo "zip is required to build the XPI." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if command -v jq >/dev/null 2>&1; then
|
||||
version="$(jq -r '.version // empty' "$manifest")"
|
||||
else
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "python3 is required to read manifest.json without jq." >&2
|
||||
exit 1
|
||||
fi
|
||||
version="$(python3 - <<'PY'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
print(data.get('version', '') or '')
|
||||
PY
|
||||
"$manifest")"
|
||||
fi
|
||||
|
||||
if [[ -z "$version" ]]; then
|
||||
echo "No version found in manifest.json" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$release_dir"
|
||||
|
||||
xpi_name="sortana-$version.xpi"
|
||||
zip_path="$release_dir/ai-filter-$version.zip"
|
||||
xpi_path="$release_dir/$xpi_name"
|
||||
|
||||
rm -f "$zip_path" "$xpi_path"
|
||||
|
||||
mapfile -d '' files < <(
|
||||
find "$script_dir" -type f \
|
||||
! -name '*.sln' \
|
||||
! -name '*.ps1' \
|
||||
! -name '*.sh' \
|
||||
! -path "$release_dir/*" \
|
||||
! -path "$script_dir/.vs/*" \
|
||||
! -path "$script_dir/.git/*" \
|
||||
-printf '%P\0'
|
||||
)
|
||||
|
||||
if [[ ${#files[@]} -eq 0 ]]; then
|
||||
echo "No files found to package." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
for rel in "${files[@]}"; do
|
||||
full="$script_dir/$rel"
|
||||
size=$(stat -c '%s' "$full")
|
||||
echo "Zipping: $rel <- $full ($size bytes)"
|
||||
done
|
||||
|
||||
(
|
||||
cd "$script_dir"
|
||||
printf '%s\n' "${files[@]}" | zip -q -9 -@ "$zip_path"
|
||||
)
|
||||
|
||||
mv -f "$zip_path" "$xpi_path"
|
||||
|
||||
echo "Built XPI at: $xpi_path"
|
||||
47
details.html
|
|
@ -1,47 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>AI Details</title>
|
||||
<link rel="stylesheet" href="options/bulma.css">
|
||||
<style>
|
||||
.probability-bar {
|
||||
position: relative;
|
||||
height: 1.75rem;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--bulma-border);
|
||||
border-radius: var(--bulma-radius-small);
|
||||
background: var(--bulma-background);
|
||||
}
|
||||
.probability-fill {
|
||||
height: 100%;
|
||||
background: var(--bulma-success);
|
||||
opacity: 0.35;
|
||||
}
|
||||
.probability-bar.is-danger .probability-fill {
|
||||
background: var(--bulma-danger);
|
||||
}
|
||||
.probability-value {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
color: var(--bulma-text-strong);
|
||||
font-weight: 600;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<section class="section">
|
||||
<div class="container">
|
||||
<h1 class="title" id="subject"></h1>
|
||||
<div id="rules"></div>
|
||||
<div class="buttons mt-4">
|
||||
<button class="button is-danger" id="clear">Clear Cache</button>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<script type="module" src="details.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
99
details.js
|
|
@ -1,99 +0,0 @@
|
|||
const aiLog = (await import(browser.runtime.getURL("logger.js"))).aiLog;
|
||||
const storage = (globalThis.messenger ?? browser).storage;
|
||||
const { detectSystemTheme } = await import(browser.runtime.getURL('modules/themeUtils.js'));
|
||||
const { theme } = await storage.local.get('theme');
|
||||
const mode = (theme || 'auto') === 'auto'
|
||||
? await detectSystemTheme()
|
||||
: theme;
|
||||
document.documentElement.dataset.theme = mode;
|
||||
|
||||
const qMid = parseInt(new URLSearchParams(location.search).get("mid"), 10);
|
||||
if (!isNaN(qMid)) {
|
||||
loadMessage(qMid);
|
||||
} else {
|
||||
const { ids } = await browser.runtime.sendMessage({
|
||||
type: "sortana:getDisplayedMessages",
|
||||
});
|
||||
if (ids && ids[0]) {
|
||||
loadMessage(ids[0]);
|
||||
} else {
|
||||
const tabs = await browser.tabs.query({ active: true, currentWindow: true });
|
||||
const tabId = tabs[0]?.id;
|
||||
const msgs = tabId ? await browser.messageDisplay.getDisplayedMessages(tabId) : [];
|
||||
let id = msgs[0]?.id;
|
||||
if (id) {
|
||||
loadMessage(id);
|
||||
}
|
||||
else {
|
||||
aiLog("Details popup: no displayed message found");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function loadMessage(id) {
|
||||
const storage = (globalThis.messenger ?? browser).storage;
|
||||
const logMod = await import(browser.runtime.getURL('logger.js'));
|
||||
const { debugLogging } = await storage.local.get('debugLogging');
|
||||
logMod.setDebug(debugLogging === true);
|
||||
const log = logMod.aiLog;
|
||||
|
||||
log('details page loaded', { debug: true });
|
||||
try {
|
||||
log('requesting message details', {}, id);
|
||||
const { subject, results } = await browser.runtime.sendMessage({ type: 'sortana:getDetails', id });
|
||||
log('received details', { debug: true }, { subject, results });
|
||||
document.getElementById('subject').textContent = subject;
|
||||
const container = document.getElementById('rules');
|
||||
for (const r of results) {
|
||||
log('rendering rule result', { debug: true }, r);
|
||||
const article = document.createElement('article');
|
||||
const color = r.matched === true ? 'is-success' : 'is-danger';
|
||||
article.className = `message ${color} mb-4`;
|
||||
const header = document.createElement('div');
|
||||
header.className = 'message-header';
|
||||
const criterion = document.createElement('p');
|
||||
criterion.textContent = r.criterion;
|
||||
header.appendChild(criterion);
|
||||
const body = document.createElement('div');
|
||||
body.className = 'message-body';
|
||||
const status = document.createElement('p');
|
||||
status.textContent = r.matched ? 'Matched' : 'Did not match';
|
||||
if (r.backend === 'systemone') {
|
||||
status.textContent += ' under current threshold';
|
||||
status.classList.add('mb-2');
|
||||
body.appendChild(status);
|
||||
const percentage = r.probability * 100;
|
||||
const bar = document.createElement('div');
|
||||
bar.className = `probability-bar ${color}`;
|
||||
bar.setAttribute('role', 'progressbar');
|
||||
bar.setAttribute('aria-label', 'Match probability');
|
||||
bar.setAttribute('aria-valuemin', '0');
|
||||
bar.setAttribute('aria-valuemax', '100');
|
||||
bar.setAttribute('aria-valuenow', String(percentage));
|
||||
bar.title = `Threshold: ${r.threshold * 100}%${r.model ? ` · Model: ${r.model}` : ''}`;
|
||||
const fill = document.createElement('div');
|
||||
fill.className = 'probability-fill';
|
||||
fill.style.width = `${percentage}%`;
|
||||
const value = document.createElement('span');
|
||||
value.className = 'probability-value';
|
||||
value.textContent = `${percentage.toFixed(2)}%`;
|
||||
bar.append(fill, value);
|
||||
body.appendChild(bar);
|
||||
} else {
|
||||
const pre = document.createElement('pre');
|
||||
pre.textContent = r.reason || '';
|
||||
body.append(status, pre);
|
||||
}
|
||||
article.appendChild(header);
|
||||
article.appendChild(body);
|
||||
container.appendChild(article);
|
||||
}
|
||||
document.getElementById('clear').addEventListener('click', async () => {
|
||||
log('clearing cache for message', {}, id);
|
||||
await browser.runtime.sendMessage({ type: 'sortana:clearCacheForMessage', id });
|
||||
window.close();
|
||||
});
|
||||
} catch (e) {
|
||||
log('failed to load details', { level: 'error' }, e);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,12 +1,13 @@
|
|||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Sortana",
|
||||
"version": "4.1.0",
|
||||
"version": "2.0.0",
|
||||
"default_locale": "en-US",
|
||||
"applications": {
|
||||
"gecko": {
|
||||
"id": "ai-filter@jordanwages",
|
||||
"strict_min_version": "128.0"
|
||||
"strict_min_version": "128.0",
|
||||
"strict_max_version": "*"
|
||||
}
|
||||
},
|
||||
"icons": {
|
||||
|
|
@ -21,10 +22,7 @@
|
|||
"default_icon": "resources/img/logo32.png"
|
||||
},
|
||||
"message_display_action": {
|
||||
"default_icon": "resources/img/logo.png",
|
||||
"default_title": "Details",
|
||||
"default_label": "Details",
|
||||
"default_popup": "details.html"
|
||||
"default_icon": "resources/img/logo32.png"
|
||||
},
|
||||
"background": { "scripts": [ "background.js" ] },
|
||||
"options_ui": {
|
||||
|
|
@ -38,11 +36,6 @@
|
|||
"messagesUpdate",
|
||||
"messagesTagsList",
|
||||
"accountsRead",
|
||||
"menus",
|
||||
"notifications",
|
||||
"scripting",
|
||||
"tabs",
|
||||
"theme",
|
||||
"compose"
|
||||
"menus"
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,209 +1,291 @@
|
|||
"use strict";
|
||||
import { aiLog, setDebug } from '../logger.js';
|
||||
import { CONFIG_DEFAULTS, validateSettings, validateThreshold } from './classificationConfig.js';
|
||||
import * as chat from './backends/chatCompletions.js';
|
||||
import * as systemone from './backends/systemOne.js';
|
||||
import { aiLog, setDebug } from "../logger.js";
|
||||
|
||||
const storage = (globalThis.messenger ?? globalThis.browser).storage;
|
||||
const adapters = { 'chat-completions': chat, systemone };
|
||||
let config = structuredClone(CONFIG_DEFAULTS);
|
||||
let cache = new Map();
|
||||
let loaded = false;
|
||||
let loading;
|
||||
let generation = 0;
|
||||
|
||||
export function normalizeEndpointBase(endpoint) {
|
||||
if (typeof endpoint !== 'string' || !endpoint.trim()) return '';
|
||||
return endpoint.trim().replace(/\/+$/, '').replace(/\/v1(?:\/(?:chat\/completions|completions|models|systemone))?$/i, '');
|
||||
}
|
||||
function urlFor(endpoint, path) {
|
||||
const base = normalizeEndpointBase(endpoint);
|
||||
return base ? `${/^https?:\/\//i.test(base) ? base : `https://${base}`}${path}` : '';
|
||||
}
|
||||
export function buildEndpointUrl(endpoint, apiType = 'chat-completions') {
|
||||
if (!adapters[apiType]) throw new Error('Unknown API type.');
|
||||
return urlFor(endpoint, adapters[apiType].path);
|
||||
}
|
||||
export function buildModelsUrl(endpoint) { return urlFor(endpoint, '/v1/models'); }
|
||||
export function getConfig() { return structuredClone(config); }
|
||||
export function setConfig(update = {}) {
|
||||
validateSettings(update);
|
||||
for (const key of Object.keys(CONFIG_DEFAULTS)) {
|
||||
if (update[key] !== undefined) config[key] = structuredClone(update[key]);
|
||||
}
|
||||
config.aiParams = { ...CONFIG_DEFAULTS.aiParams, ...config.aiParams };
|
||||
for (const key of ['model', 'apiKey', 'openaiOrganization', 'openaiProject']) config[key] = String(config[key]).trim();
|
||||
setDebug(config.debugLogging);
|
||||
}
|
||||
async function hash(value) {
|
||||
const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(value));
|
||||
return Array.from(new Uint8Array(digest), b => b.toString(16).padStart(2, '0')).join('');
|
||||
}
|
||||
async function messageHash(id) {
|
||||
const api = globalThis.messenger ?? globalThis.browser;
|
||||
if (typeof id === 'number' && api.messages?.get) {
|
||||
const header = await api.messages.get(id);
|
||||
id = header.headerMessageId || id;
|
||||
}
|
||||
return hash(String(id));
|
||||
}
|
||||
function stable(value) {
|
||||
if (Array.isArray(value)) return value.map(stable);
|
||||
if (value && typeof value === 'object') return Object.fromEntries(Object.keys(value).sort().map(k => [k, stable(value[k])]));
|
||||
return value;
|
||||
}
|
||||
export async function buildCacheKey(id, criterion, snapshot = getConfig()) {
|
||||
const fingerprint = {
|
||||
backend: snapshot.apiType, endpoint: buildEndpointUrl(snapshot.endpoint, snapshot.apiType),
|
||||
request: adapters[snapshot.apiType].buildRequest('', criterion, snapshot),
|
||||
organization: snapshot.apiType === 'chat-completions' ? snapshot.openaiOrganization : '',
|
||||
project: snapshot.apiType === 'chat-completions' ? snapshot.openaiProject : '',
|
||||
preprocessing: Object.fromEntries(['htmlToMarkdown', 'stripUrlParams', 'altTextImages', 'collapseWhitespace', 'tokenReduction', 'inputCharacterLimit'].map(k => [k, snapshot[k]]))
|
||||
};
|
||||
return `v2:${await messageHash(id)}:${await hash(JSON.stringify(stable(fingerprint)))}`;
|
||||
}
|
||||
export function validCacheEntry(key, entry) {
|
||||
return /^v2:[a-f0-9]{64}:[a-f0-9]{64}$/.test(key) && entry?.version === 2 &&
|
||||
entry.messageHash === key.split(':')[1] && typeof entry.model === 'string' &&
|
||||
(entry.backend === 'systemone'
|
||||
? typeof entry.probability === 'number' && Number.isFinite(entry.probability) && entry.probability >= 0 && entry.probability <= 1 && entry.reason === null
|
||||
: entry.backend === 'chat-completions' && typeof entry.matched === 'boolean' && entry.probability === null && (entry.reason === null || typeof entry.reason === 'string'));
|
||||
}
|
||||
export function replaceCache(data = {}) {
|
||||
cache = new Map(Object.entries(data || {}).filter(([key, entry]) => validCacheEntry(key, entry)));
|
||||
loaded = true;
|
||||
generation++;
|
||||
}
|
||||
async function saveCache() { await storage.local.set({ aiCache: Object.fromEntries(cache) }); }
|
||||
export async function init() {
|
||||
if (loaded) return;
|
||||
if (!loading) loading = (async () => {
|
||||
const { aiCache = {}, aiReasonCache } = await storage.local.get(['aiCache', 'aiReasonCache']);
|
||||
replaceCache(aiCache);
|
||||
if (Object.keys(aiCache || {}).length !== cache.size) await saveCache();
|
||||
if (aiReasonCache !== undefined) await storage.local.remove('aiReasonCache');
|
||||
})().finally(() => { loading = null; });
|
||||
await loading;
|
||||
}
|
||||
function withPolicy(entry, threshold) {
|
||||
const isDecision = entry.backend === 'systemone';
|
||||
if (isDecision) validateThreshold(threshold);
|
||||
return { matched: isDecision ? entry.probability >= threshold : entry.matched,
|
||||
reason: entry.reason, probability: entry.probability, threshold: isDecision ? threshold : null,
|
||||
backend: entry.backend, model: entry.model };
|
||||
}
|
||||
export function getCachedDetails(key, threshold = config.decisionThreshold) {
|
||||
const entry = cache.get(key);
|
||||
return entry ? withPolicy(entry, threshold) : null;
|
||||
}
|
||||
export function getCachedResult(key, threshold) { return getCachedDetails(key, threshold)?.matched ?? null; }
|
||||
export function getReason(key) { return cache.get(key)?.reason ?? null; }
|
||||
async function sendRequest(payload, snapshot) {
|
||||
try { await storage.local.set({ lastPayload: payload }); }
|
||||
catch (error) { aiLog('Failed to save last payload', { level: 'warn' }, error); }
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
if (snapshot.apiKey) headers.Authorization = `Bearer ${snapshot.apiKey}`;
|
||||
if (snapshot.apiType === 'chat-completions') {
|
||||
if (snapshot.openaiOrganization) headers['OpenAI-Organization'] = snapshot.openaiOrganization;
|
||||
if (snapshot.openaiProject) headers['OpenAI-Project'] = snapshot.openaiProject;
|
||||
}
|
||||
const endpoint = buildEndpointUrl(snapshot.endpoint, snapshot.apiType);
|
||||
const response = await fetch(endpoint, { method: 'POST', headers, body: JSON.stringify(payload) });
|
||||
if (!response.ok) {
|
||||
let detail = '';
|
||||
let Services;
|
||||
try {
|
||||
const body = await response.json();
|
||||
const message = body?.error?.message;
|
||||
if (typeof message === 'string') detail = ` Server response: ${message.slice(0, 300)}.`;
|
||||
} catch { /* Non-JSON error responses still report the URL and status. */ }
|
||||
const hint = response.status === 404 && snapshot.apiType === 'systemone'
|
||||
? ' Verify that the running server build supports /v1/systemone and the selected decision model.' : '';
|
||||
throw new Error(`Classification request failed: HTTP ${response.status} at ${endpoint}.${detail}${hint}`);
|
||||
if (typeof globalThis !== "undefined" && globalThis.Services) {
|
||||
Services = globalThis.Services;
|
||||
} else if (typeof ChromeUtils !== "undefined" && ChromeUtils.importESModule) {
|
||||
({ Services } = ChromeUtils.importESModule("resource://gre/modules/Services.sys.mjs"));
|
||||
}
|
||||
return response.json();
|
||||
} catch (e) {
|
||||
Services = undefined;
|
||||
}
|
||||
|
||||
export async function classify(text, criterion, { cacheKey = null, snapshot = getConfig(), threshold = snapshot.decisionThreshold } = {}) {
|
||||
await init();
|
||||
if (snapshot.apiType === 'systemone') validateThreshold(threshold);
|
||||
const cached = getCachedDetails(cacheKey, threshold);
|
||||
if (cached) return cached;
|
||||
const epoch = generation;
|
||||
const adapter = adapters[snapshot.apiType];
|
||||
const payload = adapter.buildRequest(text, criterion, snapshot);
|
||||
const raw = await sendRequest(payload, snapshot);
|
||||
const entry = { ...adapter.parseResponse(raw), backend: snapshot.apiType,
|
||||
model: typeof raw.model === 'string' ? raw.model : snapshot.model, version: 2,
|
||||
messageHash: cacheKey?.split(':')[1] };
|
||||
if (cacheKey && epoch === generation) {
|
||||
cache.set(cacheKey, entry);
|
||||
await saveCache();
|
||||
}
|
||||
return withPolicy(entry, threshold);
|
||||
}
|
||||
// Yield lazily so the action runner can stop without starting the next request.
|
||||
// Candidates have precomputed keys, allowing actions to move/delete the message.
|
||||
export async function* classifyRules(text, candidates, snapshot = getConfig()) {
|
||||
await init();
|
||||
validateSettings(snapshot);
|
||||
if (snapshot.apiType !== 'systemone') {
|
||||
for (const candidate of candidates) {
|
||||
const result = await classify(text, candidate.rule.criterion, {
|
||||
cacheKey: candidate.cacheKey, snapshot,
|
||||
threshold: candidate.rule.decisionThreshold ?? snapshot.decisionThreshold
|
||||
});
|
||||
yield { rule: candidate.rule, result };
|
||||
}
|
||||
const SYSTEM_PREFIX = `You are an email-classification assistant.
|
||||
Read the email below and the classification criterion provided by the user.
|
||||
`;
|
||||
|
||||
const DEFAULT_CUSTOM_SYSTEM_PROMPT = "Determine whether the email satisfies the user's criterion.";
|
||||
|
||||
const SYSTEM_SUFFIX = `
|
||||
Return ONLY a JSON object on a single line of the form:
|
||||
{"match": true} - if the email satisfies the criterion
|
||||
{"match": false} - otherwise
|
||||
|
||||
Do not add any other keys, text, or formatting.`;
|
||||
|
||||
let gEndpoint = "http://127.0.0.1:5000/v1/classify";
|
||||
let gTemplateName = "openai";
|
||||
let gCustomTemplate = "";
|
||||
let gCustomSystemPrompt = DEFAULT_CUSTOM_SYSTEM_PROMPT;
|
||||
let gTemplateText = "";
|
||||
|
||||
let gAiParams = {
|
||||
max_tokens: 4096,
|
||||
temperature: 0.6,
|
||||
top_p: 0.95,
|
||||
seed: -1,
|
||||
repetition_penalty: 1.0,
|
||||
top_k: 20,
|
||||
min_p: 0,
|
||||
presence_penalty: 0,
|
||||
frequency_penalty: 0,
|
||||
typical_p: 1,
|
||||
tfs: 1,
|
||||
};
|
||||
|
||||
let gCache = new Map();
|
||||
let gCacheLoaded = false;
|
||||
|
||||
async function loadCache() {
|
||||
if (gCacheLoaded) {
|
||||
return;
|
||||
}
|
||||
const limit = snapshot.systemOneProcessAll ? Infinity : snapshot.systemOneBatchSize;
|
||||
// Keep received evidence locally even if persistence is cleared during this run.
|
||||
const evidence = new Map();
|
||||
for (const candidate of candidates) {
|
||||
validateThreshold(candidate.rule.decisionThreshold ?? snapshot.decisionThreshold);
|
||||
const entry = cache.get(candidate.cacheKey);
|
||||
if (entry) evidence.set(candidate.cacheKey, entry);
|
||||
aiLog(`[AiClassifier] Loading cache`, {debug: true});
|
||||
try {
|
||||
const { aiCache } = await storage.local.get("aiCache");
|
||||
if (aiCache) {
|
||||
for (let [k, v] of Object.entries(aiCache)) {
|
||||
aiLog(`[AiClassifier] ⮡ Loaded entry '${k}' → ${v}`, {debug: true});
|
||||
gCache.set(k, v);
|
||||
}
|
||||
for (let index = 0; index < candidates.length; index++) {
|
||||
const candidate = candidates[index];
|
||||
if (!evidence.has(candidate.cacheKey)) {
|
||||
const pending = new Map();
|
||||
for (let next = index; next < candidates.length && pending.size < limit; next++) {
|
||||
const item = candidates[next];
|
||||
if (!evidence.has(item.cacheKey) && !pending.has(item.cacheKey)) {
|
||||
pending.set(item.cacheKey, { ...item, criterion: item.rule.criterion });
|
||||
aiLog(`[AiClassifier] Loaded ${gCache.size} cache entries`, {debug: true});
|
||||
} else {
|
||||
aiLog(`[AiClassifier] Cache is empty`, {debug: true});
|
||||
}
|
||||
} catch (e) {
|
||||
aiLog(`Failed to load cache`, {level: 'error'}, e);
|
||||
}
|
||||
const questions = [...pending.values()];
|
||||
const epoch = generation;
|
||||
const raw = await sendRequest(systemone.buildBatchRequest(text, questions, snapshot), snapshot);
|
||||
// Parse the complete batch before accepting any of its evidence.
|
||||
const parsed = systemone.parseBatchResponse(raw, questions);
|
||||
const entries = questions.map((question, i) => [question.cacheKey, {
|
||||
...parsed[i], backend: snapshot.apiType, version: 2,
|
||||
model: typeof raw.model === 'string' ? raw.model : snapshot.model,
|
||||
messageHash: question.cacheKey.split(':')[1]
|
||||
}]);
|
||||
for (const [key, entry] of entries) evidence.set(key, entry);
|
||||
if (epoch === generation) {
|
||||
for (const [key, entry] of entries) cache.set(key, entry);
|
||||
await saveCache();
|
||||
gCacheLoaded = true;
|
||||
}
|
||||
|
||||
function loadCacheSync() {
|
||||
if (!gCacheLoaded) {
|
||||
if (!Services?.tm?.spinEventLoopUntil) {
|
||||
throw new Error("loadCacheSync requires Services");
|
||||
}
|
||||
yield { rule: candidate.rule, result: withPolicy(evidence.get(candidate.cacheKey),
|
||||
candidate.rule.decisionThreshold ?? snapshot.decisionThreshold) };
|
||||
let done = false;
|
||||
loadCache().finally(() => { done = true; });
|
||||
Services.tm.spinEventLoopUntil(() => done);
|
||||
}
|
||||
}
|
||||
|
||||
export async function classifyText(text, criterion, cacheKey = null) { return (await classify(text, criterion, { cacheKey })).matched; }
|
||||
export async function removeCacheEntries(keys = []) {
|
||||
await init(); generation++;
|
||||
for (const key of Array.isArray(keys) ? keys : [keys]) cache.delete(key);
|
||||
await saveCache();
|
||||
async function saveCache(updatedKey, updatedValue) {
|
||||
if (typeof updatedKey !== "undefined") {
|
||||
aiLog(`[AiClassifier] ⮡ Persisting entry '${updatedKey}' → ${updatedValue}`, {debug: true});
|
||||
}
|
||||
export async function clearCacheForMessage(id) {
|
||||
await init();
|
||||
const identifier = await messageHash(id);
|
||||
await removeCacheEntries([...cache].filter(([, entry]) => entry.messageHash === identifier).map(([key]) => key));
|
||||
try {
|
||||
await storage.local.set({ aiCache: Object.fromEntries(gCache) });
|
||||
} catch (e) {
|
||||
aiLog(`Failed to save cache`, {level: 'error'}, e);
|
||||
}
|
||||
export async function clearCache() { await init(); generation++; cache.clear(); await saveCache(); }
|
||||
export async function getCacheSize() { await init(); return cache.size; }
|
||||
}
|
||||
|
||||
async function loadTemplate(name) {
|
||||
try {
|
||||
const url = typeof browser !== "undefined" && browser.runtime?.getURL
|
||||
? browser.runtime.getURL(`prompt_templates/${name}.txt`)
|
||||
: `resource://aifilter/prompt_templates/${name}.txt`;
|
||||
const res = await fetch(url);
|
||||
if (res.ok) {
|
||||
return await res.text();
|
||||
}
|
||||
} catch (e) {
|
||||
aiLog(`Failed to load template '${name}':`, {level: 'error'}, e);
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function loadTemplateSync(name) {
|
||||
if (!Services?.tm?.spinEventLoopUntil) {
|
||||
throw new Error("loadTemplateSync requires Services");
|
||||
}
|
||||
let text = "";
|
||||
let done = false;
|
||||
loadTemplate(name).then(t => { text = t; }).catch(() => {}).finally(() => { done = true; });
|
||||
Services.tm.spinEventLoopUntil(() => done);
|
||||
return text;
|
||||
}
|
||||
|
||||
async function setConfig(config = {}) {
|
||||
if (config.endpoint) {
|
||||
gEndpoint = config.endpoint;
|
||||
}
|
||||
if (config.templateName) {
|
||||
gTemplateName = config.templateName;
|
||||
}
|
||||
if (typeof config.customTemplate === "string") {
|
||||
gCustomTemplate = config.customTemplate;
|
||||
}
|
||||
if (typeof config.customSystemPrompt === "string") {
|
||||
gCustomSystemPrompt = config.customSystemPrompt;
|
||||
}
|
||||
if (config.aiParams && typeof config.aiParams === "object") {
|
||||
for (let [k, v] of Object.entries(config.aiParams)) {
|
||||
if (k in gAiParams && typeof v !== "undefined") {
|
||||
gAiParams[k] = v;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (typeof config.debugLogging === "boolean") {
|
||||
setDebug(config.debugLogging);
|
||||
}
|
||||
if (gTemplateName === "custom") {
|
||||
gTemplateText = gCustomTemplate;
|
||||
} else if (Services?.tm?.spinEventLoopUntil) {
|
||||
gTemplateText = loadTemplateSync(gTemplateName);
|
||||
} else {
|
||||
gTemplateText = await loadTemplate(gTemplateName);
|
||||
}
|
||||
aiLog(`[AiClassifier] Endpoint set to ${gEndpoint}`, {debug: true});
|
||||
aiLog(`[AiClassifier] Template set to ${gTemplateName}`, {debug: true});
|
||||
}
|
||||
|
||||
function buildSystemPrompt() {
|
||||
return SYSTEM_PREFIX + (gCustomSystemPrompt || DEFAULT_CUSTOM_SYSTEM_PROMPT) + SYSTEM_SUFFIX;
|
||||
}
|
||||
|
||||
function buildPrompt(body, criterion) {
|
||||
aiLog(`[AiClassifier] Building prompt with criterion: "${criterion}"`, {debug: true});
|
||||
const data = {
|
||||
system: buildSystemPrompt(),
|
||||
email: body,
|
||||
query: criterion,
|
||||
};
|
||||
let template = gTemplateText || loadTemplateSync(gTemplateName);
|
||||
return template.replace(/{{\s*(\w+)\s*}}/g, (m, key) => data[key] || "");
|
||||
}
|
||||
|
||||
function getCachedResult(cacheKey) {
|
||||
if (!gCacheLoaded) {
|
||||
if (Services?.tm?.spinEventLoopUntil) {
|
||||
loadCacheSync();
|
||||
} else {
|
||||
// In non-privileged contexts we can't block, so bail out early.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
if (cacheKey && gCache.has(cacheKey)) {
|
||||
aiLog(`[AiClassifier] Cache hit for key: ${cacheKey}`, {debug: true});
|
||||
return gCache.get(cacheKey);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function buildPayload(text, criterion) {
|
||||
let payloadObj = Object.assign({
|
||||
prompt: buildPrompt(text, criterion)
|
||||
}, gAiParams);
|
||||
return JSON.stringify(payloadObj);
|
||||
}
|
||||
|
||||
function parseMatch(result) {
|
||||
const rawText = result.choices?.[0]?.text || "";
|
||||
const thinkText = rawText.match(/<think>[\s\S]*?<\/think>/gi)?.join('') || '';
|
||||
aiLog('[AiClassifier] ⮡ Reasoning:', {debug: true}, thinkText);
|
||||
const cleanedText = rawText.replace(/<think>[\s\S]*?<\/think>/gi, "").trim();
|
||||
aiLog('[AiClassifier] ⮡ Cleaned Response Text:', {debug: true}, cleanedText);
|
||||
const obj = JSON.parse(cleanedText);
|
||||
return obj.matched === true || obj.match === true;
|
||||
}
|
||||
|
||||
function cacheResult(cacheKey, matched) {
|
||||
if (cacheKey) {
|
||||
aiLog(`[AiClassifier] Caching entry '${cacheKey}' → ${matched}`, {debug: true});
|
||||
gCache.set(cacheKey, matched);
|
||||
saveCache(cacheKey, matched);
|
||||
}
|
||||
}
|
||||
|
||||
function classifyTextSync(text, criterion, cacheKey = null) {
|
||||
if (!Services?.tm?.spinEventLoopUntil) {
|
||||
throw new Error("classifyTextSync requires Services");
|
||||
}
|
||||
const cached = getCachedResult(cacheKey);
|
||||
if (cached !== null) {
|
||||
return cached;
|
||||
}
|
||||
|
||||
const payload = buildPayload(text, criterion);
|
||||
|
||||
aiLog(`[AiClassifier] Sending classification request to ${gEndpoint}`, {debug: true});
|
||||
|
||||
let result;
|
||||
let done = false;
|
||||
(async () => {
|
||||
try {
|
||||
const response = await fetch(gEndpoint, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: payload,
|
||||
});
|
||||
if (response.ok) {
|
||||
const json = await response.json();
|
||||
aiLog(`[AiClassifier] Received response:`, {debug: true}, json);
|
||||
result = parseMatch(json);
|
||||
cacheResult(cacheKey, result);
|
||||
} else {
|
||||
aiLog(`HTTP status ${response.status}`, {level: 'warn'});
|
||||
result = false;
|
||||
}
|
||||
} catch (e) {
|
||||
aiLog(`HTTP request failed`, {level: 'error'}, e);
|
||||
result = false;
|
||||
} finally {
|
||||
done = true;
|
||||
}
|
||||
})();
|
||||
Services.tm.spinEventLoopUntil(() => done);
|
||||
return result;
|
||||
}
|
||||
|
||||
async function classifyText(text, criterion, cacheKey = null) {
|
||||
if (!gCacheLoaded) {
|
||||
await loadCache();
|
||||
}
|
||||
const cached = getCachedResult(cacheKey);
|
||||
if (cached !== null) {
|
||||
return cached;
|
||||
}
|
||||
|
||||
const payload = buildPayload(text, criterion);
|
||||
|
||||
aiLog(`[AiClassifier] Sending classification request to ${gEndpoint}`, {debug: true});
|
||||
|
||||
try {
|
||||
const response = await fetch(gEndpoint, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: payload,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
aiLog(`HTTP status ${response.status}`, {level: 'warn'});
|
||||
return false;
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
aiLog(`[AiClassifier] Received response:`, {debug: true}, result);
|
||||
const matched = parseMatch(result);
|
||||
cacheResult(cacheKey, matched);
|
||||
return matched;
|
||||
} catch (e) {
|
||||
aiLog(`HTTP request failed`, {level: 'error'}, e);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export { classifyText, classifyTextSync, setConfig };
|
||||
|
|
|
|||
99
modules/ExpressionSearchFilter.jsm
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
"use strict";
|
||||
var { ExtensionParent } = ChromeUtils.importESModule("resource://gre/modules/ExtensionParent.sys.mjs");
|
||||
var { MailServices } = ChromeUtils.importESModule("resource:///modules/MailServices.sys.mjs");
|
||||
var { aiLog } = ChromeUtils.import("resource://aifilter/modules/logger.jsm");
|
||||
var AiClassifier = ChromeUtils.importESModule("resource://aifilter/modules/AiClassifier.js");
|
||||
var { getPlainText } = ChromeUtils.import("resource://aifilter/modules/messageUtils.jsm");
|
||||
|
||||
function sha256Hex(str) {
|
||||
const hasher = Cc["@mozilla.org/security/hash;1"].createInstance(Ci.nsICryptoHash);
|
||||
hasher.init(Ci.nsICryptoHash.SHA256);
|
||||
const data = new TextEncoder().encode(str);
|
||||
hasher.update(data, data.length);
|
||||
const binary = hasher.finish(false);
|
||||
return Array.from(binary, c => ("0" + c.charCodeAt(0).toString(16)).slice(-2)).join("");
|
||||
}
|
||||
|
||||
var EXPORTED_SYMBOLS = ["AIFilter", "ClassificationTerm"];
|
||||
|
||||
class CustomerTermBase {
|
||||
constructor(nameId, operators) {
|
||||
// Lookup our extension instance using the ID from manifest.json
|
||||
// so locale strings are resolved correctly.
|
||||
this.extension = ExtensionParent.GlobalManager.getExtension("ai-filter@jordanwages");
|
||||
this.id = "aifilter#" + nameId;
|
||||
this.name = this.extension.localeData.localizeMessage(nameId);
|
||||
this.operators = operators;
|
||||
|
||||
aiLog(`[ExpressionSearchFilter] Initialized term base "${this.id}"`, {debug: true});
|
||||
}
|
||||
|
||||
|
||||
getEnabled() {
|
||||
aiLog(`[ExpressionSearchFilter] getEnabled() called on "${this.id}"`, {debug: true});
|
||||
return true;
|
||||
}
|
||||
|
||||
getAvailable() {
|
||||
aiLog(`[ExpressionSearchFilter] getAvailable() called on "${this.id}"`, {debug: true});
|
||||
return true;
|
||||
}
|
||||
|
||||
getAvailableOperators() {
|
||||
aiLog(`[ExpressionSearchFilter] getAvailableOperators() called on "${this.id}"`, {debug: true});
|
||||
return this.operators;
|
||||
}
|
||||
|
||||
getAvailableValues() {
|
||||
aiLog(`[ExpressionSearchFilter] getAvailableValues() called on "${this.id}"`, {debug: true});
|
||||
return null;
|
||||
}
|
||||
|
||||
get attrib() {
|
||||
aiLog(`[ExpressionSearchFilter] attrib getter called for "${this.id}"`, {debug: true});
|
||||
|
||||
//return Ci.nsMsgSearchAttrib.Custom;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class ClassificationTerm extends CustomerTermBase {
|
||||
constructor() {
|
||||
super("classification", [Ci.nsMsgSearchOp.Matches, Ci.nsMsgSearchOp.DoesntMatch]);
|
||||
aiLog(`[ExpressionSearchFilter] ClassificationTerm constructed`, {debug: true});
|
||||
}
|
||||
|
||||
needsBody() { return true; }
|
||||
|
||||
match(msgHdr, value, op) {
|
||||
const opName = op === Ci.nsMsgSearchOp.Matches ? "matches" :
|
||||
op === Ci.nsMsgSearchOp.DoesntMatch ? "doesn't match" : `unknown (${op})`;
|
||||
aiLog(`[ExpressionSearchFilter] Matching message ${msgHdr.messageId} using op "${opName}" and value "${value}"`, {debug: true});
|
||||
|
||||
let key = [msgHdr.messageId, op, value].map(sha256Hex).join("|");
|
||||
let body = getPlainText(msgHdr);
|
||||
|
||||
let matched = AiClassifier.classifyTextSync(body, value, key);
|
||||
|
||||
if (op === Ci.nsMsgSearchOp.DoesntMatch) {
|
||||
matched = !matched;
|
||||
aiLog(`[ExpressionSearchFilter] Operator is "doesn't match" → inverting to ${matched}`, {debug: true});
|
||||
}
|
||||
|
||||
aiLog(`[ExpressionSearchFilter] Final match result: ${matched}`, {debug: true});
|
||||
return matched;
|
||||
}
|
||||
}
|
||||
|
||||
(function register() {
|
||||
aiLog(`[ExpressionSearchFilter] Registering custom filter term...`, {debug: true});
|
||||
let term = new ClassificationTerm();
|
||||
if (!MailServices.filters.getCustomTerm(term.id)) {
|
||||
MailServices.filters.addCustomTerm(term);
|
||||
aiLog(`[ExpressionSearchFilter] Registered term: ${term.id}`, {debug: true});
|
||||
} else {
|
||||
aiLog(`[ExpressionSearchFilter] Term already registered: ${term.id}`, {debug: true});
|
||||
}
|
||||
})();
|
||||
|
||||
var AIFilter = { setConfig: AiClassifier.setConfig };
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
const SYSTEM_PREFIX = `You are an email-classification assistant.
|
||||
Read the email below and the classification criterion provided by the user.
|
||||
`;
|
||||
|
||||
const DEFAULT_CUSTOM_SYSTEM_PROMPT = "Determine whether the email satisfies the user's criterion.";
|
||||
|
||||
const SYSTEM_SUFFIX = `
|
||||
Return ONLY a JSON object on a single line of the form:
|
||||
{"match": true, "reason": "<short explanation>"} - if the email satisfies the criterion
|
||||
{"match": false, "reason": "<short explanation>"} - otherwise
|
||||
|
||||
Do not add any other keys, text, or formatting.`;
|
||||
|
||||
function extractLastJsonObject(text) {
|
||||
let last = null;
|
||||
let start = -1;
|
||||
let depth = 0;
|
||||
let inString = false;
|
||||
let escape = false;
|
||||
|
||||
for (let i = 0; i < text.length; i += 1) {
|
||||
const ch = text[i];
|
||||
if (inString) {
|
||||
if (escape) {
|
||||
escape = false;
|
||||
continue;
|
||||
}
|
||||
if (ch === "\\") {
|
||||
escape = true;
|
||||
continue;
|
||||
}
|
||||
if (ch === "\"") {
|
||||
inString = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (ch === "\"") {
|
||||
inString = true;
|
||||
continue;
|
||||
}
|
||||
if (ch === "{") {
|
||||
if (depth === 0) {
|
||||
start = i;
|
||||
}
|
||||
depth += 1;
|
||||
continue;
|
||||
}
|
||||
if (ch === "}" && depth > 0) {
|
||||
depth -= 1;
|
||||
if (depth === 0 && start !== -1) {
|
||||
last = text.slice(start, i + 1);
|
||||
start = -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return last;
|
||||
}
|
||||
|
||||
export const path = '/v1/chat/completions';
|
||||
export function buildRequest(text, criterion, config) {
|
||||
const payload = {
|
||||
messages: [
|
||||
{ role: 'system', content: SYSTEM_PREFIX + (config.customSystemPrompt || DEFAULT_CUSTOM_SYSTEM_PROMPT) + SYSTEM_SUFFIX },
|
||||
{ role: 'user', content: `**Email Contents**\n\`\`\`\n${text}\n\`\`\`\nClassification Criterion: ${criterion}` }
|
||||
], ...config.aiParams
|
||||
};
|
||||
if (config.model) payload.model = config.model;
|
||||
return payload;
|
||||
}
|
||||
export function parseResponse(result) {
|
||||
const content = result?.choices?.[0]?.message?.content;
|
||||
if (typeof content !== 'string') throw new Error('Chat response missing text content.');
|
||||
const candidate = extractLastJsonObject(content);
|
||||
if (!candidate) throw new Error('No JSON object found in AI response.');
|
||||
const obj = JSON.parse(candidate);
|
||||
const matched = Object.hasOwn(obj, 'match') ? obj.match : obj.matched;
|
||||
if (typeof matched !== 'boolean') throw new Error('AI response missing valid match boolean.');
|
||||
const reason = obj.reason ?? obj.reasoning ?? obj.explaination;
|
||||
return { matched, reason: typeof reason === 'string' ? reason : null, probability: null };
|
||||
}
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
export const path = '/v1/systemone';
|
||||
export function buildRequest(text, criterion, config) {
|
||||
return buildBatchRequest(text, [{ id: 'match', criterion }], config);
|
||||
}
|
||||
export function buildBatchRequest(text, questions, config) {
|
||||
const payload = { state: text, questions: Object.fromEntries(questions.map(({ id, criterion }) =>
|
||||
[id, { type: 'noul', instructions: `Does this email satisfy the following criterion: ${criterion}` }])) };
|
||||
if (config.model) payload.model = config.model;
|
||||
return payload;
|
||||
}
|
||||
export function parseResponse(result) { return parseAnswer(result, 'match'); }
|
||||
export function parseBatchResponse(result, questions) {
|
||||
return questions.map(({ id }) => parseAnswer(result, id));
|
||||
}
|
||||
function parseAnswer(result, id) {
|
||||
const answer = result?.answers?.[id];
|
||||
if (answer?.type !== 'noul' || typeof answer.noul !== 'number' ||
|
||||
!Number.isFinite(answer.noul) || answer.noul < 0 || answer.noul > 1) {
|
||||
throw new Error(`SystemOne response must contain answers.${id} with type noul and a probability from 0 to 1.`);
|
||||
}
|
||||
return { matched: null, reason: null, probability: answer.noul };
|
||||
}
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
import { DEFAULT_AI_PARAMS } from './defaultParams.js';
|
||||
export const CONFIG_DEFAULTS = {
|
||||
apiType: 'chat-completions', endpoint: 'http://127.0.0.1:5000', model: '',
|
||||
apiKey: '', openaiOrganization: '', openaiProject: '', customSystemPrompt: '',
|
||||
systemOneProcessAll: true, systemOneBatchSize: 8,
|
||||
decisionThreshold: 0.5, inputCharacterLimit: 3686, aiParams: DEFAULT_AI_PARAMS,
|
||||
htmlToMarkdown: false, stripUrlParams: false, altTextImages: false,
|
||||
collapseWhitespace: false, tokenReduction: false, debugLogging: false
|
||||
};
|
||||
export function validateThreshold(value) {
|
||||
if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 || value > 1) {
|
||||
throw new Error('Match threshold must be between 0% and 100%.');
|
||||
}
|
||||
return value;
|
||||
}
|
||||
export function validateSettings(data) {
|
||||
if (data.systemOneProcessAll !== undefined && typeof data.systemOneProcessAll !== 'boolean') {
|
||||
throw new Error('Process all must be a boolean.');
|
||||
}
|
||||
if (data.systemOneBatchSize !== undefined && (!Number.isSafeInteger(data.systemOneBatchSize) || data.systemOneBatchSize < 1)) {
|
||||
throw new Error('Rules per request must be a positive whole number.');
|
||||
}
|
||||
if (data.apiType !== undefined && !['chat-completions', 'systemone'].includes(data.apiType)) throw new Error('Unknown API type.');
|
||||
if (data.decisionThreshold !== undefined) validateThreshold(data.decisionThreshold);
|
||||
if (data.inputCharacterLimit !== undefined && (!Number.isSafeInteger(data.inputCharacterLimit) || data.inputCharacterLimit < 1)) {
|
||||
throw new Error('Input character limit must be a positive integer.');
|
||||
}
|
||||
if (data.aiRules !== undefined) {
|
||||
if (!Array.isArray(data.aiRules)) throw new Error('Rules must be an array.');
|
||||
for (const rule of data.aiRules) {
|
||||
if (rule.decisionThreshold !== undefined) validateThreshold(rule.decisionThreshold);
|
||||
}
|
||||
}
|
||||
}
|
||||
export function migratedInputLimit(data) {
|
||||
if (data.inputCharacterLimit !== undefined) return data.inputCharacterLimit;
|
||||
const value = Math.floor(Number(data.aiParams?.max_tokens ?? 4096) * 0.9);
|
||||
return Number.isSafeInteger(value) && value > 0 ? value : 3686;
|
||||
}
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
"use strict";
|
||||
|
||||
export const DEFAULT_AI_PARAMS = {
|
||||
max_tokens: 4096,
|
||||
temperature: 0.6,
|
||||
top_p: 0.95,
|
||||
seed: -1,
|
||||
repetition_penalty: 1.0,
|
||||
top_k: 20,
|
||||
min_p: 0,
|
||||
presence_penalty: 0,
|
||||
frequency_penalty: 0,
|
||||
typical_p: 1,
|
||||
tfs: 1,
|
||||
};
|
||||
|
||||
26
modules/logger.jsm
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
var EXPORTED_SYMBOLS = ['aiLog', 'setDebug'];
|
||||
let debugEnabled = false;
|
||||
|
||||
function setDebug(value) {
|
||||
debugEnabled = !!value;
|
||||
}
|
||||
|
||||
function getCaller() {
|
||||
try {
|
||||
let stack = new Error().stack.split('\n');
|
||||
if (stack.length >= 3) {
|
||||
return stack[2].trim().replace(/^@?\s*\(?/,'').replace(/^at\s+/, '');
|
||||
}
|
||||
} catch (e) {}
|
||||
return '';
|
||||
}
|
||||
|
||||
function aiLog(message, opts = {}, ...args) {
|
||||
const { level = 'log', debug = false } = opts;
|
||||
if (debug && !debugEnabled) {
|
||||
return;
|
||||
}
|
||||
const caller = getCaller();
|
||||
const prefix = caller ? `[ai-filter][${caller}]` : '[ai-filter]';
|
||||
console[level](`%c${prefix}`, 'color:#1c92d2;font-weight:bold', message, ...args);
|
||||
}
|
||||
89
modules/messageUtils.jsm
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
"use strict";
|
||||
var { NetUtil } = ChromeUtils.importESModule("resource://gre/modules/NetUtil.sys.mjs");
|
||||
var { MimeParser } = ChromeUtils.importESModule("resource:///modules/mimeParser.sys.mjs");
|
||||
var { aiLog } = ChromeUtils.import("resource://aifilter/modules/logger.jsm");
|
||||
|
||||
var EXPORTED_SYMBOLS = ["getPlainText"];
|
||||
|
||||
function getPlainText(msgHdr) {
|
||||
aiLog(`[ExpressionSearchFilter] Extracting plain text for message ID ${msgHdr.messageId}`, {debug: true});
|
||||
let folder = msgHdr.folder;
|
||||
if (!folder.getMsgInputStream) return "";
|
||||
let reusable = {};
|
||||
let stream = folder.getMsgInputStream(msgHdr, reusable);
|
||||
let data = NetUtil.readInputStreamToString(stream, msgHdr.messageSize);
|
||||
if (!reusable.value) stream.close();
|
||||
|
||||
let parser = Cc["@mozilla.org/parserutils;1"].getService(Ci.nsIParserUtils);
|
||||
|
||||
try {
|
||||
let root = MimeParser.parseSync(data, {strformat: "unicode"});
|
||||
let parts = [];
|
||||
|
||||
function pushPlaceholder(type, info, bytes) {
|
||||
bytes = bytes || 0;
|
||||
let prettyType = type.split("/")[1] || type;
|
||||
parts.push(`[${info}: ${prettyType}, ${bytes} bytes]`);
|
||||
}
|
||||
|
||||
function byteSizeFromBase64(str) {
|
||||
let clean = str.replace(/[^A-Za-z0-9+/=]/g, "");
|
||||
return Math.floor(clean.length * 3 / 4);
|
||||
}
|
||||
|
||||
function replaceInlineBase64(text) {
|
||||
return text.replace(/[A-Za-z0-9+/]{100,}={0,2}/g,
|
||||
m => `[base64: ${byteSizeFromBase64(m)} bytes]`);
|
||||
}
|
||||
|
||||
function walk(node) {
|
||||
if (node.parts && node.parts.length) {
|
||||
for (let child of node.parts) {
|
||||
walk(child);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
let ct = (node.contentType || "text/plain").toLowerCase();
|
||||
let cd = (node.headers?.["content-disposition"]?.[0] || "").toLowerCase();
|
||||
let enc = (node.headers?.["content-transfer-encoding"]?.[0] || "").toLowerCase();
|
||||
let bodyText = String(node.body || "");
|
||||
|
||||
if (cd.includes("attachment")) {
|
||||
pushPlaceholder(ct, "binary attachment", byteSizeFromBase64(bodyText));
|
||||
} else if (ct.startsWith("text/plain")) {
|
||||
if (enc === "base64") {
|
||||
parts.push(`[base64: ${byteSizeFromBase64(bodyText)} bytes]`);
|
||||
} else {
|
||||
parts.push(replaceInlineBase64(bodyText));
|
||||
}
|
||||
} else if (ct.startsWith("text/html")) {
|
||||
if (enc === "base64") {
|
||||
parts.push(`[base64: ${byteSizeFromBase64(bodyText)} bytes]`);
|
||||
} else {
|
||||
let txt = parser.convertToPlainText(bodyText,
|
||||
Ci.nsIDocumentEncoder.OutputLFLineBreak |
|
||||
Ci.nsIDocumentEncoder.OutputNoScriptContent |
|
||||
Ci.nsIDocumentEncoder.OutputNoFramesContent |
|
||||
Ci.nsIDocumentEncoder.OutputBodyOnly, 0);
|
||||
parts.push(replaceInlineBase64(txt));
|
||||
}
|
||||
} else {
|
||||
// Other single part types treated as attachments
|
||||
pushPlaceholder(ct, "binary attachment", byteSizeFromBase64(bodyText));
|
||||
}
|
||||
}
|
||||
|
||||
walk(root);
|
||||
return parts.join("\n");
|
||||
} catch (e) {
|
||||
// Fallback: convert entire raw message to text
|
||||
aiLog(`Failed to parse MIME, falling back to raw conversion`, {level: 'warn'}, e);
|
||||
return parser.convertToPlainText(data,
|
||||
Ci.nsIDocumentEncoder.OutputLFLineBreak |
|
||||
Ci.nsIDocumentEncoder.OutputNoScriptContent |
|
||||
Ci.nsIDocumentEncoder.OutputNoFramesContent |
|
||||
Ci.nsIDocumentEncoder.OutputBodyOnly, 0);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
"use strict";
|
||||
|
||||
export async function detectSystemTheme() {
|
||||
try {
|
||||
const t = await browser.theme.getCurrent();
|
||||
const scheme = t?.properties?.color_scheme;
|
||||
if (scheme === 'dark' || scheme === 'light') {
|
||||
return scheme;
|
||||
}
|
||||
const color = t?.colors?.frame || t?.colors?.toolbar;
|
||||
if (color && /^#/.test(color)) {
|
||||
const r = parseInt(color.slice(1, 3), 16);
|
||||
const g = parseInt(color.slice(3, 5), 16);
|
||||
const b = parseInt(color.slice(5, 7), 16);
|
||||
const lum = (0.299 * r + 0.587 * g + 0.114 * b) / 255;
|
||||
return lum < 0.5 ? 'dark' : 'light';
|
||||
}
|
||||
} catch {}
|
||||
return 'light';
|
||||
}
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
"use strict";
|
||||
import { validateSettings, migratedInputLimit } from '../modules/classificationConfig.js';
|
||||
import { validCacheEntry } from '../modules/AiClassifier.js';
|
||||
const storage = (globalThis.messenger ?? browser).storage;
|
||||
const KEY_GROUPS = {
|
||||
settings: [
|
||||
'systemOneProcessAll', 'systemOneBatchSize',
|
||||
'apiType', 'decisionThreshold', 'inputCharacterLimit', 'tokenReduction',
|
||||
'endpoint',
|
||||
'model',
|
||||
'apiKey',
|
||||
'openaiOrganization',
|
||||
'openaiProject',
|
||||
'customSystemPrompt',
|
||||
'aiParams',
|
||||
'debugLogging',
|
||||
'htmlToMarkdown',
|
||||
'stripUrlParams',
|
||||
'altTextImages',
|
||||
'collapseWhitespace'
|
||||
],
|
||||
rules: ['aiRules'],
|
||||
cache: ['aiCache']
|
||||
};
|
||||
|
||||
function collectKeys(categories = Object.keys(KEY_GROUPS)) {
|
||||
return categories.flatMap(cat => KEY_GROUPS[cat] || []);
|
||||
}
|
||||
|
||||
export async function exportData(categories) {
|
||||
const data = await storage.local.get(collectKeys(categories));
|
||||
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = 'sortana-export.json';
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
a.remove();
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
|
||||
export async function importData(file, categories = Object.keys(KEY_GROUPS)) {
|
||||
const text = await file.text();
|
||||
const parsed = JSON.parse(text);
|
||||
const data = {};
|
||||
for (const key of collectKeys(categories)) {
|
||||
if (key in parsed) data[key] = parsed[key];
|
||||
}
|
||||
if (categories.includes('settings') && data.inputCharacterLimit === undefined) data.inputCharacterLimit = migratedInputLimit(data);
|
||||
validateSettings(data);
|
||||
if (data.aiCache !== undefined) data.aiCache = Object.fromEntries(Object.entries(data.aiCache || {}).filter(([key, entry]) => validCacheEntry(key, entry)));
|
||||
await storage.local.set(data);
|
||||
}
|
||||
|
|
@ -24,95 +24,59 @@
|
|||
--bulma-warning-l: 70%;
|
||||
--bulma-danger-h: 0deg;
|
||||
--bulma-danger-s: 49%;
|
||||
--bulma-scheme-h: 200;
|
||||
--bulma-scheme-s: 25%;
|
||||
--bulma-scheme-h: 326;
|
||||
--bulma-scheme-s: 40%;
|
||||
}
|
||||
|
||||
.tag {
|
||||
--bulma-tag-h: 318;
|
||||
}
|
||||
#diff-display {
|
||||
white-space: pre-wrap;
|
||||
font-family: monospace;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<section class="section">
|
||||
<div class="container" id="options-container">
|
||||
<figure class="has-text-centered mb-4">
|
||||
<img data-icon="full-logo" src="../resources/img/full-logo.png" alt="AI Filter Logo" style="max-height:40px;">
|
||||
<img src="../resources/img/full-logo.png" alt="AI Filter Logo" style="max-height:40px;">
|
||||
</figure>
|
||||
|
||||
<div class="level mb-4">
|
||||
<div class="level-left">
|
||||
<div class="tabs" id="main-tabs">
|
||||
<ul>
|
||||
<li class="is-active" data-tab="settings"><a><span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span><span>Settings</span></a></li>
|
||||
<li data-tab="rules"><a><span class="icon is-small"><img data-icon="clipboarddata" data-size="16" src="../resources/img/clipboarddata-light-16.png" alt=""></span><span>Rules</span></a></li>
|
||||
<li data-tab="maintenance"><a><span class="icon is-small"><img data-icon="gear" data-size="16" src="../resources/img/gear-light-16.png" alt=""></span><span>Maintenance</span></a></li>
|
||||
<li id="errors-tab-button" class="is-hidden" data-tab="errors"><a><span class="icon is-small"><img data-icon="x" data-size="16" src="../resources/img/x-light-16.png" alt=""></span><span>Errors</span></a></li>
|
||||
<li id="debug-tab-button" class="is-hidden" data-tab="debug"><a><span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span><span>Debug</span></a></li>
|
||||
<li class="is-active" data-tab="settings"><a>Settings</a></li>
|
||||
<li data-tab="rules"><a>Rules</a></li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
<div class="level-right">
|
||||
<button class="button is-primary" id="save" disabled>
|
||||
<span class="icon is-small"><img data-icon="flag" data-size="16" src="../resources/img/flag-light-16.png" alt=""></span>
|
||||
<span>Save</span>
|
||||
</button>
|
||||
<button class="button is-primary" id="save" disabled>Save</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="help is-danger mb-3" id="settings-error" role="alert"></p>
|
||||
|
||||
<div id="settings-tab" class="tab-content">
|
||||
<h2 class="title is-4">
|
||||
<span class="icon is-small"><img data-icon="settings" data-size="16" src="../resources/img/settings-light-16.png" alt=""></span>
|
||||
<span>Settings</span>
|
||||
</h2>
|
||||
<div class="field">
|
||||
<label class="label" for="api-type">API type</label>
|
||||
<div class="select"><select id="api-type">
|
||||
<option value="chat-completions">Chat Completions</option>
|
||||
<option value="systemone">SystemOne</option>
|
||||
</select></div>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="checkbox"><input type="checkbox" id="systemone-process-all"> Process all</label>
|
||||
<p class="help">Group all eligible, uncached rules for each email into one request. Actions still follow rule order and stop-after-match.</p>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="label" for="systemone-batch-size">Rules per request</label>
|
||||
<input class="input" id="systemone-batch-size" type="number" min="1" step="1" required>
|
||||
<p class="help">Turn off Process all to limit each request. Use 1 for one rule at a time.</p>
|
||||
</div>
|
||||
<div class="field systemone-only">
|
||||
<label class="label" for="decision-threshold">Default match probability threshold (%)</label>
|
||||
<input class="input" id="decision-threshold" type="number" min="0" max="100" step="any">
|
||||
<p class="help">Match when probability is at least this value. Rules may override it. Evaluate thresholds on your own email examples.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="label" for="endpoint">Endpoint</label>
|
||||
<div class="control">
|
||||
<input class="input" type="text" id="endpoint" placeholder="https://api.example.com">
|
||||
</div>
|
||||
<p class="help" id="endpoint-preview"></p>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label class="label" for="model-select">Model</label>
|
||||
<div class="field has-addons">
|
||||
<div class="control is-expanded">
|
||||
<div class="select is-fullwidth">
|
||||
<select id="model-select"></select>
|
||||
</div>
|
||||
</div>
|
||||
<label class="label" for="template">Prompt template</label>
|
||||
<div class="control">
|
||||
<button class="button" id="refresh-models" type="button">Refresh</button>
|
||||
<div class="select is-fullwidth">
|
||||
<select id="template"></select>
|
||||
</div>
|
||||
</div>
|
||||
<p class="help" id="model-help"></p>
|
||||
</div>
|
||||
|
||||
<div id="custom-template-container" class="field is-hidden">
|
||||
<label class="label">Custom template</label>
|
||||
<div class="control">
|
||||
<textarea class="textarea" id="custom-template" rows="6" placeholder="Enter your custom template here..."></textarea>
|
||||
</div>
|
||||
<p class="help">Placeholders: {{system}}, {{email}}, {{query}}</p>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
|
|
@ -122,99 +86,19 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label class="label" for="theme-select">Theme</label>
|
||||
<div class="control">
|
||||
<div class="select">
|
||||
<select id="theme-select">
|
||||
<option value="auto">Match Thunderbird</option>
|
||||
<option value="light">Light</option>
|
||||
<option value="dark">Dark</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="buttons">
|
||||
<button class="button is-danger" id="reset-system">
|
||||
<span class="icon is-small"><img data-icon="reply" data-size="16" src="../resources/img/reply-light-16.png" alt=""></span>
|
||||
<span>Reset to default</span>
|
||||
</button>
|
||||
<button class="button" id="toggle-advanced" type="button">
|
||||
<span class="icon is-small"><img data-icon="gear" data-size="16" src="../resources/img/gear-light-16.png" alt=""></span>
|
||||
<span>Advanced</span>
|
||||
</button>
|
||||
<button class="button is-danger" id="reset-system">Reset to default</button>
|
||||
<button class="button" id="toggle-advanced" type="button">Advanced</button>
|
||||
</div>
|
||||
|
||||
<div id="advanced-options" class="mt-4 is-hidden">
|
||||
<div class="field">
|
||||
<label class="label" for="api-key">API key</label>
|
||||
<div class="field has-addons">
|
||||
<div class="control is-expanded">
|
||||
<input class="input" type="password" id="api-key" placeholder="sk-...">
|
||||
</div>
|
||||
<div class="control">
|
||||
<button class="button" id="toggle-api-key" type="button">Show</button>
|
||||
</div>
|
||||
</div>
|
||||
<p class="help">Leave blank for unauthenticated endpoints.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="label" for="openai-organization">OpenAI Organization</label>
|
||||
<div class="control">
|
||||
<input class="input" type="text" id="openai-organization" placeholder="org-...">
|
||||
</div>
|
||||
<p class="help">Optional header for OpenAI-hosted endpoints.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="label" for="openai-project">OpenAI Project</label>
|
||||
<div class="control">
|
||||
<input class="input" type="text" id="openai-project" placeholder="proj_...">
|
||||
</div>
|
||||
<p class="help">Optional header for OpenAI-hosted endpoints.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="debug-logging"> Enable debug logging
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="html-to-markdown"> Convert HTML body to Markdown
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="strip-url-params"> Remove URL tracking parameters
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="alt-text-images"> Replace images with alt text
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="collapse-whitespace"> Collapse long whitespace
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="token-reduction"> Aggressive token reduction
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="show-debug-tab"> Show debug information
|
||||
</label>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="label" for="input-character-limit">Input character limit</label>
|
||||
<input class="input" id="input-character-limit" type="number" min="1" step="1">
|
||||
<p class="help">Applied with aggressive reduction. Characters are not tokens; leave room for instructions within your model's context limit.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label class="label" for="max_tokens">Maximum output tokens</label>
|
||||
<label class="label" for="max_tokens">Max tokens</label>
|
||||
<div class="control">
|
||||
<input class="input" type="number" id="max_tokens">
|
||||
</div>
|
||||
|
|
@ -283,110 +167,12 @@
|
|||
</div>
|
||||
|
||||
<div id="rules-tab" class="tab-content is-hidden">
|
||||
<h2 class="title is-4">
|
||||
<span class="icon is-small"><img data-icon="clipboarddata" data-size="16" src="../resources/img/clipboarddata-light-16.png" alt=""></span>
|
||||
<span>Classification Rules</span>
|
||||
</h2>
|
||||
<h2 class="title is-4">Classification Rules</h2>
|
||||
<div id="rules-container"></div>
|
||||
<button class="button is-link" id="add-rule" type="button">Add Rule</button>
|
||||
</div>
|
||||
|
||||
<div id="maintenance-tab" class="tab-content is-hidden">
|
||||
<h2 class="title is-4">
|
||||
<span class="icon is-small"><img data-icon="gear" data-size="16" src="../resources/img/gear-light-16.png" alt=""></span>
|
||||
<span>Maintenance</span>
|
||||
</h2>
|
||||
<table class="table is-fullwidth">
|
||||
<tbody>
|
||||
<tr><th>Rule count</th><td id="rule-count"></td></tr>
|
||||
<tr><th>Cache entries</th><td id="cache-count"></td></tr>
|
||||
<tr><th>Queue items</th><td id="queue-count"></td></tr>
|
||||
<tr><th>Current run time</th><td id="current-time">--:--:--</td></tr>
|
||||
<tr><th>Last run time</th><td id="last-time">--:--:--</td></tr>
|
||||
<tr><th>Average run time</th><td id="average-time">--:--:--</td></tr>
|
||||
<tr><th>Total run time</th><td id="total-time">--:--:--</td></tr>
|
||||
<tr><th>Messages per hour</th><td id="per-hour">0</td></tr>
|
||||
<tr><th>Messages per day</th><td id="per-day">0</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="buttons">
|
||||
<button class="button is-danger" id="clear-cache" type="button">
|
||||
<span class="icon is-small"><img data-icon="trash" data-size="16" src="../resources/img/trash-light-16.png" alt=""></span>
|
||||
<span>Clear Cache</span>
|
||||
</button>
|
||||
<button class="button is-warning" id="reset-timing" type="button">
|
||||
<span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span>
|
||||
<span>Reset Timing Stats</span>
|
||||
</button>
|
||||
</div>
|
||||
<div class="field mt-4">
|
||||
<label class="label">Data categories</label>
|
||||
<div class="control">
|
||||
<label class="checkbox mr-3"><input class="transfer-category" type="checkbox" value="settings" checked> Settings</label>
|
||||
<label class="checkbox mr-3"><input class="transfer-category" type="checkbox" value="rules" checked> Rules</label>
|
||||
<label class="checkbox"><input class="transfer-category" type="checkbox" value="cache" checked> Cache</label>
|
||||
</div>
|
||||
</div>
|
||||
<div class="field is-grouped mt-4">
|
||||
<p class="control">
|
||||
<button class="button" id="export-data" type="button">
|
||||
<span class="icon is-small"><img data-icon="download" data-size="16" src="../resources/img/download-light-16.png" alt=""></span>
|
||||
<span>Export Data</span>
|
||||
</button>
|
||||
</p>
|
||||
<p class="control">
|
||||
<button class="button" id="import-data" type="button">
|
||||
<span class="icon is-small"><img data-icon="upload" data-size="16" src="../resources/img/upload-light-16.png" alt=""></span>
|
||||
<span>Import Data</span>
|
||||
</button>
|
||||
<input class="is-hidden" type="file" id="import-file" accept="application/json">
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="errors-tab" class="tab-content is-hidden">
|
||||
<h2 class="title is-4">
|
||||
<span class="icon is-small"><img data-icon="x" data-size="16" src="../resources/img/x-light-16.png" alt=""></span>
|
||||
<span>Session Errors</span>
|
||||
</h2>
|
||||
<div id="errors-empty" class="notification is-success is-light">
|
||||
No errors have been recorded since the last start.
|
||||
</div>
|
||||
<div id="errors-panel" class="is-hidden">
|
||||
<div class="box mb-4">
|
||||
<div class="level">
|
||||
<div class="level-left">
|
||||
<div>
|
||||
<p class="title is-5 mb-1">Error Log</p>
|
||||
<p class="subtitle is-6">Visible only for this session.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="level-right">
|
||||
<span class="tag is-danger is-light" id="errors-count">0</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="errors-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="debug-tab" class="tab-content is-hidden">
|
||||
<h2 class="title is-4">
|
||||
<span class="icon is-small"><img data-icon="average" data-size="16" src="../resources/img/average-light-16.png" alt=""></span>
|
||||
<span>Debug</span>
|
||||
</h2>
|
||||
<pre id="payload-display"></pre>
|
||||
<div id="diff-container" class="mt-4 is-hidden">
|
||||
<div class="is-flex is-align-items-center is-justify-content-space-between">
|
||||
<label class="label mb-0">Prompt diff</label>
|
||||
<span id="prompt-reduction" class="tag is-info is-light is-hidden">Prompt Token Reduction: 0%</span>
|
||||
</div>
|
||||
<div id="diff-display" class="box content is-family-monospace"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<script src="../resources/js/diff_match_patch_uncompressed.js"></script>
|
||||
<script src="options.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
12
prompt_templates/mistral.txt
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[INST] {{system}}
|
||||
|
||||
Email:
|
||||
{{email}}
|
||||
|
||||
Criterion: {{query}}
|
||||
Remember, return ONLY a JSON object on a single line of the form:
|
||||
{"match": true} - if the email satisfies the criterion
|
||||
{"match": false} - otherwise
|
||||
|
||||
Do not add any other keys, text, or formatting.
|
||||
[/INST]
|
||||
14
prompt_templates/openai.txt
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
<|im_start|>system
|
||||
{{system}}<|im_end|>
|
||||
<|im_start|>user
|
||||
**Email Contents**
|
||||
```
|
||||
{{email}}
|
||||
```
|
||||
Classification Criterion: {{query}}
|
||||
Remember, return ONLY a JSON object on a single line of the form:
|
||||
{"match": true} - if the email satisfies the criterion
|
||||
{"match": false} - otherwise
|
||||
|
||||
Do not add any other keys, text, or formatting.<|im_end|>
|
||||
<|im_start|>assistant
|
||||
15
prompt_templates/qwen.txt
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
<|im_start|>system
|
||||
{{system}}
|
||||
<|im_end|>
|
||||
<|im_start|>user
|
||||
Email:
|
||||
{{email}}
|
||||
|
||||
Criterion: {{query}}
|
||||
Remember, return ONLY a JSON object on a single line of the form:
|
||||
{"match": true} - if the email satisfies the criterion
|
||||
{"match": false} - otherwise
|
||||
|
||||
Do not add any other keys, text, or formatting.
|
||||
<|im_end|>
|
||||
<|im_start|>assistant
|
||||
|
Before Width: | Height: | Size: 427 B |
|
Before Width: | Height: | Size: 791 B |
|
Before Width: | Height: | Size: 1.5 KiB |
|
Before Width: | Height: | Size: 416 B |
|
Before Width: | Height: | Size: 794 B |
|
Before Width: | Height: | Size: 1.5 KiB |
|
Before Width: | Height: | Size: 307 B |
|
Before Width: | Height: | Size: 449 B |
|
Before Width: | Height: | Size: 940 B |
|
Before Width: | Height: | Size: 300 B |
|
Before Width: | Height: | Size: 450 B |
|
Before Width: | Height: | Size: 927 B |
|
Before Width: | Height: | Size: 389 B |
|
Before Width: | Height: | Size: 722 B |
|
Before Width: | Height: | Size: 1.5 KiB |
|
Before Width: | Height: | Size: 389 B |
|
Before Width: | Height: | Size: 724 B |
|
Before Width: | Height: | Size: 1.5 KiB |
|
Before Width: | Height: | Size: 413 B |
|
Before Width: | Height: | Size: 828 B |
|
Before Width: | Height: | Size: 1.7 KiB |
|
Before Width: | Height: | Size: 408 B |
|
Before Width: | Height: | Size: 813 B |
|
Before Width: | Height: | Size: 1.7 KiB |
|
Before Width: | Height: | Size: 396 B |
|
Before Width: | Height: | Size: 773 B |
|
Before Width: | Height: | Size: 1.6 KiB |
|
Before Width: | Height: | Size: 394 B |
|
Before Width: | Height: | Size: 773 B |
|
Before Width: | Height: | Size: 1.6 KiB |
|
Before Width: | Height: | Size: 392 B |
|
Before Width: | Height: | Size: 768 B |
|
Before Width: | Height: | Size: 1.6 KiB |
|
Before Width: | Height: | Size: 393 B |
|
Before Width: | Height: | Size: 764 B |
|
Before Width: | Height: | Size: 1.5 KiB |
|
Before Width: | Height: | Size: 320 B |
|
Before Width: | Height: | Size: 537 B |
|
Before Width: | Height: | Size: 969 B |
|
Before Width: | Height: | Size: 314 B |
|
Before Width: | Height: | Size: 543 B |
|
Before Width: | Height: | Size: 989 B |
|
Before Width: | Height: | Size: 340 B |
|
Before Width: | Height: | Size: 556 B |
|
Before Width: | Height: | Size: 1 KiB |
|
Before Width: | Height: | Size: 345 B |
|
Before Width: | Height: | Size: 571 B |
|
Before Width: | Height: | Size: 1,006 B |
|
Before Width: | Height: | Size: 374 B |
|
Before Width: | Height: | Size: 729 B |
|
Before Width: | Height: | Size: 1.4 KiB |
|
Before Width: | Height: | Size: 371 B |
|
Before Width: | Height: | Size: 733 B |
|
Before Width: | Height: | Size: 1.4 KiB |
|
Before Width: | Height: | Size: 293 B |
|
Before Width: | Height: | Size: 469 B |
|
Before Width: | Height: | Size: 766 B |
|
Before Width: | Height: | Size: 300 B |
|
Before Width: | Height: | Size: 475 B |
|
Before Width: | Height: | Size: 800 B |
|
Before Width: | Height: | Size: 484 B |
|
Before Width: | Height: | Size: 1 KiB |
|
Before Width: | Height: | Size: 2.1 KiB |
|
Before Width: | Height: | Size: 462 B |
|
Before Width: | Height: | Size: 993 B |
|
Before Width: | Height: | Size: 2.1 KiB |
|
Before Width: | Height: | Size: 199 B |
|
Before Width: | Height: | Size: 301 B |
|
Before Width: | Height: | Size: 431 B |
|
Before Width: | Height: | Size: 215 B |
|
Before Width: | Height: | Size: 305 B |
|
Before Width: | Height: | Size: 457 B |
|
Before Width: | Height: | Size: 280 B |
|
Before Width: | Height: | Size: 414 B |