Replace toxicity model with multi-label classifier
This commit is contained in:
parent
803df7feee
commit
1568c787f8
24 changed files with 2581129 additions and 119717 deletions
20
DESIGN.md
20
DESIGN.md
|
|
@ -6,12 +6,7 @@ Toxic Content Filter is a browser extension that automatically identifies and hi
|
|||
|
||||
The extension performs all inference locally. Social-media content is never sent to an external classification service.
|
||||
|
||||
The initial classifier produces a simple binary classification:
|
||||
|
||||
* **Toxic**
|
||||
* **Non-toxic**
|
||||
|
||||
The model should also expose its confidence/probability, allowing the user to configure the threshold at which content is filtered.
|
||||
The classifier produces independent toxicity-category probabilities. Users enable the categories they want filtered and configure one shared threshold; content is filtered when any enabled category reaches it.
|
||||
|
||||
## Goals
|
||||
|
||||
|
|
@ -27,16 +22,7 @@ The extension should:
|
|||
|
||||
## Classification Model
|
||||
|
||||
The initial implementation will use a small transformer model trained specifically for toxicity classification.
|
||||
|
||||
A candidate is Citizen Lab's multilingual DistilBERT toxicity classifier, which produces:
|
||||
|
||||
```text
|
||||
toxic
|
||||
not_toxic
|
||||
```
|
||||
|
||||
along with classification probabilities.
|
||||
The implementation uses `wagesj45/multilabel-toxic-comment-classifier`, a multilingual ModernBERT classifier distributed under Apache-2.0. It produces probability scores for toxicity, severe toxicity, obscene, threat, insult, identity attack, and sexual explicit content.
|
||||
|
||||
The model should be converted to ONNX and quantized as aggressively as practical while retaining acceptable classification accuracy.
|
||||
|
||||
|
|
@ -45,7 +31,7 @@ The deployed extension does not require Python or a native companion application
|
|||
Conceptually:
|
||||
|
||||
```text
|
||||
DistilBERT toxicity model
|
||||
ModernBERT multi-label toxicity model
|
||||
|
|
||||
v
|
||||
ONNX
|
||||
|
|
|
|||
Loading…
Reference in a new issue