← Packages
python-tokenizers
Fast, state-of-the-art tokenizers optimized for research and production
- Current
- 0.22.2
- Candidate
- 0.23.3Stale
- License
- Apache-2.0
RuntimeDeps
| Dependency | Observed source | Current requirement | Candidate requirement |
|---|---|---|---|
1.16.0 | |||
3.14.9 |
Optional RuntimeDeps
| Dependency | Observed source | Current requirement | Candidate requirement |
|---|---|---|---|
5.0.1 | |||
2.4.6 | |||
9.0.3 | |||
1.3.0 | |||
2.33.1 | |||
ruffUnmapped | — | ||
1.12.1 | |||
9.1.0 | |||
sphinx-rtd-themeUnmapped | — | ||
0.22.2 | |||
tyUnmapped | — |
Dependency conditions
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "testing"
- Applies when (any)
- extra == "docs"
- Applies when (any)
- extra == "docs"
- Applies when (any)
- extra == "docs"
- Applies when (any)
- extra == "dev"
- Applies when (any)
- extra == "testing"
Description
Tokenizers provides fast and production-ready tokenization implementations for modern natural language processing workloads.
Changelog
PR · 4
Checks
| Monitor | Observation | Last checked |
|---|---|---|
Stale Repository refresh: git fetch exited 128 (authentication) | ||
Stale Source last confirmed at 2026-11-10T09:54:44+00:00; repository refresh overdue. | ||
Checked | ||
Checked | ||
Not configured No reliable monitor identity/configuration. | — | |
Metadata unavailable Comparable SPDX license expressions are missing; free text and RPM aggregate License are not compared. | Last attempted | |
Checked | Last attempted | |
Checked Current source component only; local patches, build options, bundled dependencies and binary artifacts are not evaluated. | ||
Checked |