andylockran/parkrun-hackapi

Watch

Fork 0

Update dependency chardet to v7 #17

Open

renovate wants to merge 1 commit from renovate/chardet-7.x into master

renovate commented

2026-10-27 00:26:29 +00:00

Collaborator

This PR contains the following updates:

Package	Change	Age	Confidence
chardet (changelog)	`==3.0.4` → `==7.7.0`

Release Notes

chardet/chardet (chardet)

`v7.7.0`

Compare Source

Big release: raising max_bytes to cover a huge input no longer makes detect() slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all!

Performance

Raising max_bytes on large inputs no longer gets slow. On compiled wheels, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged.
The pure-Python wheel on CPython scores bigram models through operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster. (#404, thanks @deadstrobe5)
Fixed three scans that went quadratic on hostile input: the UTF-7 + scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed <meta and <?xml tags in the markup scan, and the EBCDIC charset-declaration regex. (#403, thanks @Samin061)

Bug Fixes

Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and <meta charset> declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (EUC-JP, SHIFT_JIS, Big5, ISO-2022-JP) are only used when those narrower codecs decode it too. (#394 and #396, thanks @Samin061)
Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a charset= inside another attribute (like a refresh URL) no longer beats a real <meta charset>, quoted or spaced content charsets work, and an ASCII page declaring utf-16le no longer comes back as UTF-16-LE. (#389 and #398, thanks @SashaMIT and @xiehuanyi)
PEP 263 cookies resolve the way tokenize.detect_encoding resolves them, including only counting a line-2 cookie when line 1 is blank or a comment. (#391, thanks @xiehuanyi)
Fixed BOM-bearing UTF-32 coming back as UTF-16 when max_bytes cuts a code unit. (#405, thanks @AryanRG)
Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104.
Fixed a mostly-ASCII Windows-1252 file with a single Ö detecting as HP-Roman8, and English with a lone É detecting as MacRoman. (#383, thanks @aadsm)
Fixed null-separated records like find -print0 output coming back as UTF-16-LE with Chinese as the language.
prefer_superset=True no longer swaps in a code page that can't decode the data, and now respects include_encodings/exclude_encodings. (#393, thanks @00200200)
chardetect and UniversalDetector no longer mistake input cut exactly at the byte cap for the whole stream. (#392, thanks @00200200)
lookup_encoding's cache is capped at 1024 entries, since charset names come from untrusted markup. (#390, thanks @Samin061)
Plus smaller fixes to detect_all() ranking, EBCDIC XML MIME types, and the deprecated chardet.equivalences shim. The changelog has the full list.

Improvements

New -m/--mime-type flag for chardetect shows the MIME type next to the encoding, like file -i. (#384, thanks @smi11 for the request)
Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it.
EncodingInfo.is_multibyte is now is_multibyte_cjk. The old name still works with a DeprecationWarning and goes away in 8.0.
pipeline/markup.py is now compiled by mypyc (15 modules total), and CI enforces 100% test coverage.

Full Changelog: https://github.com/chardet/chardet/compare/7.6.0...7.7.0

`v7.6.0`

Compare Source

Big release: a Cython scoring kernel joins mypyc in compiled wheels, every model retrained on a deduplicated corpus, UTF-7 fixed in both directions, and a guarantee that detect() never returns an encoding that can't decode your complete input.

Performance

Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14. _kernel.py stays plain Python (PyPy and pure wheels run it interpreted, unchanged), and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in ~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks: HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true.
Added support for CPython 3.15, including the free-threaded build. No code changes were needed.

Bug Fixes

detect() no longer returns an encoding that cannot decode the input it was given (#380, thanks @yarikoptic). When the whole input has been examined and the winner's only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer.
Fixed delimited ASCII data like |NAME,+LAY| misdetecting as UTF-7 (#371 follow-up, thanks @agreenburg). The whole buffer must now actually decode as UTF-7, and a lone shifted character must land in a plausible script range.
Signed UTF-7 no longer reads as ASCII: the BOM stage recognizes the four UTF-7 signature prefixes when the rest of the buffer decodes as UTF-7.
Fixed short apostrophe-heavy English being labeled Scottish Gaelic or Breton: a rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language (ADR-0005).
Fixed Hungarian text losing to a Czech reading in confusion rescoring; tied pairs are compared only under language models both encodings have.
Fixed space-padded text matching a degenerate Serbian model at high confidence; statistical scoring now skips repeated-whitespace bigrams. This also fixes windows-1251 logs misdetecting as windows-1250 (#379).
Fixed EBCDIC text being invisible to the early pipeline stages, and the last two EBCDIC sibling misdetections.
Fixed training normalization gaps that starved ISO-8859-16 and the 26 pre-euro encodings at exactly their distinguishing bytes.

Improvements

Retrained every bigram model on a refreshed, deduplicated corpus with training provenance now recorded per model.
New ANSI-art model for cp437, trained on 16,621 text-mode art files from 16colo.rs.
Rare-language arbitration (ADR-0005): low-confidence statistical winners from languages with no documented legacy-encoding population yield to near-tied mainstream candidates.
Confusion-group resolution is context-aware: per-occurrence votes, word-shape demotions, art-model exemption.
Statistical dead heats no longer resolve by candidate enumeration order.
Training pipeline hardening after a cache-loss post-mortem.

Full Changelog: https://github.com/chardet/chardet/compare/7.5.1...7.6.0

`v7.5.1`

Compare Source

Patch release: three detection fixes found while benchmarking against charset-normalizer's char-dataset.

Bug Fixes

Fixed markup-declared encodings being reported under a name that can't decode the input. A page declaring Shift_JIS but using CP932 extension characters (like ①) came back as SHIFT_JIS, which fails .decode() on those same bytes. Superset promotion (CP932, CP949) now always fires when the reported name can't decode the data but the superset can.
Fixed a lying charset declaration beating genuine UTF-8 content. A UTF-8 page declaring <meta charset="iso-8859-1"> came back as ISO-8859-1, which decodes to mojibake. Valid multi-byte UTF-8 now wins over a conflicting declaration.
Fixed BOM-less UTF-16 byte-order detection for pure-CJK text: short Chinese UTF-16 samples came back with reversed endianness at full confidence. Byte order is now chosen by decoding both ways and comparing text quality.

Full Changelog: https://github.com/chardet/chardet/compare/7.5.0...7.5.1

`v7.5.0`

Compare Source

Accuracy and speed release: truncation-proof byte validity, statistical pruning worth ~2.9x, and half the peak memory.

Bug Fixes

Fixed multi-byte encodings being eliminated when the input ends in an incomplete character. Byte-validity filtering used a one-shot strict decode, which cannot tell a truncated tail from corrupt data, so a single dangling lead byte dropped every CJK candidate and the result came down to input-length parity. Also reachable on complete files through chardet's own max_bytes and _SCAN_LIMIT slicing. Validity checks now decode incrementally with final=False. (#376, thanks @aadsm)
Fixed compat_names (the default) leaking internal Python codec names for seven encodings (ISO-8859-2, ISO-8859-6, ISO-8859-13, Windows-1250, Windows-1256, Windows-1257, CP874). (#374, thanks @aadsm)
Fixed compat_names leaking the internal cp932 codec name; detect() now returns CP932. (#375, thanks @uttam12331)

Performance

Statistical scoring now skips single-byte models that provably can't beat the current runner-up, using per-model row-maximum tables. Results are bit-identical. Mean detection time dropped ~2.9x with mypyc, with the largest gains on legacy CJK (p99 from 9.5ms to 3.3ms).
Model tables are now bytes (native array indexing under mypyc) and the model blob is decompressed in chunks: peak process memory dropped from 53.9 to 27.4 MiB.
Confusion-group resolution and post-processing use bytes.translate prefilters instead of per-byte Python scans.

Improvements

prefer_superset=True is now documented as the recommended mode and will become the default in chardet 8.0. Callers that depend on subset names should start passing prefer_superset=False explicitly.
chardet.equivalences is now a deprecation shim: evaluation predicates moved to chardet.evaluation, output-name remapping to chardet.output_names. Removal in 8.0.
Internal pipeline reorganization: language fill, markup-superset promotion, and post-processing moved into their own modules. No behavior change.

Full Changelog: https://github.com/chardet/chardet/compare/7.4.3...7.5.0

`v7.4.3`

Compare Source

Patch release: fixes a crash when input contains null bytes inside a <meta charset> declaration.

Bug Fixes

Fixed ValueError: embedded null character crash when input contained a <meta charset> declaration with a null byte in the encoding name (e.g. b'<meta charset="\x00utf-8">'). codecs.lookup() raises ValueError on embedded nulls, and lookup_encoding() was only catching LookupError. Also added defensive ValueError catches in _validate_bytes() and _to_utf8() for completeness. (#369, thanks @DRMacIver for the report)

Full Changelog: https://github.com/chardet/chardet/compare/7.4.2...7.4.3

`v7.4.2`

Compare Source

Patch release: fixes a crash on short inputs and closes a bunch of WHATWG/IANA alias gaps.

Bug Fixes

Fixed RuntimeError: pipeline must always return at least one result on ~2% of all possible two-byte inputs (e.g. b"\xf9\x92"). Multi-byte encodings like CP932 and Johab could score above the structural confidence threshold on very short inputs, but then statistical scoring would return nothing, leaving an empty result list instead of falling through to the fallback. (#367, #368, thanks @jasonwbarnett)

Improvements

Added ~90 encoding aliases from the WHATWG Encoding Standard and IANA Character Sets registry so that <meta charset> labels like x-cp1252, x-sjis, dos-874, csUTF8, and the cswindows* family all resolve correctly through the markup detection stage. Every alias was driven by a failing spec-compliance test, not speculative. (#366)
Added a spec-compliance test suite covering Python decode round-trips for all 86 registry encodings, WHATWG label resolution, IANA preferred MIME names, and Unicode/RFC conformance (BOM sniffing, UTF-8 boundary cases, UTF-16 surrogate pairs). This is the test suite that would have caught the 7.4.1 BOM bug before release. (#366)

Full Changelog: https://github.com/chardet/chardet/compare/7.4.1...7.4.2

`v7.4.1`

Compare Source

Bug Fixes

BOM-prefixed UTF-16/32 input now returns utf-16/utf-32 instead of utf-16-le/utf-16-be/utf-32-le/utf-32-be. The endian-specific codecs don't strip the BOM on decode, so callers were getting a stray U+FEFF at the start of their text. BOM-less detection is unchanged. (#364, #365)

Full Changelog: https://github.com/chardet/chardet/compare/7.4.0...7.4.1

License

0BSD license — the project license has been changed from MIT to 0BSD, a maximally permissive license with no attribution requirement. All prior 7.x releases should also be considered 0BSD licensed as of this release.

Features

Added mime_type field to detection results — identifies file types for both binary (via magic number matching) and text content. Returned in all detect(), detect_all(), and UniversalDetector results. (#350)
New pipeline/magic.py module detects 40+ binary file formats including images, audio/video, archives, documents, executables, and fonts. ZIP-based formats (XLSX, DOCX, JAR, APK, EPUB, wheel, OpenDocument) are distinguished by entry filenames. (#350)

Bug Fixes

Fixed incorrect equivalence between UTF-16-LE and UTF-16-BE in accuracy testing — these are distinct encodings with different byte order, not interchangeable

Performance

Added 4 new modules to mypyc compilation (orchestrator, confusion, magic, ascii), bringing the total to 11 compiled modules
Capped statistical scoring at 16 KB — bigram models converge quickly, so large files no longer score the full 200 KB. Worst-case detection time dropped from 62ms to 26ms with no accuracy loss.
Replaced dataclasses.replace() with direct DetectionResult construction on hot paths, eliminating ~354k function calls per full test suite run

Build

Added riscv64 to the mypyc wheel build matrix — prebuilt wheels are now published for RISC-V Linux alongside existing architectures (#348, thanks @gounthar)

`v7.2.0`: chardet 7.2.0

Compare Source

Features

Added include_encodings and exclude_encodings parameters to detect(), detect_all(), and UniversalDetector — restrict or exclude specific encodings from the candidate set, with corresponding -i/--include-encodings and -x/--exclude-encodings CLI flags (#343)
Added no_match_encoding (default "cp1252") and empty_input_encoding (default "utf-8") parameters — control which encoding is returned when no candidate survives the pipeline or the input is empty, with corresponding CLI flags (#343)
Added -l/--language flag to chardetect CLI — shows the detected language (ISO 639-1 code and English name) alongside the encoding (#342)

Fixes

Fixed null-separated ASCII data being misdetected as UTF-16-BE (#346, #347)

Full changelog: https://chardet.readthedocs.io/en/latest/changelog.html

`v7.1.0`: chardet 7.1.0

Compare Source

Features

Added PEP 263 encoding declaration detection — # -*- coding: ... -*- and # coding=... declarations on lines 1–2 of Python source files are now recognized with confidence 0.95 (#249)
Added chardet.universaldetector backward-compatibility stub so that from chardet.universaldetector import UniversalDetector works with a deprecation warning (#341)

Fixes

Fixed false UTF-7 detection of ASCII text containing ++ or +word patterns (#332)
Fixed 0.5s startup cost on first detect() call — model norms are now computed during loading instead of lazily iterating 21M entries (#333)
Fixed undocumented encoding name changes between chardet 5.x and 7.0 — detect() now returns chardet 5.x-compatible names by default (#338)
Improved ISO-2022-JP family detection — recognizes ESC sequences for ISO-2022-JP-2004 (JIS X 0213) and ISO-2022-JP-EXT (JIS X 0201 Kana)
Fixed silent truncation of corrupt model data (iter_unpack yielded fewer tuples instead of raising)
Fixed incorrect date in LICENSE

Performance

5.5x faster first-detect time (~0.42s → ~0.075s) by computing model norms as a side-product of load_models()
~40% faster model parsing via struct.iter_unpack for bulk entry extraction (eliminates ~305K individual unpack calls)

New API parameters

Added compat_names parameter (default True) to detect(), detect_all(), and UniversalDetector — set to False to get raw Python codec names instead of chardet 5.x/6.x compatible display names
Added prefer_superset parameter (default False) — remaps legacy ISO/subset encodings to their modern Windows/CP superset equivalents (e.g., ASCII → Windows-1252, ISO-8859-1 → Windows-1252). This will default to True in the next major version (8.0).
Deprecated should_rename_legacy in favor of prefer_superset — a deprecation warning is emitted when used

Improvements

Switched internal canonical encoding names to Python codec names (e.g., "utf-8" instead of "UTF-8"), with compat_names controlling the public output format
Added lookup_encoding() to registry for case-insensitive resolution of arbitrary encoding name input to canonical names
Achieved 100% line coverage across all source modules (+31 tests)
Updated benchmark numbers: 98.2% encoding accuracy, 95.2% language accuracy on 2,510 test files
Pinned test-data cloning to chardet release version tags for reproducible builds

Full changelog: https://chardet.readthedocs.io/en/latest/changelog.html

`v7.0.1`

Compare Source

Fixes

Fixed false UTF-7 detection of SHA-1 git hashes (#324, fixing #323) — requirements files with VCS pins (e.g., +4bafdea3...) were misdetected as UTF-7, breaking tools like tox
Fixed _SINGLE_LANG_MAP missing aliases for single-language encoding lookup (e.g., big5 → big5hkscs)
Fixed PyPy TypeError in UTF-7 codec handling

Improvements

Retrained bigram models — 24 previously failing test cases now pass
Updated language equivalences for mutual intelligibility (Slovak/Czech, East Slavic + Bulgarian, Malay/Indonesian, Scandinavian languages)

New Contributors

@rembish made their first contribution — both reporting the UTF-7 false detection issue and submitting the fix! (#323, #324)

`v7.0.0`

Compare Source

Ground-up, MIT-licensed rewrite of chardet. Same package name, same public API — drop-in replacement for chardet 5.x/6.x. Just way faster and more accurate!

Highlights:

MIT license (previous versions were LGPL)
96.8% accuracy on 2,179 test files (+2.3pp vs chardet 6.0.0, +7.7pp vs charset-normalizer)
41x faster than chardet 6.0.0 with mypyc (28x pure Python), 7.5x faster than charset-normalizer
Language detection for every result (90.5% accuracy across 49 languages)
99 encodings across six eras (MODERN_WEB, LEGACY_ISO, LEGACY_MAC, LEGACY_REGIONAL, DOS, MAINFRAME)
12-stage detection pipeline — BOM, UTF-16/32 patterns, escape sequences, binary detection, markup charset, ASCII, UTF-8 validation, byte validity, CJK gating, structural probing, statistical scoring, post-processing
Bigram frequency models trained on CulturaX multilingual corpus data for all supported language/encoding pairs
Optional mypyc compilation — 1.49x additional speedup on CPython
Thread-safe detect() and detect_all() with no measurable overhead; scales on free-threaded Python 3.13t+
Negligible import memory (96 B)
Zero runtime dependencies

Breaking changes vs 6.0.0:

detect() and detect_all() now default to encoding_era=EncodingEra.ALL (6.0.0 defaulted to MODERN_WEB)
Internal architecture is completely different (probers replaced by pipeline stages). Only the public API is preserved.
LanguageFilter is accepted but ignored (deprecation warning emitted)
chunk_size is accepted but ignored (deprecation warning emitted)

`v6.0.0.post1`

Compare Source

Fixed version number in chardet/version.py still being set to 6.0.0dev0. Otherwise identical to 6.0.0.

`v6.0.0`

Compare Source

Features

Unified single-byte charset detection: Instead of only having trained language models for a handful of languages (Bulgarian, Greek, Hebrew, Hungarian, Russian, Thai, Turkish) and relying on special-case Latin1Prober and MacRomanProber heuristics for Western encodings, chardet now treats all single-byte charsets the same way: every encoding gets proper language-specific bigram models trained on CulturaX corpus data. This means chardet can now accurately detect both the encoding and the language for all supported single-byte encodings.
38 new languages: Arabic, Belarusian, Breton, Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Farsi, Finnish, French, German, Icelandic, Indonesian, Irish, Italian, Kazakh, Latvian, Lithuanian, Macedonian, Malay, Maltese, Norwegian, Polish, Portuguese, Romanian, Scottish Gaelic, Serbian, Slovak, Slovene, Spanish, Swedish, Tajik, Ukrainian, Vietnamese, and Welsh. Existing models for Bulgarian, Greek, Hebrew, Hungarian, Russian, Thai, and Turkish were also retrained with the new pipeline.
EncodingEra filtering: New encoding_era parameter to detect allows filtering by an EncodingEra flag enum (MODERN_WEB, LEGACY_ISO, LEGACY_MAC, LEGACY_REGIONAL, DOS, MAINFRAME, ALL) allows callers to restrict detection to encodings from a specific era. detect() and detect_all() default to MODERN_WEB. The new MODERN_WEB default should drastically improve accuracy for users who are not working with legacy data. The tiers are:
- MODERN_WEB: UTF-8/16/32, Windows-125x, CP874, CJK multi-byte (widely used on the web)
- LEGACY_ISO: ISO-8859-x, KOI8-R/U (legacy but well-known standards)
- LEGACY_MAC: Mac-specific encodings (MacRoman, MacCyrillic, etc.)
- LEGACY_REGIONAL: Uncommon regional/national encodings (KOI8-T, KZ1048, CP1006, etc.)
- DOS: DOS/OEM code pages (CP437, CP850, CP866, etc.)
- MAINFRAME: EBCDIC variants (CP037, CP500, etc.)
--encoding-era CLI flag: The chardetect CLI now accepts -e/--encoding-era to control which encoding eras are considered during detection.
max_bytes and chunk_size parameters: detect(), detect_all(), and UniversalDetector now accept max_bytes (default 200KB) and chunk_size (default 64KB) parameters for controlling how much data is examined. (#314, @bysiber)
Encoding era preference tie-breaking: When multiple encodings have very close confidence scores, the detector now prefers more modern/Unicode encodings over legacy ones.
Charset metadata registry: New chardet.metadata.charsets module provides structured metadata about all supported encodings, including their era classification and language filter.
should_rename_legacy now defaults intelligently: When set to None (the new default), legacy renaming is automatically enabled when encoding_era is MODERN_WEB.
Direct GB18030 support: Replaced the redundant GB2312 prober with a proper GB18030 prober.
EBCDIC detection: Added CP037 and CP500 EBCDIC model registrations for mainframe encoding detection.
Binary file detection: Added basic binary file detection to abort analysis earlier on non-text files.
Python 3.12, 3.13, and 3.14 support (#283, @hugovk; #311)
GitHub Codespace support (#312, @oxygen-dioxide)

Fixes

Fix CP949 state machine: Corrected the state machine for Korean CP949 encoding detection. (#268, @nenw)
Fix SJIS distribution analysis: Fixed SJISDistributionAnalysis discarding valid second-byte range >= 0x80. (#315, @bysiber)
Fix UTF-16/32 detection for non-ASCII-heavy text: Improved detection of UTF-16/32 encoded CJK and other non-ASCII text by adding a MIN_RATIO threshold alongside the existing EXPECTED_RATIO.
Fix get_charset crash: Resolved a crash when looking up unknown charset names.
Fix GB18030 char_len_table: Corrected the character length table for GB18030 multi-byte sequences.
Fix UTF-8 state machine: Updated to be more spec-compliant.
Fix detect_all() returning inactive probers: Results from probers that determined "definitely not this encoding" are now excluded.
Fix early cutoff bug: Resolved an issue where detection could terminate prematurely.
Default UTF-8 fallback: If UTF-8 has not been ruled out and nothing else is above the minimum threshold, UTF-8 is now returned as the default.

Breaking changes

Dropped Python 3.7, 3.8, and 3.9 support: Now requires Python 3.10+. (#283, @hugovk)
Removed Latin1Prober and MacRomanProber: These special-case probers have been replaced by the unified model-based approach described above. Latin-1, MacRoman, and all other single-byte encodings are now detected by SingleByteCharSetProber with trained language models, giving better accuracy and language identification.
Removed EUC-TW support: EUC-TW encoding detection has been removed as it is extremely rare in practice.
LanguageFilter.NONE removed: Use specific language filters or LanguageFilter.ALL instead.
Enum types changed: InputState, ProbingState, MachineState, SequenceLikelihood, and CharacterCategory are now IntEnum (previously plain classes or Enum). LanguageFilter values changed from hardcoded hex to auto().
detect() default behavior change: detect() now defaults to encoding_era=EncodingEra.MODERN_WEB and should_rename_legacy=None (auto-enabled for MODERN_WEB), whereas previously it defaulted to considering all encodings with no legacy renaming.

Misc changes

Switched from Poetry/setuptools to uv + hatchling: Build system modernized with hatch-vcs for version management.
License text updated: Updated LGPLv2.1 license text and FSF notices to use URL instead of mailing address. (#304, #307, @musicinmybrain)
CulturaX-based model training: The create_language_model.py training script was rewritten to use the CulturaX multilingual corpus instead of Wikipedia, producing higher quality bigram frequency models.
Language class converted to frozen dataclass: The language metadata class now uses @dataclass(frozen=True) with num_training_docs and num_training_chars fields replacing wiki_start_pages.
Test infrastructure: Added pytest-timeout and pytest-xdist for faster parallel test execution. Reorganized test data directories.

Contributors

Thank you to everyone who contributed to this release!

@dan-blanchard (Dan Blanchard)
@bysiber (Kadir Can Ozden)
@musicinmybrain (Ben Beasley)
@hugovk (Hugo van Kemenade)
@oxygen-dioxide
@nenw

And a special thanks to @helour, whose earlier Latin-1 prober work from an abandoned PR helped inform the approach taken in this release.

`v5.2.0`: chardet 5.2.0

Compare Source

Adds support for running chardet CLI via python -m chardet (0e9b7bc, @dan-blanchard)

`v5.1.0`: chardet 5.1.0

Compare Source

Features

Add should_rename_legacy argument to most functions, which will rename older encodings to their more modern equivalents (e.g., GB2312 becomes GB18030) (#264, @dan-blanchard)
Add capital letter sharp S and ISO-8859-15 support (#222, @SimonWaldherr)
Add a prober for MacRoman encoding (#5 updated as c292b52, Rob Speer and @dan-blanchard )
Add --minimal flag to chardetect command (#214, @dan-blanchard)
Add type annotations to the project and run mypy on CI (#261, @jdufresne)
Add support for Python 3.11 (#274, @hugovk)

Fixes

Clarify LGPL version in License trove classifier (#255, @musicinmybrain)
Remove support for EOL Python 3.6 (#260, @jdufresne)
Remove unnecessary guards for non-falsey values (#259, @jdufresne)

Misc changes

Switch to Python 3.10 release in GitHub actions (#257, @jdufresne)
Remove setup.py in favor of build package (#262, @jdufresne)
Run tests on macos, Windows, and 3.11-dev (#267, @dan-blanchard)

`v5.0.0`: chardet 5.0.0

Compare Source

⚠️ This release is the first release of chardet that no longer supports Python < 3.6 ⚠️

In addition to that change, it features the following user-facing changes:

Added a prober for Johab Korean (#207, @grizlupo)
Added a prober for UTF-16/32 BE/LE (#109, #206, @jpz)
Added test data for Croatian, Czech, Hungarian, Polish, Slovak, Slovene, Greek, and Turkish, which should help prevent future errors with those languages
Improved XML tag filtering, which should improve accuracy for XML files (#208)
Tweaked SingleByteCharSetProber confidence to match latest uchardet (#209)
Made detect_all return child prober confidences (#210)
Updated examples in docs (#223, @domdfcoding)
Documentation fixes (#212, #224, #225, #226, #220, #221, #244 from too many to mention)
Minor performance improvements (#252, @deedy5)
Add support for Python 3.10 when testing (#232, @jdufresne)
Lots of little development cycle improvements, mostly thanks to @jdufresne

`v4.0.0`: chardet 4.0.0

Compare Source

⚠️ This will be the last release of chardet to support Python 2.7. chardet 5.0 will only support 3.6+ ⚠️

Major Changes

This release is multiple years in the making, and provides some quality of life improvements to chardet. The primary user-facing changes are:

Single-byte charset probers now use nested dictionaries under the hood, so they are usually a little faster than before. (See #121 for details)
The CharsetGroupProber class now properly short-circuits when one of the probers in the group is considered a definite match. This lead to a substantial speedup.
There is now a chardet.detect_all function that returns a list of possible encodings for the input with associated confidences.
We have dropped support for Python 2.6, 3.4, and 3.5 as they are all past end-of-life.

The changes in this release have also laid the groundwork for retraining the models to make them more accurate, and to support some more encodings/languages (see #99 for progress). This is our main focus for chardet 5.0 (beyond dropping Python 2 support).

Benchmarks

Running on a MacBook Pro (15-inch, 2018) with 2.2GHz 6-core i7 processor and 32GB RAM

old version (chardet 3.0.4)

Benchmarking chardet 3.0.4 on CPython 3.7.5 (default, Sep  8 2020, 12:19:42)
[Clang 11.0.3 (clang-1103.0.32.62)]
--------------------------------------------------------------------------------
Calls per second for each encoding:
ascii: 25559.439366240098
big5: 7.187002209518091
cp932: 4.71090956645177
cp949: 2.937256786994428
euc-jp: 4.870580412090848
euc-kr: 6.6910755971933416
euc-tw: 87.71098043480079
gb2312: 6.614302607154443
ibm855: 27.595893549680685
ibm866: 29.93483661732791
iso-2022-jp: 3379.5052775763434
iso-2022-kr: 26181.67290886392
iso-8859-1: 120.63424740403983
iso-8859-5: 32.65106262196898
iso-8859-7: 62.480089080556084
koi8-r: 13.72481001727257
maccyrillic: 33.018537255804496
shift_jis: 4.996013583677438
tis-620: 14.323112928341818
utf-16: 166771.53081510935
utf-32: 198782.18009478672
utf-8: 13.966236809766901
utf-8-sig: 193732.28637413395
windows-1251: 23.038910006925768
windows-1252: 99.48409117053738 
windows-1255: 6.336261495718825

Total time: 357.05358052253723s (10.054513372323958 calls per second)

new version (chardet 4.0.0)


Benchmarking chardet 4.0.0 on CPython 3.7.5 (default, Sep  8 2020, 12:19:42)
[Clang 11.0.3 (clang-1103.0.32.62)]
--------------------------------------------------------------------------------
.......................................................................................................................................................................................................................................................................................................................................................................
Calls per second for each encoding:
ascii: 38176.31067961165
big5: 12.86915132656389
cp932: 4.656400877065864
cp949: 7.282976434315926
euc-jp: 4.329381447610525
euc-kr: 8.16386823884839
euc-tw: 90.230745070368
gb2312: 14.248865889128146
ibm855: 33.30225548069821
ibm866: 44.181691968506
iso-2022-jp: 3024.2295767539117
iso-2022-kr: 25055.57945041816
iso-8859-1: 59.25262902122995
iso-8859-5: 39.7069713674529
iso-8859-7: 61.008422013862194
koi8-r: 41.21560517643845
maccyrillic: 31.402474369805002
shift_jis: 4.9091652743515155
tis-620: 14.408875278821073
utf-16: 177349.00634249471
utf-32: 186413.51111111112
utf-8: 108.62174360115105
utf-8-sig: 181965.46637744035
windows-1251: 43.16933400329809
windows-1252: 211.27653358317968
windows-1255: 16.15113643694104

Total time: 268.0230791568756s (13.394368915143872 calls per second)

Thank you to @aaaxx, @edumco, @hrnciar, @hroncok, @jdufresne, @mdamien, @saintamh , @xeor for submitting pull requests, to all of our users for being patient with how long this release has taken.

Full changelog

Convert single-byte charset probers to use nested dicts for language models (#121) @dan-blanchard
Add API option to get all the encodings confidence (#111) @mdamien
Make sure pyc files are not in tarballs (d7c7343) @dan-blanchard
Add benchmark script (d702545, 8dccd00, 726973e, 71a0fad) @dan-blanchard
Include license file in the generated wheel package (#141) @jdufresne
Drop support for Python 2.6 (#143) @jdufresne
Remove unused coverage configuration (#142) @jdufresne
Doc the chardet package suitable for production (#144) @jdufresne
Pass python_requires argument to setuptools (#150) @jdufresne
Update pypi.python.org URL to pypi.org (#155) @jdufresne
Typo fix (#159) @saintamh
Support pytest 4, don't apply marks directly to parameters (PR #174, Issue #173) @hroncok
Test Python 3.7 and 3.8 and document support (#175) @jdufresne
Drop support for end-of-life Python 3.4 (#181) @jdufresne
Workaround for distutils bug in python 2.7 (#165) @xeor
Remove deprecated license_file from setup.cfg (#182) @jdufresne
Remove deprecated 'sudo: false' from Travis configuraiton (#200) @jdufresne
Add testing for Python 3.9 (#201) @jdufresne
Adds explicit os and distro definitions (#140) @edumco
Remove shebang from nonexecutable script (#192) @hrnciar
Remove use of deprecated 'setup.py test' (#187) @jdufresne
Remove unnecessary numeric placeholders from format strings (#176) @jdufresne
Update links (#152) @aaaxx
Remove shebang and executable bit from chardet/cli/chardetect.py (#171) @jdufresne
Handle weird logging edge case in universaldetector.py (056a2a4) @dan-blanchard
Switch from Travis to GitHub Actions (#204) @dan-blanchard
Properly set CharsetGroupProber.state to FOUND_IT (PR #203, Issue #202) @dan-blanchard
Add language to detect_all output (1e208b7) @dan-blanchard

Configuration

📅 Schedule: (UTC)

Branch creation
- At any time (no schedule defined)
Automerge
- At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate CLI.

This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [chardet](https://github.com/chardet/chardet) ([changelog](https://chardet.readthedocs.io/en/latest/changelog.html)) | `==3.0.4` → `==7.7.0` | ![age](https://developer.mend.io/api/mc/badges/age/pypi/chardet/7.7.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/chardet/3.0.4/7.7.0?slim=true) | --- ### Release Notes <details> <summary>chardet/chardet (chardet)</summary> ### [`v7.7.0`](https://github.com/chardet/chardet/releases/tag/7.7.0) [Compare Source](https://github.com/chardet/chardet/compare/7.6.0...7.7.0) Big release: raising `max_bytes` to cover a huge input no longer makes `detect()` slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all! #### Performance - Raising `max_bytes` on large inputs no longer gets slow. On compiled wheels, `detect(data, max_bytes=len(data))` on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged. - The pure-Python wheel on CPython scores bigram models through `operator.itemgetter` instead of a Python loop, making `detect()` about 1.25x faster and `detect_all()` about 1.35x faster. ([#404](https://github.com/chardet/chardet/issues/404), thanks [@deadstrobe5](https://github.com/deadstrobe5)) - Fixed three scans that went quadratic on hostile input: the UTF-7 `+` scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed `<meta` and `<?xml` tags in the markup scan, and the EBCDIC charset-declaration regex. ([#403](https://github.com/chardet/chardet/issues/403), thanks [@Samin061](https://github.com/Samin061)) #### Bug Fixes - Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and `<meta charset>` declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (`EUC-JP`, `SHIFT_JIS`, `Big5`, `ISO-2022-JP`) are only used when those narrower codecs decode it too. ([#394](https://github.com/chardet/chardet/issues/394) and [#396](https://github.com/chardet/chardet/issues/396), thanks [@Samin061](https://github.com/Samin061)) - Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a `charset=` inside another attribute (like a refresh URL) no longer beats a real `<meta charset>`, quoted or spaced `content` charsets work, and an ASCII page declaring `utf-16le` no longer comes back as UTF-16-LE. ([#389](https://github.com/chardet/chardet/issues/389) and [#398](https://github.com/chardet/chardet/issues/398), thanks [@SashaMIT](https://github.com/SashaMIT) and [@xiehuanyi](https://github.com/xiehuanyi)) - PEP 263 cookies resolve the way `tokenize.detect_encoding` resolves them, including only counting a line-2 cookie when line 1 is blank or a comment. ([#391](https://github.com/chardet/chardet/issues/391), thanks [@xiehuanyi](https://github.com/xiehuanyi)) - Fixed BOM-bearing UTF-32 coming back as UTF-16 when `max_bytes` cuts a code unit. ([#405](https://github.com/chardet/chardet/issues/405), thanks [@AryanRG](https://github.com/AryanRG)) - Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104. - Fixed a mostly-ASCII Windows-1252 file with a single `Ö` detecting as HP-Roman8, and English with a lone `É` detecting as MacRoman. ([#383](https://github.com/chardet/chardet/issues/383), thanks [@aadsm](https://github.com/aadsm)) - Fixed null-separated records like `find -print0` output coming back as UTF-16-LE with Chinese as the language. - `prefer_superset=True` no longer swaps in a code page that can't decode the data, and now respects `include_encodings`/`exclude_encodings`. ([#393](https://github.com/chardet/chardet/issues/393), thanks [@00200200](https://github.com/00200200)) - `chardetect` and `UniversalDetector` no longer mistake input cut exactly at the byte cap for the whole stream. ([#392](https://github.com/chardet/chardet/issues/392), thanks [@00200200](https://github.com/00200200)) - `lookup_encoding`'s cache is capped at 1024 entries, since charset names come from untrusted markup. ([#390](https://github.com/chardet/chardet/issues/390), thanks [@Samin061](https://github.com/Samin061)) - Plus smaller fixes to `detect_all()` ranking, EBCDIC XML MIME types, and the deprecated `chardet.equivalences` shim. The [changelog](https://chardet.readthedocs.io/en/latest/changelog.html) has the full list. #### Improvements - New `-m`/`--mime-type` flag for `chardetect` shows the MIME type next to the encoding, like `file -i`. ([#384](https://github.com/chardet/chardet/issues/384), thanks [@smi11](https://github.com/smi11) for the request) - Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it. - `EncodingInfo.is_multibyte` is now `is_multibyte_cjk`. The old name still works with a `DeprecationWarning` and goes away in 8.0. - `pipeline/markup.py` is now compiled by mypyc (15 modules total), and CI enforces 100% test coverage. **Full Changelog**: <https://github.com/chardet/chardet/compare/7.6.0...7.7.0> ### [`v7.6.0`](https://github.com/chardet/chardet/releases/tag/7.6.0) [Compare Source](https://github.com/chardet/chardet/compare/7.5.1...7.6.0) Big release: a Cython scoring kernel joins mypyc in compiled wheels, every model retrained on a deduplicated corpus, UTF-7 fixed in both directions, and a guarantee that `detect()` never returns an encoding that can't decode your complete input. #### Performance - Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14. `_kernel.py` stays plain Python (PyPy and pure wheels run it interpreted, unchanged), and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in \~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks: `HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true`. - Added support for CPython 3.15, including the free-threaded build. No code changes were needed. #### Bug Fixes - `detect()` no longer returns an encoding that cannot decode the input it was given ([#380](https://github.com/chardet/chardet/issues/380), thanks [@yarikoptic](https://github.com/yarikoptic)). When the whole input has been examined and the winner's only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer. - Fixed delimited ASCII data like `|NAME,+LAY|` misdetecting as UTF-7 ([#371](https://github.com/chardet/chardet/issues/371) follow-up, thanks [@agreenburg](https://github.com/agreenburg)). The whole buffer must now actually decode as UTF-7, and a lone shifted character must land in a plausible script range. - Signed UTF-7 no longer reads as ASCII: the BOM stage recognizes the four UTF-7 signature prefixes when the rest of the buffer decodes as UTF-7. - Fixed short apostrophe-heavy English being labeled Scottish Gaelic or Breton: a rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language (ADR-0005). - Fixed Hungarian text losing to a Czech reading in confusion rescoring; tied pairs are compared only under language models both encodings have. - Fixed space-padded text matching a degenerate Serbian model at high confidence; statistical scoring now skips repeated-whitespace bigrams. This also fixes windows-1251 logs misdetecting as windows-1250 ([#379](https://github.com/chardet/chardet/issues/379)). - Fixed EBCDIC text being invisible to the early pipeline stages, and the last two EBCDIC sibling misdetections. - Fixed training normalization gaps that starved ISO-8859-16 and the 26 pre-euro encodings at exactly their distinguishing bytes. #### Improvements - Retrained every bigram model on a refreshed, deduplicated corpus with training provenance now recorded per model. - New ANSI-art model for cp437, trained on 16,621 text-mode art files from [16colo.rs](https://16colo.rs/). - Rare-language arbitration (ADR-0005): low-confidence statistical winners from languages with no documented legacy-encoding population yield to near-tied mainstream candidates. - Confusion-group resolution is context-aware: per-occurrence votes, word-shape demotions, art-model exemption. - Statistical dead heats no longer resolve by candidate enumeration order. - Training pipeline hardening after a cache-loss post-mortem. **Full Changelog**: <https://github.com/chardet/chardet/compare/7.5.1...7.6.0> ### [`v7.5.1`](https://github.com/chardet/chardet/releases/tag/7.5.1) [Compare Source](https://github.com/chardet/chardet/compare/7.5.0...7.5.1) Patch release: three detection fixes found while benchmarking against charset-normalizer's char-dataset. #### Bug Fixes - Fixed markup-declared encodings being reported under a name that can't decode the input. A page declaring `Shift_JIS` but using CP932 extension characters (like ①) came back as `SHIFT_JIS`, which fails `.decode()` on those same bytes. Superset promotion (`CP932`, `CP949`) now always fires when the reported name can't decode the data but the superset can. - Fixed a lying charset declaration beating genuine UTF-8 content. A UTF-8 page declaring `<meta charset="iso-8859-1">` came back as ISO-8859-1, which decodes to mojibake. Valid multi-byte UTF-8 now wins over a conflicting declaration. - Fixed BOM-less UTF-16 byte-order detection for pure-CJK text: short Chinese UTF-16 samples came back with reversed endianness at full confidence. Byte order is now chosen by decoding both ways and comparing text quality. **Full Changelog**: <https://github.com/chardet/chardet/compare/7.5.0...7.5.1> ### [`v7.5.0`](https://github.com/chardet/chardet/releases/tag/7.5.0) [Compare Source](https://github.com/chardet/chardet/compare/7.4.3...7.5.0) Accuracy and speed release: truncation-proof byte validity, statistical pruning worth \~2.9x, and half the peak memory. #### Bug Fixes - Fixed multi-byte encodings being eliminated when the input ends in an incomplete character. Byte-validity filtering used a one-shot strict decode, which cannot tell a truncated tail from corrupt data, so a single dangling lead byte dropped every CJK candidate and the result came down to input-length parity. Also reachable on complete files through chardet's own `max_bytes` and `_SCAN_LIMIT` slicing. Validity checks now decode incrementally with `final=False`. ([#376](https://github.com/chardet/chardet/issues/376), thanks [@aadsm](https://github.com/aadsm)) - Fixed `compat_names` (the default) leaking internal Python codec names for seven encodings (`ISO-8859-2`, `ISO-8859-6`, `ISO-8859-13`, `Windows-1250`, `Windows-1256`, `Windows-1257`, `CP874`). ([#374](https://github.com/chardet/chardet/issues/374), thanks [@aadsm](https://github.com/aadsm)) - Fixed `compat_names` leaking the internal `cp932` codec name; `detect()` now returns `CP932`. ([#375](https://github.com/chardet/chardet/issues/375), thanks [@uttam12331](https://github.com/uttam12331)) #### Performance - Statistical scoring now skips single-byte models that provably can't beat the current runner-up, using per-model row-maximum tables. Results are bit-identical. Mean detection time dropped \~2.9x with mypyc, with the largest gains on legacy CJK (p99 from 9.5ms to 3.3ms). - Model tables are now `bytes` (native array indexing under mypyc) and the model blob is decompressed in chunks: peak process memory dropped from 53.9 to 27.4 MiB. - Confusion-group resolution and post-processing use `bytes.translate` prefilters instead of per-byte Python scans. #### Improvements - `prefer_superset=True` is now documented as the recommended mode and **will become the default in chardet 8.0**. Callers that depend on subset names should start passing `prefer_superset=False` explicitly. - `chardet.equivalences` is now a deprecation shim: evaluation predicates moved to `chardet.evaluation`, output-name remapping to `chardet.output_names`. Removal in 8.0. - Internal pipeline reorganization: language fill, markup-superset promotion, and post-processing moved into their own modules. No behavior change. **Full Changelog**: <https://github.com/chardet/chardet/compare/7.4.3...7.5.0> ### [`v7.4.3`](https://github.com/chardet/chardet/releases/tag/7.4.3) [Compare Source](https://github.com/chardet/chardet/compare/7.4.2...7.4.3) Patch release: fixes a crash when input contains null bytes inside a `<meta charset>` declaration. #### Bug Fixes - Fixed `ValueError: embedded null character` crash when input contained a `<meta charset>` declaration with a null byte in the encoding name (e.g. `b'<meta charset="\x00utf-8">'`). `codecs.lookup()` raises `ValueError` on embedded nulls, and `lookup_encoding()` was only catching `LookupError`. Also added defensive `ValueError` catches in `_validate_bytes()` and `_to_utf8()` for completeness. ([#369](https://github.com/chardet/chardet/issues/369), thanks [@DRMacIver](https://github.com/DRMacIver) for the report) **Full Changelog**: <https://github.com/chardet/chardet/compare/7.4.2...7.4.3> ### [`v7.4.2`](https://github.com/chardet/chardet/releases/tag/7.4.2) [Compare Source](https://github.com/chardet/chardet/compare/7.4.1...7.4.2) Patch release: fixes a crash on short inputs and closes a bunch of WHATWG/IANA alias gaps. #### Bug Fixes - Fixed `RuntimeError: pipeline must always return at least one result` on \~2% of all possible two-byte inputs (e.g. `b"\xf9\x92"`). Multi-byte encodings like CP932 and Johab could score above the structural confidence threshold on very short inputs, but then statistical scoring would return nothing, leaving an empty result list instead of falling through to the fallback. ([#367](https://github.com/chardet/chardet/issues/367), [#368](https://github.com/chardet/chardet/issues/368), thanks [@jasonwbarnett](https://github.com/jasonwbarnett)) #### Improvements - Added \~90 encoding aliases from the WHATWG Encoding Standard and IANA Character Sets registry so that `<meta charset>` labels like `x-cp1252`, `x-sjis`, `dos-874`, `csUTF8`, and the `cswindows*` family all resolve correctly through the markup detection stage. Every alias was driven by a failing spec-compliance test, not speculative. ([#366](https://github.com/chardet/chardet/issues/366)) - Added a spec-compliance test suite covering Python decode round-trips for all 86 registry encodings, WHATWG label resolution, IANA preferred MIME names, and Unicode/RFC conformance (BOM sniffing, UTF-8 boundary cases, UTF-16 surrogate pairs). This is the test suite that would have caught the 7.4.1 BOM bug before release. ([#366](https://github.com/chardet/chardet/issues/366)) **Full Changelog**: <https://github.com/chardet/chardet/compare/7.4.1...7.4.2> ### [`v7.4.1`](https://github.com/chardet/chardet/releases/tag/7.4.1) [Compare Source](https://github.com/chardet/chardet/compare/7.4.0.post2...7.4.1) #### Bug Fixes - BOM-prefixed UTF-16/32 input now returns `utf-16`/`utf-32` instead of `utf-16-le`/`utf-16-be`/`utf-32-le`/`utf-32-be`. The endian-specific codecs don't strip the BOM on decode, so callers were getting a stray U+FEFF at the start of their text. BOM-less detection is unchanged. ([#364](https://github.com/chardet/chardet/issues/364), [#365](https://github.com/chardet/chardet/issues/365)) **Full Changelog**: <https://github.com/chardet/chardet/compare/7.4.0...7.4.1> ### [`v7.4.0.post2`](https://github.com/chardet/chardet/compare/7.4.0.post1...7.4.0.post2) [Compare Source](https://github.com/chardet/chardet/compare/7.4.0.post1...7.4.0.post2) ### [`v7.4.0.post1`](https://github.com/chardet/chardet/compare/7.3.0...7.4.0.post1) [Compare Source](https://github.com/chardet/chardet/compare/7.3.0...7.4.0.post1) ### [`v7.3.0`](https://github.com/chardet/chardet/releases/tag/7.3.0) [Compare Source](https://github.com/chardet/chardet/compare/7.2.0...7.3.0) #### License - **0BSD license** — the project license has been changed from MIT to [0BSD](https://opensource.org/license/0bsd), a maximally permissive license with no attribution requirement. All prior 7.x releases should also be considered 0BSD licensed as of this release. #### Features - Added `mime_type` field to detection results — identifies file types for both binary (via magic number matching) and text content. Returned in all `detect()`, `detect_all()`, and `UniversalDetector` results. ([#350](https://github.com/chardet/chardet/pull/350)) - New `pipeline/magic.py` module detects 40+ binary file formats including images, audio/video, archives, documents, executables, and fonts. ZIP-based formats (XLSX, DOCX, JAR, APK, EPUB, wheel, OpenDocument) are distinguished by entry filenames. ([#350](https://github.com/chardet/chardet/pull/350)) #### Bug Fixes - Fixed incorrect equivalence between UTF-16-LE and UTF-16-BE in accuracy testing — these are distinct encodings with different byte order, not interchangeable #### Performance - Added 4 new modules to mypyc compilation (orchestrator, confusion, magic, ascii), bringing the total to 11 compiled modules - Capped statistical scoring at 16 KB — bigram models converge quickly, so large files no longer score the full 200 KB. Worst-case detection time dropped from 62ms to 26ms with no accuracy loss. - Replaced `dataclasses.replace()` with direct `DetectionResult` construction on hot paths, eliminating \~354k function calls per full test suite run #### Build - Added riscv64 to the mypyc wheel build matrix — prebuilt wheels are now published for RISC-V Linux alongside existing architectures ([#348](https://github.com/chardet/chardet/pull/348), thanks [@gounthar](https://github.com/gounthar)) ### [`v7.2.0`](https://github.com/chardet/chardet/releases/tag/7.2.0): chardet 7.2.0 [Compare Source](https://github.com/chardet/chardet/compare/7.1.0...7.2.0) #### Features - Added `include_encodings` and `exclude_encodings` parameters to `detect()`, `detect_all()`, and `UniversalDetector` — restrict or exclude specific encodings from the candidate set, with corresponding `-i`/`--include-encodings` and `-x`/`--exclude-encodings` CLI flags ([#343](https://github.com/chardet/chardet/pull/343)) - Added `no_match_encoding` (default `"cp1252"`) and `empty_input_encoding` (default `"utf-8"`) parameters — control which encoding is returned when no candidate survives the pipeline or the input is empty, with corresponding CLI flags ([#343](https://github.com/chardet/chardet/pull/343)) - Added `-l`/`--language` flag to `chardetect` CLI — shows the detected language (ISO 639-1 code and English name) alongside the encoding ([#342](https://github.com/chardet/chardet/pull/342)) #### Fixes - Fixed null-separated ASCII data being misdetected as UTF-16-BE ([#346](https://github.com/chardet/chardet/issues/346), [#347](https://github.com/chardet/chardet/pull/347)) **Full changelog:** <https://chardet.readthedocs.io/en/latest/changelog.html> ### [`v7.1.0`](https://github.com/chardet/chardet/releases/tag/7.1.0): chardet 7.1.0 [Compare Source](https://github.com/chardet/chardet/compare/7.0.1...7.1.0) #### Features - Added PEP 263 encoding declaration detection — `# -*- coding: ... -*-` and `# coding=...` declarations on lines 1–2 of Python source files are now recognized with confidence 0.95 ([#249](https://github.com/chardet/chardet/issues/249)) - Added `chardet.universaldetector` backward-compatibility stub so that `from chardet.universaldetector import UniversalDetector` works with a deprecation warning ([#341](https://github.com/chardet/chardet/issues/341)) #### Fixes - Fixed false UTF-7 detection of ASCII text containing `++` or `+word` patterns ([#332](https://github.com/chardet/chardet/issues/332)) - Fixed 0.5s startup cost on first `detect()` call — model norms are now computed during loading instead of lazily iterating 21M entries ([#333](https://github.com/chardet/chardet/issues/333)) - Fixed undocumented encoding name changes between chardet 5.x and 7.0 — `detect()` now returns chardet 5.x-compatible names by default ([#338](https://github.com/chardet/chardet/issues/338)) - Improved ISO-2022-JP family detection — recognizes ESC sequences for ISO-2022-JP-2004 (JIS X 0213) and ISO-2022-JP-EXT (JIS X 0201 Kana) - Fixed silent truncation of corrupt model data (`iter_unpack` yielded fewer tuples instead of raising) - Fixed incorrect date in LICENSE #### Performance - 5.5x faster first-detect time (\~0.42s → \~0.075s) by computing model norms as a side-product of `load_models()` - \~40% faster model parsing via `struct.iter_unpack` for bulk entry extraction (eliminates \~305K individual `unpack` calls) #### New API parameters - Added `compat_names` parameter (default `True`) to `detect()`, `detect_all()`, and `UniversalDetector` — set to `False` to get raw Python codec names instead of chardet 5.x/6.x compatible display names - Added `prefer_superset` parameter (default `False`) — remaps legacy ISO/subset encodings to their modern Windows/CP superset equivalents (e.g., ASCII → Windows-1252, ISO-8859-1 → Windows-1252). **This will default to `True` in the next major version (8.0).** - Deprecated `should_rename_legacy` in favor of `prefer_superset` — a deprecation warning is emitted when used #### Improvements - Switched internal canonical encoding names to Python codec names (e.g., `"utf-8"` instead of `"UTF-8"`), with `compat_names` controlling the public output format - Added `lookup_encoding()` to `registry` for case-insensitive resolution of arbitrary encoding name input to canonical names - Achieved 100% line coverage across all source modules (+31 tests) - Updated benchmark numbers: 98.2% encoding accuracy, 95.2% language accuracy on 2,510 test files - Pinned test-data cloning to chardet release version tags for reproducible builds **Full changelog:** <https://chardet.readthedocs.io/en/latest/changelog.html> ### [`v7.0.1`](https://github.com/chardet/chardet/releases/tag/7.0.1) [Compare Source](https://github.com/chardet/chardet/compare/7.0.0...7.0.1) #### Fixes - Fixed false UTF-7 detection of SHA-1 git hashes ([#324](https://github.com/chardet/chardet/pull/324), fixing [#323](https://github.com/chardet/chardet/issues/323)) — requirements files with VCS pins (e.g., `+4bafdea3...`) were misdetected as UTF-7, breaking tools like tox - Fixed `_SINGLE_LANG_MAP` missing aliases for single-language encoding lookup (e.g., `big5` → `big5hkscs`) - Fixed PyPy `TypeError` in UTF-7 codec handling #### Improvements - Retrained bigram models — 24 previously failing test cases now pass - Updated language equivalences for mutual intelligibility (Slovak/Czech, East Slavic + Bulgarian, Malay/Indonesian, Scandinavian languages) #### New Contributors - [@rembish](https://github.com/rembish) made their first contribution — both reporting the UTF-7 false detection issue and submitting the fix! ([#323](https://github.com/chardet/chardet/issues/323), [#324](https://github.com/chardet/chardet/pull/324)) ### [`v7.0.0`](https://github.com/chardet/chardet/releases/tag/7.0.0) [Compare Source](https://github.com/chardet/chardet/compare/6.0.0.post1...7.0.0) Ground-up, MIT-licensed rewrite of chardet. Same package name, same public API — drop-in replacement for chardet 5.x/6.x. Just way faster and more accurate! **Highlights:** - **MIT license** (previous versions were LGPL) - **96.8% accuracy** on 2,179 test files (+2.3pp vs chardet 6.0.0, +7.7pp vs charset-normalizer) - **41x faster** than chardet 6.0.0 with mypyc (**28x** pure Python), **7.5x faster** than charset-normalizer - **Language detection** for every result (90.5% accuracy across 49 languages) - **99 encodings** across six eras (MODERN_WEB, LEGACY_ISO, LEGACY_MAC, LEGACY_REGIONAL, DOS, MAINFRAME) - **12-stage detection pipeline** — BOM, UTF-16/32 patterns, escape sequences, binary detection, markup charset, ASCII, UTF-8 validation, byte validity, CJK gating, structural probing, statistical scoring, post-processing - **Bigram frequency models** trained on CulturaX multilingual corpus data for all supported language/encoding pairs - **Optional mypyc compilation** — 1.49x additional speedup on CPython - **Thread-safe** `detect()` and `detect_all()` with no measurable overhead; scales on free-threaded Python 3.13t+ - **Negligible import memory** (96 B) - **Zero runtime dependencies** **Breaking changes vs 6.0.0:** - `detect()` and `detect_all()` now default to `encoding_era=EncodingEra.ALL` (6.0.0 defaulted to `MODERN_WEB`) - Internal architecture is completely different (probers replaced by pipeline stages). Only the public API is preserved. - `LanguageFilter` is accepted but ignored (deprecation warning emitted) - `chunk_size` is accepted but ignored (deprecation warning emitted) ### [`v6.0.0.post1`](https://github.com/chardet/chardet/releases/tag/6.0.0.post1) [Compare Source](https://github.com/chardet/chardet/compare/6.0.0...6.0.0.post1) - Fixed version number in chardet/version.py still being set to `6.0.0dev0`. Otherwise identical to [6.0.0](https://github.com/chardet/chardet/releases/6.0.0). ### [`v6.0.0`](https://github.com/chardet/chardet/releases/tag/6.0.0) [Compare Source](https://github.com/chardet/chardet/compare/5.2.0...6.0.0) ##### Features - **Unified single-byte charset detection**: Instead of only having trained language models for a handful of languages (Bulgarian, Greek, Hebrew, Hungarian, Russian, Thai, Turkish) and relying on special-case `Latin1Prober` and `MacRomanProber` heuristics for Western encodings, chardet now treats all single-byte charsets the same way: every encoding gets proper language-specific bigram models trained on CulturaX corpus data. This means chardet can now accurately detect both the encoding *and* the language for all supported single-byte encodings. - **38 new languages**: Arabic, Belarusian, Breton, Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Farsi, Finnish, French, German, Icelandic, Indonesian, Irish, Italian, Kazakh, Latvian, Lithuanian, Macedonian, Malay, Maltese, Norwegian, Polish, Portuguese, Romanian, Scottish Gaelic, Serbian, Slovak, Slovene, Spanish, Swedish, Tajik, Ukrainian, Vietnamese, and Welsh. Existing models for Bulgarian, Greek, Hebrew, Hungarian, Russian, Thai, and Turkish were also retrained with the new pipeline. - **`EncodingEra` filtering**: New `encoding_era` parameter to `detect` allows filtering by an `EncodingEra` flag enum (`MODERN_WEB`, `LEGACY_ISO`, `LEGACY_MAC`, `LEGACY_REGIONAL`, `DOS`, `MAINFRAME`, `ALL`) allows callers to restrict detection to encodings from a specific era. `detect()` and `detect_all()` default to `MODERN_WEB`. The new `MODERN_WEB` default should drastically improve accuracy for users who are not working with legacy data. The tiers are: - `MODERN_WEB`: UTF-8/16/32, Windows-125x, CP874, CJK multi-byte (widely used on the web) - `LEGACY_ISO`: ISO-8859-x, KOI8-R/U (legacy but well-known standards) - `LEGACY_MAC`: Mac-specific encodings (MacRoman, MacCyrillic, etc.) - `LEGACY_REGIONAL`: Uncommon regional/national encodings (KOI8-T, KZ1048, CP1006, etc.) - `DOS`: DOS/OEM code pages (CP437, CP850, CP866, etc.) - `MAINFRAME`: EBCDIC variants (CP037, CP500, etc.) - **`--encoding-era` CLI flag**: The `chardetect` CLI now accepts `-e`/`--encoding-era` to control which encoding eras are considered during detection. - **`max_bytes` and `chunk_size` parameters**: `detect()`, `detect_all()`, and `UniversalDetector` now accept `max_bytes` (default 200KB) and `chunk_size` (default 64KB) parameters for controlling how much data is examined. ([#314](https://github.com/chardet/chardet/issues/314), [@bysiber](https://github.com/bysiber)) - **Encoding era preference tie-breaking**: When multiple encodings have very close confidence scores, the detector now prefers more modern/Unicode encodings over legacy ones. - **Charset metadata registry**: New `chardet.metadata.charsets` module provides structured metadata about all supported encodings, including their era classification and language filter. - **`should_rename_legacy` now defaults intelligently**: When set to `None` (the new default), legacy renaming is automatically enabled when `encoding_era` is `MODERN_WEB`. - **Direct GB18030 support**: Replaced the redundant GB2312 prober with a proper GB18030 prober. - **EBCDIC detection**: Added CP037 and CP500 EBCDIC model registrations for mainframe encoding detection. - **Binary file detection**: Added basic binary file detection to abort analysis earlier on non-text files. - **Python 3.12, 3.13, and 3.14 support** ([#283](https://github.com/chardet/chardet/issues/283), [@hugovk](https://github.com/hugovk); [#311](https://github.com/chardet/chardet/issues/311)) - **GitHub Codespace support** ([#312](https://github.com/chardet/chardet/issues/312), [@oxygen-dioxide](https://github.com/oxygen-dioxide)) ##### Fixes - **Fix CP949 state machine**: Corrected the state machine for Korean CP949 encoding detection. ([#268](https://github.com/chardet/chardet/issues/268), [@nenw](https://github.com/nenw)) - **Fix SJIS distribution analysis**: Fixed `SJISDistributionAnalysis` discarding valid second-byte range >= 0x80. ([#315](https://github.com/chardet/chardet/issues/315), [@bysiber](https://github.com/bysiber)) - **Fix UTF-16/32 detection for non-ASCII-heavy text**: Improved detection of UTF-16/32 encoded CJK and other non-ASCII text by adding a `MIN_RATIO` threshold alongside the existing `EXPECTED_RATIO`. - **Fix `get_charset` crash**: Resolved a crash when looking up unknown charset names. - **Fix GB18030 `char_len_table`**: Corrected the character length table for GB18030 multi-byte sequences. - **Fix UTF-8 state machine**: Updated to be more spec-compliant. - **Fix `detect_all()` returning inactive probers**: Results from probers that determined "definitely not this encoding" are now excluded. - **Fix early cutoff bug**: Resolved an issue where detection could terminate prematurely. - **Default UTF-8 fallback**: If UTF-8 has not been ruled out and nothing else is above the minimum threshold, UTF-8 is now returned as the default. ##### Breaking changes - **Dropped Python 3.7, 3.8, and 3.9 support**: Now requires Python 3.10+. ([#283](https://github.com/chardet/chardet/issues/283), [@hugovk](https://github.com/hugovk)) - **Removed `Latin1Prober` and `MacRomanProber`**: These special-case probers have been replaced by the unified model-based approach described above. Latin-1, MacRoman, and all other single-byte encodings are now detected by `SingleByteCharSetProber` with trained language models, giving better accuracy and language identification. - **Removed EUC-TW support**: EUC-TW encoding detection has been removed as it is extremely rare in practice. - **`LanguageFilter.NONE` removed**: Use specific language filters or `LanguageFilter.ALL` instead. - **Enum types changed**: `InputState`, `ProbingState`, `MachineState`, `SequenceLikelihood`, and `CharacterCategory` are now `IntEnum` (previously plain classes or `Enum`). `LanguageFilter` values changed from hardcoded hex to `auto()`. - **`detect()` default behavior change**: `detect()` now defaults to `encoding_era=EncodingEra.MODERN_WEB` and `should_rename_legacy=None` (auto-enabled for `MODERN_WEB`), whereas previously it defaulted to considering all encodings with no legacy renaming. ##### Misc changes - **Switched from Poetry/setuptools to uv + hatchling**: Build system modernized with `hatch-vcs` for version management. - **License text updated**: Updated LGPLv2.1 license text and FSF notices to use URL instead of mailing address. ([#304](https://github.com/chardet/chardet/issues/304), [#307](https://github.com/chardet/chardet/issues/307), [@musicinmybrain](https://github.com/musicinmybrain)) - **CulturaX-based model training**: The `create_language_model.py` training script was rewritten to use the CulturaX multilingual corpus instead of Wikipedia, producing higher quality bigram frequency models. - **`Language` class converted to frozen dataclass**: The language metadata class now uses `@dataclass(frozen=True)` with `num_training_docs` and `num_training_chars` fields replacing `wiki_start_pages`. - **Test infrastructure**: Added `pytest-timeout` and `pytest-xdist` for faster parallel test execution. Reorganized test data directories. ##### Contributors Thank you to everyone who contributed to this release! - [@dan-blanchard](https://github.com/dan-blanchard) (Dan Blanchard) - [@bysiber](https://github.com/bysiber) (Kadir Can Ozden) - [@musicinmybrain](https://github.com/musicinmybrain) (Ben Beasley) - [@hugovk](https://github.com/hugovk) (Hugo van Kemenade) - [@oxygen-dioxide](https://github.com/oxygen-dioxide) - [@nenw](https://github.com/nenw) And a special thanks to [@helour](https://github.com/helour), whose earlier Latin-1 prober work from an abandoned PR helped inform the approach taken in this release. ### [`v5.2.0`](https://github.com/chardet/chardet/releases/tag/5.2.0): chardet 5.2.0 [Compare Source](https://github.com/chardet/chardet/compare/5.1.0...5.2.0) Adds support for running chardet CLI via `python -m chardet` ([`0e9b7bc`](https://github.com/chardet/chardet/commit/0e9b7bc20366163efcc221281201baff4100fe19), [@dan-blanchard](https://github.com/dan-blanchard)) ### [`v5.1.0`](https://github.com/chardet/chardet/releases/tag/5.1.0): chardet 5.1.0 [Compare Source](https://github.com/chardet/chardet/compare/5.0.0...5.1.0) ##### Features - Add `should_rename_legacy` argument to most functions, which will rename older encodings to their more modern equivalents (e.g., `GB2312` becomes `GB18030`) ([#264](https://github.com/chardet/chardet/issues/264), [@dan-blanchard](https://github.com/dan-blanchard)) - Add capital letter sharp S and ISO-8859-15 support ([#222](https://github.com/chardet/chardet/issues/222), [@SimonWaldherr](https://github.com/SimonWaldherr)) - Add a prober for MacRoman encoding ([#5](https://github.com/chardet/chardet/issues/5) updated as [`c292b52`](https://github.com/chardet/chardet/commit/c292b52a97e57c95429ef559af36845019b88b33), Rob Speer and [@dan-blanchard](https://github.com/dan-blanchard) ) - Add `--minimal` flag to `chardetect` command ([#214](https://github.com/chardet/chardet/issues/214), [@dan-blanchard](https://github.com/dan-blanchard)) - Add type annotations to the project and run mypy on CI ([#261](https://github.com/chardet/chardet/issues/261), [@jdufresne](https://github.com/jdufresne)) - Add support for Python 3.11 ([#274](https://github.com/chardet/chardet/issues/274), [@hugovk](https://github.com/hugovk)) ##### Fixes - Clarify LGPL version in License trove classifier ([#255](https://github.com/chardet/chardet/issues/255), [@musicinmybrain](https://github.com/musicinmybrain)) - Remove support for EOL Python 3.6 ([#260](https://github.com/chardet/chardet/issues/260), [@jdufresne](https://github.com/jdufresne)) - Remove unnecessary guards for non-falsey values ([#259](https://github.com/chardet/chardet/issues/259), [@jdufresne](https://github.com/jdufresne)) ##### Misc changes - Switch to Python 3.10 release in GitHub actions ([#257](https://github.com/chardet/chardet/issues/257), [@jdufresne](https://github.com/jdufresne)) - Remove setup.py in favor of build package ([#262](https://github.com/chardet/chardet/issues/262), [@jdufresne](https://github.com/jdufresne)) - Run tests on macos, Windows, and 3.11-dev ([#267](https://github.com/chardet/chardet/issues/267), [@dan-blanchard](https://github.com/dan-blanchard)) ### [`v5.0.0`](https://github.com/chardet/chardet/releases/tag/5.0.0): chardet 5.0.0 [Compare Source](https://github.com/chardet/chardet/compare/4.0.0...5.0.0) ⚠️ This release is the first release of chardet that no longer supports Python < 3.6 ⚠️ In addition to that change, it features the following user-facing changes: - Added a prober for Johab Korean ([#207](https://github.com/chardet/chardet/issues/207), [@grizlupo](https://github.com/grizlupo)) - Added a prober for UTF-16/32 BE/LE ([#109](https://github.com/chardet/chardet/issues/109), [#206](https://github.com/chardet/chardet/issues/206), [@jpz](https://github.com/jpz)) - Added test data for Croatian, Czech, Hungarian, Polish, Slovak, Slovene, Greek, and Turkish, which should help prevent future errors with those languages - Improved XML tag filtering, which should improve accuracy for XML files ([#208](https://github.com/chardet/chardet/issues/208)) - Tweaked `SingleByteCharSetProber` confidence to match latest uchardet ([#209](https://github.com/chardet/chardet/issues/209)) - Made `detect_all` return child prober confidences ([#210](https://github.com/chardet/chardet/issues/210)) - Updated examples in docs ([#223](https://github.com/chardet/chardet/issues/223), [@domdfcoding](https://github.com/domdfcoding)) - Documentation fixes ([#212](https://github.com/chardet/chardet/issues/212), [#224](https://github.com/chardet/chardet/issues/224), [#225](https://github.com/chardet/chardet/issues/225), [#226](https://github.com/chardet/chardet/issues/226), [#220](https://github.com/chardet/chardet/issues/220), [#221](https://github.com/chardet/chardet/issues/221), [#244](https://github.com/chardet/chardet/issues/244) from too many to mention) - Minor performance improvements ([#252](https://github.com/chardet/chardet/issues/252), [@deedy5](https://github.com/deedy5)) - Add support for Python 3.10 when testing ([#232](https://github.com/chardet/chardet/issues/232), [@jdufresne](https://github.com/jdufresne)) - Lots of little development cycle improvements, mostly thanks to [@jdufresne](https://github.com/jdufresne) ### [`v4.0.0`](https://github.com/chardet/chardet/releases/tag/4.0.0): chardet 4.0.0 [Compare Source](https://github.com/chardet/chardet/compare/3.0.4...4.0.0) ⚠️ This will be the last release of chardet to support Python 2.7. chardet 5.0 will only support 3.6+ ⚠️ ##### Major Changes This release is multiple years in the making, and provides some quality of life improvements to chardet. The primary user-facing changes are: 1. Single-byte charset probers now use nested dictionaries under the hood, so they are usually a little faster than before. (See [#121](https://github.com/chardet/chardet/issues/121) for details) 2. The `CharsetGroupProber` class now properly short-circuits when one of the probers in the group is considered a definite match. This lead to a substantial speedup. 3. There is now a `chardet.detect_all` function that returns a list of possible encodings for the input with associated confidences. 4. We have dropped support for Python 2.6, 3.4, and 3.5 as they are all past end-of-life. The changes in this release have also laid the groundwork for retraining the models to make them more accurate, and to support some more encodings/languages (see [#99](https://github.com/chardet/chardet/issues/99) for progress). This is our main focus for chardet 5.0 (beyond dropping Python 2 support). ##### Benchmarks Running on a MacBook Pro (15-inch, 2018) with 2.2GHz 6-core i7 processor and 32GB RAM ##### old version (chardet 3.0.4) ``` Benchmarking chardet 3.0.4 on CPython 3.7.5 (default, Sep 8 2020, 12:19:42) [Clang 11.0.3 (clang-1103.0.32.62)] -------------------------------------------------------------------------------- Calls per second for each encoding: ascii: 25559.439366240098 big5: 7.187002209518091 cp932: 4.71090956645177 cp949: 2.937256786994428 euc-jp: 4.870580412090848 euc-kr: 6.6910755971933416 euc-tw: 87.71098043480079 gb2312: 6.614302607154443 ibm855: 27.595893549680685 ibm866: 29.93483661732791 iso-2022-jp: 3379.5052775763434 iso-2022-kr: 26181.67290886392 iso-8859-1: 120.63424740403983 iso-8859-5: 32.65106262196898 iso-8859-7: 62.480089080556084 koi8-r: 13.72481001727257 maccyrillic: 33.018537255804496 shift_jis: 4.996013583677438 tis-620: 14.323112928341818 utf-16: 166771.53081510935 utf-32: 198782.18009478672 utf-8: 13.966236809766901 utf-8-sig: 193732.28637413395 windows-1251: 23.038910006925768 windows-1252: 99.48409117053738 windows-1255: 6.336261495718825 Total time: 357.05358052253723s (10.054513372323958 calls per second) ``` ##### new version (chardet 4.0.0) ``` Benchmarking chardet 4.0.0 on CPython 3.7.5 (default, Sep 8 2020, 12:19:42) [Clang 11.0.3 (clang-1103.0.32.62)] -------------------------------------------------------------------------------- ....................................................................................................................................................................................................................................................................................................................................................................... Calls per second for each encoding: ascii: 38176.31067961165 big5: 12.86915132656389 cp932: 4.656400877065864 cp949: 7.282976434315926 euc-jp: 4.329381447610525 euc-kr: 8.16386823884839 euc-tw: 90.230745070368 gb2312: 14.248865889128146 ibm855: 33.30225548069821 ibm866: 44.181691968506 iso-2022-jp: 3024.2295767539117 iso-2022-kr: 25055.57945041816 iso-8859-1: 59.25262902122995 iso-8859-5: 39.7069713674529 iso-8859-7: 61.008422013862194 koi8-r: 41.21560517643845 maccyrillic: 31.402474369805002 shift_jis: 4.9091652743515155 tis-620: 14.408875278821073 utf-16: 177349.00634249471 utf-32: 186413.51111111112 utf-8: 108.62174360115105 utf-8-sig: 181965.46637744035 windows-1251: 43.16933400329809 windows-1252: 211.27653358317968 windows-1255: 16.15113643694104 Total time: 268.0230791568756s (13.394368915143872 calls per second) ``` Thank you to [@aaaxx](https://github.com/aaaxx), [@edumco](https://github.com/edumco), [@hrnciar](https://github.com/hrnciar), [@hroncok](https://github.com/hroncok), [@jdufresne](https://github.com/jdufresne), [@mdamien](https://github.com/mdamien), [@saintamh](https://github.com/saintamh) , [@xeor](https://github.com/xeor) for submitting pull requests, to all of our users for being patient with how long this release has taken. ##### Full changelog - Convert single-byte charset probers to use nested dicts for language models ([#121](https://github.com/chardet/chardet/issues/121)) [@dan-blanchard](https://github.com/dan-blanchard) - Add API option to get all the encodings confidence ([#111](https://github.com/chardet/chardet/issues/111)) [@mdamien](https://github.com/mdamien) - Make sure pyc files are not in tarballs ([`d7c7343`](https://github.com/chardet/chardet/commit/d7c7343)) [@dan-blanchard](https://github.com/dan-blanchard) - Add benchmark script ([`d702545`](https://github.com/chardet/chardet/commit/d702545), [`8dccd00`](https://github.com/chardet/chardet/commit/8dccd00), [`726973e`](https://github.com/chardet/chardet/commit/726973e), [`71a0fad`](https://github.com/chardet/chardet/commit/71a0fad)) [@dan-blanchard](https://github.com/dan-blanchard) - Include license file in the generated wheel package ([#141](https://github.com/chardet/chardet/issues/141)) [@jdufresne](https://github.com/jdufresne) - Drop support for Python 2.6 ([#143](https://github.com/chardet/chardet/issues/143)) [@jdufresne](https://github.com/jdufresne) - Remove unused coverage configuration ([#142](https://github.com/chardet/chardet/issues/142)) [@jdufresne](https://github.com/jdufresne) - Doc the chardet package suitable for production ([#144](https://github.com/chardet/chardet/issues/144)) [@jdufresne](https://github.com/jdufresne) - Pass python_requires argument to setuptools ([#150](https://github.com/chardet/chardet/issues/150)) [@jdufresne](https://github.com/jdufresne) - Update pypi.python.org URL to pypi.org ([#155](https://github.com/chardet/chardet/issues/155)) [@jdufresne](https://github.com/jdufresne) - Typo fix ([#159](https://github.com/chardet/chardet/issues/159)) [@saintamh](https://github.com/saintamh) - Support pytest 4, don't apply marks directly to parameters (PR [#174](https://github.com/chardet/chardet/issues/174), Issue [#173](https://github.com/chardet/chardet/issues/173)) [@hroncok](https://github.com/hroncok) - Test Python 3.7 and 3.8 and document support ([#175](https://github.com/chardet/chardet/issues/175)) [@jdufresne](https://github.com/jdufresne) - Drop support for end-of-life Python 3.4 ([#181](https://github.com/chardet/chardet/issues/181)) [@jdufresne](https://github.com/jdufresne) - Workaround for distutils bug in python 2.7 ([#165](https://github.com/chardet/chardet/issues/165)) [@xeor](https://github.com/xeor) - Remove deprecated license_file from setup.cfg ([#182](https://github.com/chardet/chardet/issues/182)) [@jdufresne](https://github.com/jdufresne) - Remove deprecated 'sudo: false' from Travis configuraiton ([#200](https://github.com/chardet/chardet/issues/200)) [@jdufresne](https://github.com/jdufresne) - Add testing for Python 3.9 ([#201](https://github.com/chardet/chardet/issues/201)) [@jdufresne](https://github.com/jdufresne) - Adds explicit os and distro definitions ([#140](https://github.com/chardet/chardet/issues/140)) [@edumco](https://github.com/edumco) - Remove shebang from nonexecutable script ([#192](https://github.com/chardet/chardet/issues/192)) [@hrnciar](https://github.com/hrnciar) - Remove use of deprecated 'setup.py test' ([#187](https://github.com/chardet/chardet/issues/187)) [@jdufresne](https://github.com/jdufresne) - Remove unnecessary numeric placeholders from format strings ([#176](https://github.com/chardet/chardet/issues/176)) [@jdufresne](https://github.com/jdufresne) - Update links ([#152](https://github.com/chardet/chardet/issues/152)) [@aaaxx](https://github.com/aaaxx) - Remove shebang and executable bit from chardet/cli/chardetect.py ([#171](https://github.com/chardet/chardet/issues/171)) [@jdufresne](https://github.com/jdufresne) - Handle weird logging edge case in universaldetector.py ([`056a2a4`](https://github.com/chardet/chardet/commit/056a2a4)) [@dan-blanchard](https://github.com/dan-blanchard) - Switch from Travis to GitHub Actions ([#204](https://github.com/chardet/chardet/issues/204)) [@dan-blanchard](https://github.com/dan-blanchard) - Properly set CharsetGroupProber.state to FOUND_IT (PR [#203](https://github.com/chardet/chardet/issues/203), Issue [#202](https://github.com/chardet/chardet/issues/202)) [@dan-blanchard](https://github.com/dan-blanchard) - Add language to detect_all output ([`1e208b7`](https://github.com/chardet/chardet/commit/1e208b7)) [@dan-blanchard](https://github.com/dan-blanchard) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).

renovate added 1 commit

2026-10-27 00:26:29 +00:00

Update dependency chardet to v7 69bd85da6e

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin "renovate/chardet-7.x:renovate/chardet-7.x"

git switch "renovate/chardet-7.x"

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch "master"

git merge --no-ff "renovate/chardet-7.x"

git switch "renovate/chardet-7.x"

git rebase "master"

git switch "master"

git merge --ff-only "renovate/chardet-7.x"

git switch "renovate/chardet-7.x"

git rebase "master"

git switch "master"

git merge --no-ff "renovate/chardet-7.x"

git switch "master"

git merge --squash "renovate/chardet-7.x"

git switch "master"

git merge --ff-only "renovate/chardet-7.x"

git switch "master"

git merge "renovate/chardet-7.x"

git push origin "master"

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

andylockran/parkrun-hackapi!17

No description provided.

Rows
Columns

Update dependency chardet to v7 #17

Release Notes

Performance

Bug Fixes

Improvements

Performance

Bug Fixes

Improvements

Bug Fixes

Bug Fixes

Performance

Improvements

Bug Fixes

Bug Fixes

Improvements

Bug Fixes

License

Features

Bug Fixes

Performance

Build

v7.2.0: chardet 7.2.0

Features

Fixes

v7.1.0: chardet 7.1.0

Features

Fixes

Performance

New API parameters

Improvements

Fixes

Improvements

New Contributors

Features

Fixes

Breaking changes

Misc changes

Contributors

v5.2.0: chardet 5.2.0

v5.1.0: chardet 5.1.0

Features

Fixes

Misc changes

v5.0.0: chardet 5.0.0

v4.0.0: chardet 4.0.0

Major Changes

Benchmarks

old version (chardet 3.0.4)

new version (chardet 4.0.0)

Full changelog

Configuration

Checkout

Merge

`v7.2.0`: chardet 7.2.0

`v7.1.0`: chardet 7.1.0

`v5.2.0`: chardet 5.2.0

`v5.1.0`: chardet 5.1.0

`v5.0.0`: chardet 5.0.0

`v4.0.0`: chardet 4.0.0