v4.2.6: Self-describing format, byte pipeline, and a performance pass

Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
This commit is contained in:
dvs
2026-09-27 22:06:34 +08:00
parent 24a36d6fb4
commit dd49043171
25 changed files with 5348 additions and 852 deletions
+122
View File
@@ -21,6 +21,12 @@ from chaoscrypt import ChaosCryptError # base error class
| `decrypt` | function | One-shot decrypt using a key file |
| `create_key` | function | One-shot key file generation |
| `DEFAULT_KEY_FILE` | constant | `"encryption.key"` |
| `MODES` | constant | Tuple of the four mode names |
| `DEFAULT_MODE` | constant | `"fast"` |
| `MODE_REFERENCE` | constant | `"reference"` |
| `MODE_FAST` | constant | `"fast"` |
| `MODE_TURBO` | constant | `"turbo"` |
| `MODE_RAW` | constant | `"raw"` |
| `KEYFILE_VERSION` | constant | On-disk key file format version (`1`) |
| `__version__` | constant | Library version string |
| `__license__` | constant | `"MIT"` |
@@ -50,6 +56,9 @@ ChaosCrypt(
| `key_password` | `str \| None` | `None` | Password protecting that key file |
| `auto_load` | `bool` | `False` | Load the key file during construction |
| `default_key_file` | `str` | `"encryption.key"` | File name used when `key_file` is omitted |
| `mode` | `str` | `"fast"` | Performance mode: `reference`, `fast`, `turbo` or `raw` |
| `encoding` | `str` | `"hex"` | Output encoding for `turbo`/`raw`: `hex` or `base64` (also accepted as `base`) |
| `mid` | `str` | `"bt"` | Intermediate representation for `turbo`/`raw`: `bt` or `hex` (also accepted as `bytes`) |
When `auto_load=True` and the file cannot be loaded, the corresponding
exception is raised from the constructor.
@@ -292,6 +301,117 @@ compatibility guarantee.
---
## Performance modes
Four modes, plus two settings that apply to the two byte-pipeline modes.
| Constant | Value | Pipeline | Ciphertext |
|---|---|---|---|
| `MODE_REFERENCE` | `"reference"` | original algorithm, character pipeline | hex, 5.33x |
| `MODE_FAST` | `"fast"` | same algorithm, bulk operations | hex, 5.33x |
| `MODE_TURBO` | `"turbo"` | SHAKE256 keystream, byte pipeline | 2.00x / 1.33x |
| `MODE_RAW` | `"raw"` | SHAKE256 plus the long-key layer | 2.00x / 1.33x |
```python
from chaoscrypt import ChaosCrypt, MODES, DEFAULT_MODE
print(MODES) # ('reference', 'fast', 'turbo', 'raw')
print(DEFAULT_MODE) # 'fast'
```
### Settings for the byte-pipeline modes
| Constant | Value | Applies to | Meaning |
|---|---|---|---|
| `ENC_HEX` | `"hex"` | `turbo`, `raw` | Two characters per octet (2.00x) |
| `ENC_BASE64` | `"base64"` | `turbo`, `raw` | Four characters per three octets (1.33x) |
| `MID_BYTES` | `"bt"` | `turbo`, `raw` | XOR layers exchange raw bytes |
| `MID_HEX` | `"hex"` | `turbo`, `raw` | XOR layers exchange hexadecimal text |
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="raw", encoding="base64", mid="bt")
cc.load("my.key", "key-password")
token = cc.encrypt("hello", "user-password")
```
```python
from chaoscrypt import ChaosCryptCore, MODE_TURBO, ENC_BASE64, MID_HEX
core = ChaosCryptCore(MODE_TURBO, "hash", ENC_BASE64, MID_HEX)
core.generate_keymaterial()
token = core.encrypt_text("hello", "password")
assert core.decrypt_text(token, "password") == "hello"
```
### Which mode to choose
| If you… | Use |
|---|---|
| Are not sure | `fast` — the default |
| Need maximum speed from a standardised keystream primitive | `raw` with `mid="bt"` |
| Need the smallest ciphertext | `turbo` or `raw` with `encoding="base64"` |
| Need to interoperate with the character pipeline's shape | `mid="hex"` |
| Need a payload byte-compatible with pre-4.2.6 material | `reference` or `fast` |
`fast` performs the same arithmetic as `reference` and produces the same bytes;
only the implementation differs. `turbo` and `raw` change the algorithm and
define their own formats.
> **Artefacts are not interchangeable between mode families**, and `turbo`/`raw`
> artefacts are also specific to their `encoding` and `mid`. The header records
> all three, so a reader configured differently still reads the artefact
> correctly — the ordinary decrypt takes its parameters from the header.
An unrecognised `mode`, `encoding` or `mid` raises `ValueError` at construction.
## Reading pre-4.2.6 artefacts
Artefacts written before the header existed have none, and are rejected by the
ordinary path with `E017`. They are read through the compatibility API, which
synthesises a minimal header and then uses the ordinary path.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="kf-pwd", auto_load=True)
# a ciphertext with no header
plain = cc.decrypt_legacy(old_ciphertext, "user-password")
# if you know it is a legacy key file
cc.load_legacy_keys(old_key_text, "key-password")
# let the library work out which it is
plain = cc.decrypt_legacy_auto(old_ciphertext, "user-password")
# migrate once, then use the modern path
open("new.key", "w", encoding="utf-8").write(
cc.wrap_legacy(old_key_text, is_key=True)
)
```
| Method | Purpose |
|---|---|
| `wrap_legacy(text, is_key=False)` | Prefix a minimal header; the payload is untouched. Idempotent. |
| `decrypt_legacy(text, password, is_key=False)` | Read an artefact that may predate the header |
| `decrypt_legacy_auto(text, password)` | As above, probing whether the input is a key file or a ciphertext |
| `load_legacy_keys(text, key_password)` | Load a pre-header key file |
`is_old` in the synthesised header means every other field is ignored; the
legacy path tries `fast` and falls back to `reference`. Both implement the
original algorithm.
> The compatibility API is a **wrapper** around the ordinary path, never a
> lenient branch inside it. Passing headerless input to `decrypt` still raises
> `E017`.
See [FORMAT.md §12](FORMAT.md) for the legacy standard.
---
## Error hierarchy
Every error is a subclass of `ChaosCryptError`, so a single `except` clause
@@ -326,6 +446,8 @@ except ChaosCryptError as e:
| `E014` | `ChaosCrypt_ERR_FileReadFailed_E014` | Reading a file failed |
| `E015` | `ChaosCrypt_ERR_FileWriteFailed_E015` | Writing a file failed |
| `E016` | `ChaosCrypt_ERR_UnsupportedKeyVersion_E016` | Key file version not supported |
| `E017` | `ChaosCrypt_ERR_MissingMagicHeader_E017` | Input is not a CES artefact |
| `E018` | `ChaosCrypt_ERR_MalformedHeader_E018` | Header unusable, or `kind` mismatch |
Codes are stable and safe to branch on. The human-readable portion of the
message may be reworded between minor releases.