v4.2.6: Self-describing format, byte pipeline, and a performance pass

Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
This commit is contained in:
dvs
2026-09-27 22:06:34 +08:00
parent 24a36d6fb4
commit dd49043171
25 changed files with 5348 additions and 852 deletions
+122
View File
@@ -21,6 +21,12 @@ from chaoscrypt import ChaosCryptError # base error class
| `decrypt` | function | One-shot decrypt using a key file |
| `create_key` | function | One-shot key file generation |
| `DEFAULT_KEY_FILE` | constant | `"encryption.key"` |
| `MODES` | constant | Tuple of the four mode names |
| `DEFAULT_MODE` | constant | `"fast"` |
| `MODE_REFERENCE` | constant | `"reference"` |
| `MODE_FAST` | constant | `"fast"` |
| `MODE_TURBO` | constant | `"turbo"` |
| `MODE_RAW` | constant | `"raw"` |
| `KEYFILE_VERSION` | constant | On-disk key file format version (`1`) |
| `__version__` | constant | Library version string |
| `__license__` | constant | `"MIT"` |
@@ -50,6 +56,9 @@ ChaosCrypt(
| `key_password` | `str \| None` | `None` | Password protecting that key file |
| `auto_load` | `bool` | `False` | Load the key file during construction |
| `default_key_file` | `str` | `"encryption.key"` | File name used when `key_file` is omitted |
| `mode` | `str` | `"fast"` | Performance mode: `reference`, `fast`, `turbo` or `raw` |
| `encoding` | `str` | `"hex"` | Output encoding for `turbo`/`raw`: `hex` or `base64` (also accepted as `base`) |
| `mid` | `str` | `"bt"` | Intermediate representation for `turbo`/`raw`: `bt` or `hex` (also accepted as `bytes`) |
When `auto_load=True` and the file cannot be loaded, the corresponding
exception is raised from the constructor.
@@ -292,6 +301,117 @@ compatibility guarantee.
---
## Performance modes
Four modes, plus two settings that apply to the two byte-pipeline modes.
| Constant | Value | Pipeline | Ciphertext |
|---|---|---|---|
| `MODE_REFERENCE` | `"reference"` | original algorithm, character pipeline | hex, 5.33x |
| `MODE_FAST` | `"fast"` | same algorithm, bulk operations | hex, 5.33x |
| `MODE_TURBO` | `"turbo"` | SHAKE256 keystream, byte pipeline | 2.00x / 1.33x |
| `MODE_RAW` | `"raw"` | SHAKE256 plus the long-key layer | 2.00x / 1.33x |
```python
from chaoscrypt import ChaosCrypt, MODES, DEFAULT_MODE
print(MODES) # ('reference', 'fast', 'turbo', 'raw')
print(DEFAULT_MODE) # 'fast'
```
### Settings for the byte-pipeline modes
| Constant | Value | Applies to | Meaning |
|---|---|---|---|
| `ENC_HEX` | `"hex"` | `turbo`, `raw` | Two characters per octet (2.00x) |
| `ENC_BASE64` | `"base64"` | `turbo`, `raw` | Four characters per three octets (1.33x) |
| `MID_BYTES` | `"bt"` | `turbo`, `raw` | XOR layers exchange raw bytes |
| `MID_HEX` | `"hex"` | `turbo`, `raw` | XOR layers exchange hexadecimal text |
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="raw", encoding="base64", mid="bt")
cc.load("my.key", "key-password")
token = cc.encrypt("hello", "user-password")
```
```python
from chaoscrypt import ChaosCryptCore, MODE_TURBO, ENC_BASE64, MID_HEX
core = ChaosCryptCore(MODE_TURBO, "hash", ENC_BASE64, MID_HEX)
core.generate_keymaterial()
token = core.encrypt_text("hello", "password")
assert core.decrypt_text(token, "password") == "hello"
```
### Which mode to choose
| If you… | Use |
|---|---|
| Are not sure | `fast` — the default |
| Need maximum speed from a standardised keystream primitive | `raw` with `mid="bt"` |
| Need the smallest ciphertext | `turbo` or `raw` with `encoding="base64"` |
| Need to interoperate with the character pipeline's shape | `mid="hex"` |
| Need a payload byte-compatible with pre-4.2.6 material | `reference` or `fast` |
`fast` performs the same arithmetic as `reference` and produces the same bytes;
only the implementation differs. `turbo` and `raw` change the algorithm and
define their own formats.
> **Artefacts are not interchangeable between mode families**, and `turbo`/`raw`
> artefacts are also specific to their `encoding` and `mid`. The header records
> all three, so a reader configured differently still reads the artefact
> correctly — the ordinary decrypt takes its parameters from the header.
An unrecognised `mode`, `encoding` or `mid` raises `ValueError` at construction.
## Reading pre-4.2.6 artefacts
Artefacts written before the header existed have none, and are rejected by the
ordinary path with `E017`. They are read through the compatibility API, which
synthesises a minimal header and then uses the ordinary path.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="kf-pwd", auto_load=True)
# a ciphertext with no header
plain = cc.decrypt_legacy(old_ciphertext, "user-password")
# if you know it is a legacy key file
cc.load_legacy_keys(old_key_text, "key-password")
# let the library work out which it is
plain = cc.decrypt_legacy_auto(old_ciphertext, "user-password")
# migrate once, then use the modern path
open("new.key", "w", encoding="utf-8").write(
cc.wrap_legacy(old_key_text, is_key=True)
)
```
| Method | Purpose |
|---|---|
| `wrap_legacy(text, is_key=False)` | Prefix a minimal header; the payload is untouched. Idempotent. |
| `decrypt_legacy(text, password, is_key=False)` | Read an artefact that may predate the header |
| `decrypt_legacy_auto(text, password)` | As above, probing whether the input is a key file or a ciphertext |
| `load_legacy_keys(text, key_password)` | Load a pre-header key file |
`is_old` in the synthesised header means every other field is ignored; the
legacy path tries `fast` and falls back to `reference`. Both implement the
original algorithm.
> The compatibility API is a **wrapper** around the ordinary path, never a
> lenient branch inside it. Passing headerless input to `decrypt` still raises
> `E017`.
See [FORMAT.md §12](FORMAT.md) for the legacy standard.
---
## Error hierarchy
Every error is a subclass of `ChaosCryptError`, so a single `except` clause
@@ -326,6 +446,8 @@ except ChaosCryptError as e:
| `E014` | `ChaosCrypt_ERR_FileReadFailed_E014` | Reading a file failed |
| `E015` | `ChaosCrypt_ERR_FileWriteFailed_E015` | Writing a file failed |
| `E016` | `ChaosCrypt_ERR_UnsupportedKeyVersion_E016` | Key file version not supported |
| `E017` | `ChaosCrypt_ERR_MissingMagicHeader_E017` | Input is not a CES artefact |
| `E018` | `ChaosCrypt_ERR_MalformedHeader_E018` | Header unusable, or `kind` mismatch |
Codes are stable and safe to branch on. The human-readable portion of the
message may be reworded between minor releases.
+618
View File
@@ -0,0 +1,618 @@
# CES Artifact Format Specification
**Normative reference for the ChaosCrypt (CES) container format**
| | |
|---|---|
| Format version | 1 |
| Status | Normative |
| Applies to | Key files and ciphertexts produced from 4.2.6 onward |
| Library | `chaoscrypt` |
---
## 1. Scope
This document specifies the byte-level layout of every artefact the library
produces. It is the authority for implementers: where this document and the
code disagree, this document is wrong and the code is to be fixed.
It defines:
- the fixed magic and delimiter values;
- the layout of the self-describing header;
- the complete field set of the configuration record;
- the payload that follows;
- the conformance requirements a reader must enforce.
It does **not** describe the cipher itself. For the algorithm see
[SECURITY.md](../SECURITY.md); for performance see [benchmark.md](benchmark.md).
---
## 2. Design rationale
Every artefact carries a header that states how it was produced. The reader
never infers a parameter from context and never guesses.
Earlier revisions left the mode and encoding to the caller's memory. That is a
silent-corruption hazard: a ciphertext decrypted under the wrong mode yields
garbage that is indistinguishable from a wrong password. Recording the
parameters alongside the artefact removes the ambiguity, and rejecting
headerless input removes the guesswork entirely.
The consequence is deliberate and absolute: **input without a valid header is
rejected, not interpreted.**
---
## 3. Fixed values
These byte sequences are constants of the format. They are reproduced in three
representations for convenience; all three denote the same octets.
### 3.1 Magic
| Representation | Value |
|---|---|
| Bytes | `b"69CES"` |
| Hex | `3639434553` |
| Decimal byte list | `[54, 57, 67, 69, 83]` |
Length: 5 octets. Position: the first octet of every artefact.
### 3.2 Begin marker
```
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
64 characters of lowercase hexadecimal. Written verbatim; it is **not** decoded
before comparison.
### 3.3 End marker
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
```
64 characters of lowercase hexadecimal. Written verbatim.
> **Note.** Both markers are fixed constants of format version 1. They are
> delimiters, not integrity checks: they carry no signature semantics and do
> not authenticate the payload. Message integrity, where required, is provided
> by the cipher and by an external MAC.
---
## 4. Container layout
An artefact is a single character string with five regions:
```
+-------------------------------------------------------------------+
| MAGIC "69CES" 5 octets |
+-------------------------------------------------------------------+
| BEGIN MARKER 64 hex chars |
+-------------------------------------------------------------------+
| CONFIGURATION UTF-8 JSON object variable |
+-------------------------------------------------------------------+
| END MARKER 64 hex chars |
+-------------------------------------------------------------------+
| PAYLOAD key material or ciphertext variable |
+-------------------------------------------------------------------+
```
The three fixed regions are literals. Only the configuration and the payload
vary between artefacts.
### 4.1 Formal grammar
```
artifact = MAGIC begin-marker config end-marker payload
MAGIC = "69CES"
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
config = <UTF-8 encoded JSON object, see section 5>
payload = <encoding-dependent text, see section 6>
```
There is no whitespace, newline or padding between regions. A reader must not
skip optional characters.
---
## 5. Configuration record
The configuration is a JSON object encoded as UTF-8, with no line breaks.
Keys are sorted; separators are `,` and `:` with no surrounding spaces. The
canonical form is therefore deterministic: the same configuration always
renders as the same octets.
### 5.1 Required fields
A configuration missing any of these is invalid, and the artefact must be
rejected rather than partially interpreted.
| Field | Type | Meaning |
|---|---|---|
| `kind` | string | `"ciphertext"` or `"key"` |
| `v` | integer | Format version; `1` for this specification |
| `mode` | string | Performance mode, see §5.2 |
| `mid` | string | Intermediate representation, see §5.3 |
| `out` | string | Output encoding, see §5.4 |
| `alg` | string | Algorithm family; `"chaoscrypt"` |
| `kdf` | string | Keystream derivation, see §5.5 |
| `keybits` | integer | Length of the long key in bits |
| `ksum` | string | Non-secret key digest, see §5.6 |
### 5.2 `mode`
| Value | Pipeline |
|---|---|
| `reference` | Original algorithm, character pipeline |
| `fast` | Original algorithm, bulk-operation implementation |
| `turbo` | SHAKE256 keystream, byte pipeline |
| `raw` | SHAKE256 keystream, byte pipeline with the long-key layer retained |
`reference` and `fast` produce identical ciphertexts. The distinction is
recorded so a reader can report which implementation wrote the artefact; it
does not affect decryption.
### 5.3 `mid` — intermediate representation
The `mid` field names how the two XOR layers of the byte pipeline exchange
their data. It is only meaningful for `turbo` and `raw`; for the character
pipelines it is descriptive.
| Value | Meaning |
|---|---|
| `bt` | Intermediate values carried as raw bytes (default) |
| `hex` | Intermediate values carried as hexadecimal text |
`turbo` and `raw` accept either value, and `bt` is the default. The choice is
recorded in the header and honoured when the artefact is read: a `turbo` or
`raw` artefact is decrypted with the `mid` it carries, and no fallback to the
other representation is attempted.
`reference` and `fast` are character pipelines and always use `hex`. They
ignore `mid`: whatever value is supplied, the header records `hex` for them.
**Expansion.** `bt` passes the bytes between the XOR layers unchanged, so the
only growth is the output encoding: 2.00x with `hex` output, 1.33x with
`base` output. `hex` re-encodes each layer's output as hexadecimal text
before the next layer consumes it; with the two layers of this format that
doubles the payload at each layer, so the byte payload handed to the output
encoder is about **4x** the `bt` size (roughly 8 hexadecimal characters per
input octet with `hex` output).
**Speed.** `bt` is about **4x faster** than `hex`: it skips the per-layer hex
encode and decode and XORs the shorter byte buffers directly. It is therefore
the default.
### 5.4 `out` — output encoding
| Value | Meaning | Expansion |
|---|---|---|
| `hex` | Two characters per octet | 2.00x |
| `base` | Base64, four characters per three octets | 1.33x |
`reference` and `fast` always emit `hex` and ignore any attempt to set it.
`turbo` and `raw` accept either.
> **Reader requirement.** The payload must be decoded according to `out`.
> Attempting to decode with the wrong encoding must fail, never silently produce
> a different result.
### 5.5 `kdf` — keystream derivation
| Value | Meaning |
|---|---|
| `expand` | Hand-rolled mixing and rotation expansion |
| `shake256` | SHAKE256 extensible-output function |
### 5.6 `ksum` — key digest
A 16-character lowercase hexadecimal string: the first 16 characters of
`SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key)`, where `‖` is
concatenation of the ASCII encodings.
It exists so a reader can confirm that a key file and a ciphertext belong
together without exposing any usable secret. It is a **correlation aid, not a
security control**: it reveals nothing that helps an attacker, and its absence
of secrecy must not be relied upon as an integrity guarantee.
### 5.7 Legacy fields
Two optional fields exist for reading artefacts produced before the header was
introduced. They are never written by the current library except when wrapping a
legacy artefact, and they are **not** part of :data:`REQUIRED_FIELDS`.
| Field | Type | Meaning |
|---|---|---|
| `is_old` | boolean | The artefact predates this format |
| `old_is_key` | boolean | With `is_old`, whether it is a key file (`true`) or a ciphertext (`false`) |
> **Precedence rule.** When `is_old` is present and true, **every other field is
> ignored**, including `mode`, `mid`, `out` and `kind`. The artefact predates the
> field set, so there is nothing else meaningful to read. A reader must not
> reject such an artefact for missing `kind`, nor act on any field beside
> `is_old` and `old_is_key`.
An artefact that carries neither field is a normal artefact and is read
normally. Their absence is not an error.
### 5.8 Descriptive fields
The record also carries fields that describe the construction. They are
informational — a reader may use them for display or diagnostics — and they do
not modify the decryption path.
| Field | Meaning |
|---|---|
| `lib` | Library version that wrote the artefact |
| `cache` | Keystream cache strategy in effect; does not affect output |
| `subst` | Substitution kind, `"bijective"` |
| `flip` | Case-flip kind, `"conditional"` |
| `order` | The pipeline, as an ordered list of stage names |
| `xor_layers` | Number of XOR layers applied |
| `compact` | Whether the byte pipeline was used |
| `keyshort` | Length of the short key in bits |
| `flipbits` | Number of bits in the flip pattern |
| `b64` | Whether Base64 is used as a byte-safety layer |
| `utf8` | Whether text is transported as UTF-8 |
### 5.9 Example
```json
{
"alg": "chaoscrypt",
"b64": true,
"cache": "hash",
"compact": true,
"flip": "conditional",
"flipbits": 10,
"kdf": "shake256",
"keybits": 16384,
"keyshort": 2048,
"kind": "ciphertext",
"ksum": "3f9a1c7e5b2d4086",
"lib": "4.2.6",
"mid": "bt",
"mode": "turbo",
"order": "base64>subst>flip>reverse>xor>xor",
"out": "hex",
"subst": "bijective",
"utf8": true,
"v": 1,
"xor_layers": 2
}
```
Rendered between the markers, this is a single unbroken line.
---
## 6. Payload
### 6.1 Ciphertext payload
When `kind` is `"ciphertext"`, the payload is the ciphertext encoded per `out`:
- `out = "hex"`: lowercase hexadecimal, even length, alphabet `[0-9a-f]`.
- `out = "base"`: standard Base64 with `=` padding, alphabet
`[A-Za-z0-9+/]`.
The payload is the final output of the pipeline named in `mode`. Nothing is
appended after it.
### 6.2 Key payload
When `kind` is `"key"`, the payload is the password-wrapped key material, in the
same two encodings.
A caller reading a key file must check that `kind` is `"key"`. A key file is not
a ciphertext and must not be accepted where a ciphertext is expected, nor the
reverse. This is enforced, not advisory.
---
## 7. Reader conformance
A conforming reader performs these steps, in order, and fails at the first
that does not hold. It must not attempt recovery, interpretation or fallback.
| # | Check | Failure |
|---|---|---|
| 1 | Input is non-empty | `E017` |
| 2 | Input starts with `69CES` | `E017` |
| 3 | The 64 characters after the magic equal the begin marker | `E018` |
| 4 | The end marker occurs after the begin marker | `E018` |
| 5 | Text between the markers is valid UTF-8 JSON | `E018` |
| 6 | That JSON is an object | `E018` |
| 7 | Every required field of §5.1 is present | `E018` |
| 8 | `kind`, `mid` and `out` hold permitted values | `E018` |
| 9 | `kind` matches what the caller asked for | `E018` |
| 10 | The payload decodes under `out` | `E010` |
| 11 | Decryption succeeds under `mode` and the key | `E011` |
### 7.1 No fallback
There is no lenient path. An artefact without a header is not a ChaosCrypt
artefact and is rejected with `E017`. Artefacts written by revisions before the
header existed are not read by this path either: they are recovered only
through the explicit compatibility API, which wraps them in a header first
(see section 12). The ordinary reader never gains a lenient branch.
### 7.2 No partial trust
A header is either wholly valid or wholly rejected. A reader must not act on
fields it did manage to parse, because a partially parsed header may describe a
different construction than the one that produced the payload.
---
## 8. Error codes
| Code | Name | Raised when |
|---|---|---|
| `E010` | `MalformedCiphertext` | Payload does not decode under `out` |
| `E011` | `DecryptionFailed` | Decryption produced invalid data |
| `E017` | `MissingMagicHeader` | No magic present; input is not a CES artefact |
| `E018` | `MalformedHeader` | Marker missing, JSON invalid, field absent or out of range, or `kind` mismatch |
Codes `E001`–`E016` cover key handling, argument and I/O failures and are
unchanged.
---
## 9. Versioning
`v` is the format version and is independent of the library version in `lib`.
A reader must reject an artefact whose `v` it does not implement, with `E018`.
Adding optional descriptive fields does not change `v`. Changing the magic, the
markers, the region order, or the meaning of a required field does, and requires
a new major library release.
---
## 10. Conformance checklist
An implementation claims conformance when it:
- [ ] writes the magic, both markers and the configuration in the order of §4;
- [ ] sorts configuration keys and uses `,` and `:` as separators with no spaces;
- [ ] includes every required field of §5.1;
- [ ] rejects input without the magic with `E017`;
- [ ] rejects malformed headers with `E018`, never partially trusting them;
- [ ] enforces the `kind` match on both read and write paths;
- [ ] decodes the payload strictly under `out`, failing rather than guessing;
- [ ] implements no lenient or legacy fallback path.
---
---
## 11. Worked examples
### 11.1 A normal ciphertext
Text that is encrypted in `turbo` with `base64` output, then the layout read
back region by region.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="turbo", encoding="base64")
cc.load("my.key", "key-file-password")
token = cc.encrypt("meet me at the usual place", "encryption-password")
```
The token begins with the five magic octets, then the begin marker:
```
69CES
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
then the configuration, which states the parameters actually used:
```json
{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
"flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
"keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
"lib":"4.2.6","mid":"bt","mode":"turbo",
"order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
"utf8":true,"v":1,"xor_layers":2}
```
then the end marker, then the base64 payload:
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...
```
Decrypting needs no configuration at all — the record supplies it:
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt._header import read_header, strip_header
cfg = read_header(token)
print(cfg["mode"], cfg["out"]) # turbo base
# a fresh instance with no settings reads it correctly
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True).decrypt(token, "encryption-password")
```
### 11.2 Reading an artefact without knowing anything about it
Because the header is complete, a reader needs only the key file and the
password. The mode, the intermediate representation and the output encoding are
all taken from the artefact.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
cc.load("my.key", "key-file-password")
# whatever produced this, it will be read correctly
plain = cc.decrypt(open("message.ces").read(), "encryption-password")
```
### 11.3 Inspecting a key file
```python
from chaoscrypt._header import read_header, strip_header
text = open("my.key", encoding="utf-8").read()
cfg = read_header(text)
print(cfg["kind"]) # key
print(cfg["alg"]) # chaoscrypt
print(cfg["keybits"]) # 16384
print(cfg["ksum"]) # 16 hex characters
```
`ksum` lets a caller confirm that a key file and a ciphertext belong together
without exposing anything usable:
```python
cs = open("message.ces", encoding="utf-8").read()
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"
```
### 11.4 A rejection
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt.errors import ChaosCryptError
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
try:
cc.decrypt("7a3f9c...", "encryption-password") # a bare payload
except ChaosCryptError as e:
print(e.code) # E017
print(e.detail) # MissingMagicHeader
```
There is no fallback path here. The input is not a CES artefact, and guessing
would risk returning garbage that looks like success.
---
## 12. Legacy artefacts
Artefacts written before this format existed have no header. They are handled by
**wrapping**, never by loosening the reader.
### 12.1 What a legacy artefact looks like
| Kind | Content |
|---|---|
| Legacy ciphertext | Hexadecimal text, no prefix of any kind |
| Legacy key file | Password-wrapped key material, no prefix of any kind |
Neither carries a magic value, a marker or a configuration record.
### 12.2 Wrapping
To read one, a minimal header is synthesised and prefixed. The payload is not
touched.
```json
{"is_old":true,"old_is_key":false}
```
That header is a normal, valid header. Wrapping therefore produces an artefact
that the ordinary path can read, and **the ordinary path never gains a lenient
branch**: it still rejects headerless input with `E017`.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
# a legacy ciphertext: no header
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
"encryption-password")
```
`decrypt_legacy` first tries the input as it stands. If that reports a missing
header, it wraps the input and tries again. `decrypt_legacy_auto` additionally
probes whether the artefact is a key file or a ciphertext, so the caller does
not have to say which.
A legacy key file is loaded the same way:
```python
legacy_text = open("old-encryption.key", encoding="utf-8").read()
cc = ChaosCrypt()
cc.load_legacy_keys(legacy_text, "key-file-password")
```
Or wrapped explicitly, when the artefact is to be stored in the new form:
```python
modernised = cc.wrap_legacy(legacy_text, is_key=True)
open("my.key", "w", encoding="utf-8").write(modernised)
```
### 12.3 Which algorithm reads a legacy artefact
`is_old` is true, so every other field is ignored — there is no `mode` to
consult. The legacy path tries the bulk-operation implementation first and
falls back to the original character pipeline:
| Attempt | Why |
|---|---|
| 1. `fast` | Same algorithm, faster; correct for anything the bulk path can read |
| 2. `reference` | The original pipeline, in case the artefact needs it |
Both implement the original algorithm, so either can read pre-header material.
The fallback costs a second attempt only when the first fails.
### 12.4 What is not supported
- **Writing legacy artefacts.** The library only reads them. Output always
carries a header.
- **Partial trust of a legacy artefact.** It is wrapped whole; nothing is
inferred from its content.
- **Reading a legacy artefact without wrapping.** The ordinary path will reject
it with `E017`. Use the compatibility API.
### 12.5 Conformance for readers
An implementation that supports legacy artefacts must:
- [ ] detect a headerless input and treat it as legacy, not as an error at the
API boundary;
- [ ] synthesise a header rather than bypass the parser;
- [ ] ignore every field except `is_old` and `old_is_key` when `is_old` is true;
- [ ] keep the ordinary path strict — a headerless input passed to `decrypt`
still raises `E017`;
- [ ] never write a legacy artefact.
---
## 13. Relation to the cipher
The header is metadata. It does not add secrecy, and it is not authenticated.
An attacker who modifies the header can cause a decryption to fail, which is a
denial of service and no more — the payload cannot be recovered without the key.
Where an application needs to detect tampering with the header as well as the
payload, it must compute a MAC over the **entire artefact**, header included,
and verify that MAC before parsing. The format does not do this for you.
See [SECURITY.md](../SECURITY.md) for what the cipher guarantees and what it
does not.
+5
View File
@@ -7,6 +7,7 @@ Reference material for ChaosCrypt.
| [API.md](API.md) | Complete Python API reference |
| [CLI.md](CLI.md) | Command line reference, pipelines, troubleshooting |
| [ARCHITECTURE.md](ARCHITECTURE.md) | Module map, data flow, design decisions |
| [benchmark.md](benchmark.md) | Performance measurements and mode comparison |
| [BUILDING.md](BUILDING.md) | Building, testing and releasing |
| [../SECURITY.md](../SECURITY.md) | Design, key space, threat model, attack analysis |
| [../README.md](../README.md) | Project overview and quick start |
@@ -26,6 +27,10 @@ Read the [Scope](../README.md#scope) section of the README, then
**Using the command line**
[CLI.md](CLI.md) — covers every subcommand, plus pipelines and shell aliases.
**Choosing a performance mode**
[benchmark.md](benchmark.md) for the numbers, [SECURITY.md §4](../SECURITY.md)
for what each mode costs in cryptographic margin.
**Understanding how it works**
[ARCHITECTURE.md](ARCHITECTURE.md) for the code structure, and
[SECURITY.md §3](../SECURITY.md) for the algorithm itself.
+251
View File
@@ -0,0 +1,251 @@
# Benchmark
Performance reference for `chaoscrypt` **4.2.6**.
> **Read this first.** These numbers are not a claim that ChaosCrypt competes
> with a C-backed cipher. It is pure Python. AES-GCM in `cryptography` runs at
> hundreds of megabytes per second; ChaosCrypt's fastest mode manages tens.
> What the tables below show is what the implementation costs, how the modes
> differ, and which parameters are realistic for the workloads this library is
> meant for.
---
## 1. Headline figures
`raw` mode, hex output, keystream cache warm, best of several runs.
| Payload | `bt` encrypt | `bt` decrypt | `hex` encrypt | `hex` decrypt |
|---|---:|---:|---:|---:|
| 64 KiB | 26.9 | 60.6 | 20.7 | 20.2 |
| 128 KiB | 54.9 | 85.5 | 15.8 | 14.3 |
| 1 MiB | 57.3 | 61.0 | 19.3 | 21.8 |
| 16 MiB | 63.4 | 69.0 | 18.9 | 22.7 |
| 128 MiB | 59.4 | 64.5 | 18.2 | 18.0 |
| 1024 MiB | 47.9 | 31.7 | — | — |
MB/s. Ciphertext expansion: `bt` **2.00x**, `hex` **8.00x**.
**The one-line summary:** `raw` with the default `bt` intermediate runs at
roughly **60 MB/s encryption and 70 MB/s decryption**, with ciphertext twice the
payload size. Throughput holds steady across four orders of magnitude, which is
what a linear algorithm looks like.
### Against the original implementation
| | Original | `raw` bt | Improvement |
|---|---:|---:|---:|
| 1 MiB encrypt | 12.050 s (0.08 MB/s) | 0.017 s (57 MB/s) | **≈700x** |
| Ciphertext | 5.33x | 2.00x | 62% smaller |
---
## 2. Test environment
| Item | Value |
|---|---|
| CPU | 16 logical cores |
| Python | CPython 3.13.4 |
| OS | Windows 10 (10.0.26200) |
| Library | `chaoscrypt` 4.2.6, pure Python, zero dependencies |
| Payload | ASCII, `"A" * n` |
| Measurement | Best of several runs; each size warmed once so the keystream cache is populated |
Best-of-N is used because the machine is not otherwise idle. That makes the
figures optimistic; a loaded system will be slower, and the ordering between
modes is unaffected.
> **1024 MiB with `hex`** needs an 8 GB ciphertext as a single Python string,
> which is not addressable. That cell is left blank rather than measured in a
> way that would only report the allocator's behaviour.
---
## 3. Throughput across four orders of magnitude
`raw`, `bt` intermediate, encryption:
| Size | MB/s | Ratio to previous |
|---|---:|---:|
| 64 KiB | 26.9 | — |
| 128 KiB | 54.9 | 2.04x |
| 1 MiB | 57.3 | 1.04x |
| 16 MiB | 63.4 | 1.11x |
| 128 MiB | 59.4 | 0.94x |
| 1024 MiB | 47.9 | 0.81x |
From 128 KiB upward the figure is flat within ±15%, which is measurement noise
plus cache effects rather than algorithmic behaviour. **64 KiB reads low
because fixed setup costs are a larger share of a short operation**, not because
something is quadratic.
The 1024 MiB figure is lower because a 2 GB ciphertext string puts the process
under memory pressure. It is an observation about the machine, not the
algorithm.
### The quadratic bug that used to be here
An earlier revision was genuinely super-linear, and it showed up exactly this
way: throughput fell as payload grew. The cause was `_get_dynamic_key`:
```python
key = self.long_key
while len(key) < text_length:
key += self.long_key # reallocates the whole accumulator every round
```
Covering a 5.6 MB intermediate took **866 ms**. Multiplication instead:
```python
return (lk * (text_length // ll + 1))[:text_length]
```
The same work takes **3 ms**, with byte-identical output. `_expand_keystream_slow`
shared the pattern and was fixed the same way.
---
## 4. Intermediate representation
The setting with the largest effect on both speed and size.
| Payload | `bt` enc | `bt` dec | `hex` enc | `hex` dec | Ciphertext `bt` / `hex` |
|---|---:|---:|---:|---:|---:|
| 64 KiB | 26.9 | 60.6 | 20.7 | 20.2 | 0.1 / 0.5 MB |
| 1 MiB | 57.3 | 61.0 | 19.3 | 21.8 | 2 / 8 MB |
| 16 MiB | 63.4 | 69.0 | 18.9 | 22.7 | 32 / 128 MB |
| 128 MiB | 59.4 | 64.5 | 18.2 | 18.0 | 256 / 1024 MB |
`hex` is about **3x slower** and produces **4x more** ciphertext. It re-encodes
the payload between the XOR layers, which doubles it twice. It exists for
interoperability with the character pipeline's shape, not for performance.
---
## 5. Output encoding
| Encoding | Expansion | Notes |
|---|---:|---|
| `hex` | 2.00x | Two characters per octet |
| `base64` | 1.33x | Four characters per three octets |
Base64 produces a third less output at broadly the same speed. The choice
affects only the final encode.
---
## 6. How the modes compare
| | `reference` | `fast` | `turbo` | `raw` |
|---|---|---|---|---|
| 1 MiB encrypt | 12.05 s | 0.35 s | 0.05 s | 0.017 s |
| Ciphertext | 5.33x | 5.33x | 2.00x | 2.00x |
| Payload compatible with pre-4.2.6 | yes | yes | no | no |
| Keystream primitive | hand-rolled expansion | hand-rolled expansion | SHAKE256 | SHAKE256 |
`fast` is `reference` with the same arithmetic performed by C-level bulk
operations; its payload is byte-identical. The frozen vectors assert this, so
the equivalence cannot silently stop holding.
`turbo` and `raw` replace the keystream derivation with SHAKE256 and stop
re-encoding between layers. `raw` keeps the long-key XOR layer, `turbo` does not.
---
## 7. Where the time goes
Profiling `raw` with the `bt` intermediate, 10 MiB payload, encryption:
| Stage | Time | Share |
|---|---:|---:|
| `_bytes_stages` (substitution, flip, reversal) | 29 ms | 24% |
| XOR with the long key | 38 ms | 32% |
| XOR with the SHAKE keystream | 38 ms | 32% |
| Keystream derivation (cached) | ~0 ms | 0% |
| Output encode (`.hex()`) | 9 ms | 8% |
| **Total** | **119 ms** | |
Decryption is the mirror image, at 122 ms, and is if anything slightly quicker
end to end because it does not encode its output.
### The changes that mattered
**1. Keeping the payload in bytes between XOR layers (4x).**
The original re-encoded to hexadecimal after each XOR, doubling the payload at
every layer. At 1 MiB that meant a 5.6 MB intermediate. Bytes between layers
means a 1.4 MB intermediate instead.
**2. Per-character loops to `bytes.translate` (61x on substitution alone).**
Substitution was a dictionary lookup per character; it is now one translation
over a 256-entry table. The case flip is periodic, so it became ten strided
slice translations instead of a per-character branch.
**3. Per-byte XOR to one big-integer XOR (43x).**
```python
# before
for i, char in enumerate(text):
result.append(f"{ord(char) ^ ord(key[i % key_len]):02x}")
# after
xored = int.from_bytes(data, 'big') ^ int.from_bytes(key, 'big')
return xored.to_bytes(n, 'big').hex()
```
**4. Not copying or scanning the whole artefact on decrypt (2x).**
The decrypt path used to call `strip()`, then `int(payload, 16)`, and
`strip_header` sliced the whole string three times. On a 20 MB artefact that was
about 65 ms of pure copying. Removing it took decryption at 512 KiB from
34 MB/s to 87 MB/s.
---
## 8. What did not work
An attempt to build all expansion chunks with `numpy` as a two-dimensional
array reached only 2.9x and added a dependency. Cached translation tables were
faster and dependency-free.
A later attempt to fold the hex round trip in `_derive_final_key` by processing
only the prefix the output consumes gained 1.6x — worth keeping, but far less
than the byte-pipeline change that made the whole stage cheaper.
---
## 9. Reproducing these numbers
```bash
python bench/benchmark.py
python bench/benchmark.py --sizes 64,1024,65536 --budget 0.05 --repeats 3
python bench/benchmark.py --groups Text,Binary
```
The harness dispatches every task to a process pool, one worker per core, and
reports the best of N rounds per task.
### A note on large payloads
Beyond roughly 128 MiB the ciphertext itself becomes the practical limit: at
1024 MiB inputs, `bt` needs a 2 GB string and `hex` needs 8 GB. For payloads
that large, encrypt in blocks. The per-block throughput is the same, as §3
shows.
---
## 10. Summary
| Dimension | Result |
|---|---|
| Fastest throughput | **~63 MB/s encryption, ~85 MB/s decryption** (`raw`, `bt`, hex) |
| Ciphertext overhead | 2.00x (`bt`) or 1.33x (`base64`) |
| Speed with a payload compatible with pre-4.2.6 | `fast`, ~30x the original |
| Linear across | 64 KiB to 1 GiB |
| Dependencies | none, in every mode |
| Best suited for | Notes, credentials, documents, personal backups |
| Not suited for | Bulk media at gigabyte scale, high-throughput services, KDF stretching |
> **Bottom line.** The original implementation's cost was almost entirely
> Python-level per-byte work, not the cipher's arithmetic. Keeping the payload
> in bytes between the XOR layers removed the largest remaining factor, and the
> result is a library that runs at tens of megabytes per second rather than
> fractions of one — with the same construction, and with the modes and their
> trade-offs documented in [SECURITY.md](../SECURITY.md).