v4.2.6: Self-describing format, byte pipeline, and a performance pass

Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
This commit is contained in:
dvs
2026-09-27 22:06:34 +08:00
parent 24a36d6fb4
commit dd49043171
25 changed files with 5348 additions and 852 deletions
+618
View File
@@ -0,0 +1,618 @@
# CES Artifact Format Specification
**Normative reference for the ChaosCrypt (CES) container format**
| | |
|---|---|
| Format version | 1 |
| Status | Normative |
| Applies to | Key files and ciphertexts produced from 4.2.6 onward |
| Library | `chaoscrypt` |
---
## 1. Scope
This document specifies the byte-level layout of every artefact the library
produces. It is the authority for implementers: where this document and the
code disagree, this document is wrong and the code is to be fixed.
It defines:
- the fixed magic and delimiter values;
- the layout of the self-describing header;
- the complete field set of the configuration record;
- the payload that follows;
- the conformance requirements a reader must enforce.
It does **not** describe the cipher itself. For the algorithm see
[SECURITY.md](../SECURITY.md); for performance see [benchmark.md](benchmark.md).
---
## 2. Design rationale
Every artefact carries a header that states how it was produced. The reader
never infers a parameter from context and never guesses.
Earlier revisions left the mode and encoding to the caller's memory. That is a
silent-corruption hazard: a ciphertext decrypted under the wrong mode yields
garbage that is indistinguishable from a wrong password. Recording the
parameters alongside the artefact removes the ambiguity, and rejecting
headerless input removes the guesswork entirely.
The consequence is deliberate and absolute: **input without a valid header is
rejected, not interpreted.**
---
## 3. Fixed values
These byte sequences are constants of the format. They are reproduced in three
representations for convenience; all three denote the same octets.
### 3.1 Magic
| Representation | Value |
|---|---|
| Bytes | `b"69CES"` |
| Hex | `3639434553` |
| Decimal byte list | `[54, 57, 67, 69, 83]` |
Length: 5 octets. Position: the first octet of every artefact.
### 3.2 Begin marker
```
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
64 characters of lowercase hexadecimal. Written verbatim; it is **not** decoded
before comparison.
### 3.3 End marker
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
```
64 characters of lowercase hexadecimal. Written verbatim.
> **Note.** Both markers are fixed constants of format version 1. They are
> delimiters, not integrity checks: they carry no signature semantics and do
> not authenticate the payload. Message integrity, where required, is provided
> by the cipher and by an external MAC.
---
## 4. Container layout
An artefact is a single character string with five regions:
```
+-------------------------------------------------------------------+
| MAGIC "69CES" 5 octets |
+-------------------------------------------------------------------+
| BEGIN MARKER 64 hex chars |
+-------------------------------------------------------------------+
| CONFIGURATION UTF-8 JSON object variable |
+-------------------------------------------------------------------+
| END MARKER 64 hex chars |
+-------------------------------------------------------------------+
| PAYLOAD key material or ciphertext variable |
+-------------------------------------------------------------------+
```
The three fixed regions are literals. Only the configuration and the payload
vary between artefacts.
### 4.1 Formal grammar
```
artifact = MAGIC begin-marker config end-marker payload
MAGIC = "69CES"
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
config = <UTF-8 encoded JSON object, see section 5>
payload = <encoding-dependent text, see section 6>
```
There is no whitespace, newline or padding between regions. A reader must not
skip optional characters.
---
## 5. Configuration record
The configuration is a JSON object encoded as UTF-8, with no line breaks.
Keys are sorted; separators are `,` and `:` with no surrounding spaces. The
canonical form is therefore deterministic: the same configuration always
renders as the same octets.
### 5.1 Required fields
A configuration missing any of these is invalid, and the artefact must be
rejected rather than partially interpreted.
| Field | Type | Meaning |
|---|---|---|
| `kind` | string | `"ciphertext"` or `"key"` |
| `v` | integer | Format version; `1` for this specification |
| `mode` | string | Performance mode, see §5.2 |
| `mid` | string | Intermediate representation, see §5.3 |
| `out` | string | Output encoding, see §5.4 |
| `alg` | string | Algorithm family; `"chaoscrypt"` |
| `kdf` | string | Keystream derivation, see §5.5 |
| `keybits` | integer | Length of the long key in bits |
| `ksum` | string | Non-secret key digest, see §5.6 |
### 5.2 `mode`
| Value | Pipeline |
|---|---|
| `reference` | Original algorithm, character pipeline |
| `fast` | Original algorithm, bulk-operation implementation |
| `turbo` | SHAKE256 keystream, byte pipeline |
| `raw` | SHAKE256 keystream, byte pipeline with the long-key layer retained |
`reference` and `fast` produce identical ciphertexts. The distinction is
recorded so a reader can report which implementation wrote the artefact; it
does not affect decryption.
### 5.3 `mid` — intermediate representation
The `mid` field names how the two XOR layers of the byte pipeline exchange
their data. It is only meaningful for `turbo` and `raw`; for the character
pipelines it is descriptive.
| Value | Meaning |
|---|---|
| `bt` | Intermediate values carried as raw bytes (default) |
| `hex` | Intermediate values carried as hexadecimal text |
`turbo` and `raw` accept either value, and `bt` is the default. The choice is
recorded in the header and honoured when the artefact is read: a `turbo` or
`raw` artefact is decrypted with the `mid` it carries, and no fallback to the
other representation is attempted.
`reference` and `fast` are character pipelines and always use `hex`. They
ignore `mid`: whatever value is supplied, the header records `hex` for them.
**Expansion.** `bt` passes the bytes between the XOR layers unchanged, so the
only growth is the output encoding: 2.00x with `hex` output, 1.33x with
`base` output. `hex` re-encodes each layer's output as hexadecimal text
before the next layer consumes it; with the two layers of this format that
doubles the payload at each layer, so the byte payload handed to the output
encoder is about **4x** the `bt` size (roughly 8 hexadecimal characters per
input octet with `hex` output).
**Speed.** `bt` is about **4x faster** than `hex`: it skips the per-layer hex
encode and decode and XORs the shorter byte buffers directly. It is therefore
the default.
### 5.4 `out` — output encoding
| Value | Meaning | Expansion |
|---|---|---|
| `hex` | Two characters per octet | 2.00x |
| `base` | Base64, four characters per three octets | 1.33x |
`reference` and `fast` always emit `hex` and ignore any attempt to set it.
`turbo` and `raw` accept either.
> **Reader requirement.** The payload must be decoded according to `out`.
> Attempting to decode with the wrong encoding must fail, never silently produce
> a different result.
### 5.5 `kdf` — keystream derivation
| Value | Meaning |
|---|---|
| `expand` | Hand-rolled mixing and rotation expansion |
| `shake256` | SHAKE256 extensible-output function |
### 5.6 `ksum` — key digest
A 16-character lowercase hexadecimal string: the first 16 characters of
`SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key)`, where `‖` is
concatenation of the ASCII encodings.
It exists so a reader can confirm that a key file and a ciphertext belong
together without exposing any usable secret. It is a **correlation aid, not a
security control**: it reveals nothing that helps an attacker, and its absence
of secrecy must not be relied upon as an integrity guarantee.
### 5.7 Legacy fields
Two optional fields exist for reading artefacts produced before the header was
introduced. They are never written by the current library except when wrapping a
legacy artefact, and they are **not** part of :data:`REQUIRED_FIELDS`.
| Field | Type | Meaning |
|---|---|---|
| `is_old` | boolean | The artefact predates this format |
| `old_is_key` | boolean | With `is_old`, whether it is a key file (`true`) or a ciphertext (`false`) |
> **Precedence rule.** When `is_old` is present and true, **every other field is
> ignored**, including `mode`, `mid`, `out` and `kind`. The artefact predates the
> field set, so there is nothing else meaningful to read. A reader must not
> reject such an artefact for missing `kind`, nor act on any field beside
> `is_old` and `old_is_key`.
An artefact that carries neither field is a normal artefact and is read
normally. Their absence is not an error.
### 5.8 Descriptive fields
The record also carries fields that describe the construction. They are
informational — a reader may use them for display or diagnostics — and they do
not modify the decryption path.
| Field | Meaning |
|---|---|
| `lib` | Library version that wrote the artefact |
| `cache` | Keystream cache strategy in effect; does not affect output |
| `subst` | Substitution kind, `"bijective"` |
| `flip` | Case-flip kind, `"conditional"` |
| `order` | The pipeline, as an ordered list of stage names |
| `xor_layers` | Number of XOR layers applied |
| `compact` | Whether the byte pipeline was used |
| `keyshort` | Length of the short key in bits |
| `flipbits` | Number of bits in the flip pattern |
| `b64` | Whether Base64 is used as a byte-safety layer |
| `utf8` | Whether text is transported as UTF-8 |
### 5.9 Example
```json
{
"alg": "chaoscrypt",
"b64": true,
"cache": "hash",
"compact": true,
"flip": "conditional",
"flipbits": 10,
"kdf": "shake256",
"keybits": 16384,
"keyshort": 2048,
"kind": "ciphertext",
"ksum": "3f9a1c7e5b2d4086",
"lib": "4.2.6",
"mid": "bt",
"mode": "turbo",
"order": "base64>subst>flip>reverse>xor>xor",
"out": "hex",
"subst": "bijective",
"utf8": true,
"v": 1,
"xor_layers": 2
}
```
Rendered between the markers, this is a single unbroken line.
---
## 6. Payload
### 6.1 Ciphertext payload
When `kind` is `"ciphertext"`, the payload is the ciphertext encoded per `out`:
- `out = "hex"`: lowercase hexadecimal, even length, alphabet `[0-9a-f]`.
- `out = "base"`: standard Base64 with `=` padding, alphabet
`[A-Za-z0-9+/]`.
The payload is the final output of the pipeline named in `mode`. Nothing is
appended after it.
### 6.2 Key payload
When `kind` is `"key"`, the payload is the password-wrapped key material, in the
same two encodings.
A caller reading a key file must check that `kind` is `"key"`. A key file is not
a ciphertext and must not be accepted where a ciphertext is expected, nor the
reverse. This is enforced, not advisory.
---
## 7. Reader conformance
A conforming reader performs these steps, in order, and fails at the first
that does not hold. It must not attempt recovery, interpretation or fallback.
| # | Check | Failure |
|---|---|---|
| 1 | Input is non-empty | `E017` |
| 2 | Input starts with `69CES` | `E017` |
| 3 | The 64 characters after the magic equal the begin marker | `E018` |
| 4 | The end marker occurs after the begin marker | `E018` |
| 5 | Text between the markers is valid UTF-8 JSON | `E018` |
| 6 | That JSON is an object | `E018` |
| 7 | Every required field of §5.1 is present | `E018` |
| 8 | `kind`, `mid` and `out` hold permitted values | `E018` |
| 9 | `kind` matches what the caller asked for | `E018` |
| 10 | The payload decodes under `out` | `E010` |
| 11 | Decryption succeeds under `mode` and the key | `E011` |
### 7.1 No fallback
There is no lenient path. An artefact without a header is not a ChaosCrypt
artefact and is rejected with `E017`. Artefacts written by revisions before the
header existed are not read by this path either: they are recovered only
through the explicit compatibility API, which wraps them in a header first
(see section 12). The ordinary reader never gains a lenient branch.
### 7.2 No partial trust
A header is either wholly valid or wholly rejected. A reader must not act on
fields it did manage to parse, because a partially parsed header may describe a
different construction than the one that produced the payload.
---
## 8. Error codes
| Code | Name | Raised when |
|---|---|---|
| `E010` | `MalformedCiphertext` | Payload does not decode under `out` |
| `E011` | `DecryptionFailed` | Decryption produced invalid data |
| `E017` | `MissingMagicHeader` | No magic present; input is not a CES artefact |
| `E018` | `MalformedHeader` | Marker missing, JSON invalid, field absent or out of range, or `kind` mismatch |
Codes `E001`–`E016` cover key handling, argument and I/O failures and are
unchanged.
---
## 9. Versioning
`v` is the format version and is independent of the library version in `lib`.
A reader must reject an artefact whose `v` it does not implement, with `E018`.
Adding optional descriptive fields does not change `v`. Changing the magic, the
markers, the region order, or the meaning of a required field does, and requires
a new major library release.
---
## 10. Conformance checklist
An implementation claims conformance when it:
- [ ] writes the magic, both markers and the configuration in the order of §4;
- [ ] sorts configuration keys and uses `,` and `:` as separators with no spaces;
- [ ] includes every required field of §5.1;
- [ ] rejects input without the magic with `E017`;
- [ ] rejects malformed headers with `E018`, never partially trusting them;
- [ ] enforces the `kind` match on both read and write paths;
- [ ] decodes the payload strictly under `out`, failing rather than guessing;
- [ ] implements no lenient or legacy fallback path.
---
---
## 11. Worked examples
### 11.1 A normal ciphertext
Text that is encrypted in `turbo` with `base64` output, then the layout read
back region by region.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="turbo", encoding="base64")
cc.load("my.key", "key-file-password")
token = cc.encrypt("meet me at the usual place", "encryption-password")
```
The token begins with the five magic octets, then the begin marker:
```
69CES
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
then the configuration, which states the parameters actually used:
```json
{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
"flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
"keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
"lib":"4.2.6","mid":"bt","mode":"turbo",
"order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
"utf8":true,"v":1,"xor_layers":2}
```
then the end marker, then the base64 payload:
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...
```
Decrypting needs no configuration at all — the record supplies it:
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt._header import read_header, strip_header
cfg = read_header(token)
print(cfg["mode"], cfg["out"]) # turbo base
# a fresh instance with no settings reads it correctly
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True).decrypt(token, "encryption-password")
```
### 11.2 Reading an artefact without knowing anything about it
Because the header is complete, a reader needs only the key file and the
password. The mode, the intermediate representation and the output encoding are
all taken from the artefact.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
cc.load("my.key", "key-file-password")
# whatever produced this, it will be read correctly
plain = cc.decrypt(open("message.ces").read(), "encryption-password")
```
### 11.3 Inspecting a key file
```python
from chaoscrypt._header import read_header, strip_header
text = open("my.key", encoding="utf-8").read()
cfg = read_header(text)
print(cfg["kind"]) # key
print(cfg["alg"]) # chaoscrypt
print(cfg["keybits"]) # 16384
print(cfg["ksum"]) # 16 hex characters
```
`ksum` lets a caller confirm that a key file and a ciphertext belong together
without exposing anything usable:
```python
cs = open("message.ces", encoding="utf-8").read()
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"
```
### 11.4 A rejection
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt.errors import ChaosCryptError
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
try:
cc.decrypt("7a3f9c...", "encryption-password") # a bare payload
except ChaosCryptError as e:
print(e.code) # E017
print(e.detail) # MissingMagicHeader
```
There is no fallback path here. The input is not a CES artefact, and guessing
would risk returning garbage that looks like success.
---
## 12. Legacy artefacts
Artefacts written before this format existed have no header. They are handled by
**wrapping**, never by loosening the reader.
### 12.1 What a legacy artefact looks like
| Kind | Content |
|---|---|
| Legacy ciphertext | Hexadecimal text, no prefix of any kind |
| Legacy key file | Password-wrapped key material, no prefix of any kind |
Neither carries a magic value, a marker or a configuration record.
### 12.2 Wrapping
To read one, a minimal header is synthesised and prefixed. The payload is not
touched.
```json
{"is_old":true,"old_is_key":false}
```
That header is a normal, valid header. Wrapping therefore produces an artefact
that the ordinary path can read, and **the ordinary path never gains a lenient
branch**: it still rejects headerless input with `E017`.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
# a legacy ciphertext: no header
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
"encryption-password")
```
`decrypt_legacy` first tries the input as it stands. If that reports a missing
header, it wraps the input and tries again. `decrypt_legacy_auto` additionally
probes whether the artefact is a key file or a ciphertext, so the caller does
not have to say which.
A legacy key file is loaded the same way:
```python
legacy_text = open("old-encryption.key", encoding="utf-8").read()
cc = ChaosCrypt()
cc.load_legacy_keys(legacy_text, "key-file-password")
```
Or wrapped explicitly, when the artefact is to be stored in the new form:
```python
modernised = cc.wrap_legacy(legacy_text, is_key=True)
open("my.key", "w", encoding="utf-8").write(modernised)
```
### 12.3 Which algorithm reads a legacy artefact
`is_old` is true, so every other field is ignored — there is no `mode` to
consult. The legacy path tries the bulk-operation implementation first and
falls back to the original character pipeline:
| Attempt | Why |
|---|---|
| 1. `fast` | Same algorithm, faster; correct for anything the bulk path can read |
| 2. `reference` | The original pipeline, in case the artefact needs it |
Both implement the original algorithm, so either can read pre-header material.
The fallback costs a second attempt only when the first fails.
### 12.4 What is not supported
- **Writing legacy artefacts.** The library only reads them. Output always
carries a header.
- **Partial trust of a legacy artefact.** It is wrapped whole; nothing is
inferred from its content.
- **Reading a legacy artefact without wrapping.** The ordinary path will reject
it with `E017`. Use the compatibility API.
### 12.5 Conformance for readers
An implementation that supports legacy artefacts must:
- [ ] detect a headerless input and treat it as legacy, not as an error at the
API boundary;
- [ ] synthesise a header rather than bypass the parser;
- [ ] ignore every field except `is_old` and `old_is_key` when `is_old` is true;
- [ ] keep the ordinary path strict — a headerless input passed to `decrypt`
still raises `E017`;
- [ ] never write a legacy artefact.
---
## 13. Relation to the cipher
The header is metadata. It does not add secrecy, and it is not authenticated.
An attacker who modifies the header can cause a decryption to fail, which is a
denial of service and no more — the payload cannot be recovered without the key.
Where an application needs to detect tampering with the header as well as the
payload, it must compute a MAC over the **entire artefact**, header included,
and verify that MAC before parsing. The format does not do this for you.
See [SECURITY.md](../SECURITY.md) for what the cipher guarantees and what it
does not.