Files
dvs dd49043171 v4.2.6: Self-describing format, byte pipeline, and a performance pass
Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
2026-09-27 22:06:34 +08:00

619 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CES Artifact Format Specification
**Normative reference for the ChaosCrypt (CES) container format**
| | |
|---|---|
| Format version | 1 |
| Status | Normative |
| Applies to | Key files and ciphertexts produced from 4.2.6 onward |
| Library | `chaoscrypt` |
---
## 1. Scope
This document specifies the byte-level layout of every artefact the library
produces. It is the authority for implementers: where this document and the
code disagree, this document is wrong and the code is to be fixed.
It defines:
- the fixed magic and delimiter values;
- the layout of the self-describing header;
- the complete field set of the configuration record;
- the payload that follows;
- the conformance requirements a reader must enforce.
It does **not** describe the cipher itself. For the algorithm see
[SECURITY.md](../SECURITY.md); for performance see [benchmark.md](benchmark.md).
---
## 2. Design rationale
Every artefact carries a header that states how it was produced. The reader
never infers a parameter from context and never guesses.
Earlier revisions left the mode and encoding to the caller's memory. That is a
silent-corruption hazard: a ciphertext decrypted under the wrong mode yields
garbage that is indistinguishable from a wrong password. Recording the
parameters alongside the artefact removes the ambiguity, and rejecting
headerless input removes the guesswork entirely.
The consequence is deliberate and absolute: **input without a valid header is
rejected, not interpreted.**
---
## 3. Fixed values
These byte sequences are constants of the format. They are reproduced in three
representations for convenience; all three denote the same octets.
### 3.1 Magic
| Representation | Value |
|---|---|
| Bytes | `b"69CES"` |
| Hex | `3639434553` |
| Decimal byte list | `[54, 57, 67, 69, 83]` |
Length: 5 octets. Position: the first octet of every artefact.
### 3.2 Begin marker
```
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
64 characters of lowercase hexadecimal. Written verbatim; it is **not** decoded
before comparison.
### 3.3 End marker
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
```
64 characters of lowercase hexadecimal. Written verbatim.
> **Note.** Both markers are fixed constants of format version 1. They are
> delimiters, not integrity checks: they carry no signature semantics and do
> not authenticate the payload. Message integrity, where required, is provided
> by the cipher and by an external MAC.
---
## 4. Container layout
An artefact is a single character string with five regions:
```
+-------------------------------------------------------------------+
| MAGIC "69CES" 5 octets |
+-------------------------------------------------------------------+
| BEGIN MARKER 64 hex chars |
+-------------------------------------------------------------------+
| CONFIGURATION UTF-8 JSON object variable |
+-------------------------------------------------------------------+
| END MARKER 64 hex chars |
+-------------------------------------------------------------------+
| PAYLOAD key material or ciphertext variable |
+-------------------------------------------------------------------+
```
The three fixed regions are literals. Only the configuration and the payload
vary between artefacts.
### 4.1 Formal grammar
```
artifact = MAGIC begin-marker config end-marker payload
MAGIC = "69CES"
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
config = <UTF-8 encoded JSON object, see section 5>
payload = <encoding-dependent text, see section 6>
```
There is no whitespace, newline or padding between regions. A reader must not
skip optional characters.
---
## 5. Configuration record
The configuration is a JSON object encoded as UTF-8, with no line breaks.
Keys are sorted; separators are `,` and `:` with no surrounding spaces. The
canonical form is therefore deterministic: the same configuration always
renders as the same octets.
### 5.1 Required fields
A configuration missing any of these is invalid, and the artefact must be
rejected rather than partially interpreted.
| Field | Type | Meaning |
|---|---|---|
| `kind` | string | `"ciphertext"` or `"key"` |
| `v` | integer | Format version; `1` for this specification |
| `mode` | string | Performance mode, see §5.2 |
| `mid` | string | Intermediate representation, see §5.3 |
| `out` | string | Output encoding, see §5.4 |
| `alg` | string | Algorithm family; `"chaoscrypt"` |
| `kdf` | string | Keystream derivation, see §5.5 |
| `keybits` | integer | Length of the long key in bits |
| `ksum` | string | Non-secret key digest, see §5.6 |
### 5.2 `mode`
| Value | Pipeline |
|---|---|
| `reference` | Original algorithm, character pipeline |
| `fast` | Original algorithm, bulk-operation implementation |
| `turbo` | SHAKE256 keystream, byte pipeline |
| `raw` | SHAKE256 keystream, byte pipeline with the long-key layer retained |
`reference` and `fast` produce identical ciphertexts. The distinction is
recorded so a reader can report which implementation wrote the artefact; it
does not affect decryption.
### 5.3 `mid` — intermediate representation
The `mid` field names how the two XOR layers of the byte pipeline exchange
their data. It is only meaningful for `turbo` and `raw`; for the character
pipelines it is descriptive.
| Value | Meaning |
|---|---|
| `bt` | Intermediate values carried as raw bytes (default) |
| `hex` | Intermediate values carried as hexadecimal text |
`turbo` and `raw` accept either value, and `bt` is the default. The choice is
recorded in the header and honoured when the artefact is read: a `turbo` or
`raw` artefact is decrypted with the `mid` it carries, and no fallback to the
other representation is attempted.
`reference` and `fast` are character pipelines and always use `hex`. They
ignore `mid`: whatever value is supplied, the header records `hex` for them.
**Expansion.** `bt` passes the bytes between the XOR layers unchanged, so the
only growth is the output encoding: 2.00x with `hex` output, 1.33x with
`base` output. `hex` re-encodes each layer's output as hexadecimal text
before the next layer consumes it; with the two layers of this format that
doubles the payload at each layer, so the byte payload handed to the output
encoder is about **4x** the `bt` size (roughly 8 hexadecimal characters per
input octet with `hex` output).
**Speed.** `bt` is about **4x faster** than `hex`: it skips the per-layer hex
encode and decode and XORs the shorter byte buffers directly. It is therefore
the default.
### 5.4 `out` — output encoding
| Value | Meaning | Expansion |
|---|---|---|
| `hex` | Two characters per octet | 2.00x |
| `base` | Base64, four characters per three octets | 1.33x |
`reference` and `fast` always emit `hex` and ignore any attempt to set it.
`turbo` and `raw` accept either.
> **Reader requirement.** The payload must be decoded according to `out`.
> Attempting to decode with the wrong encoding must fail, never silently produce
> a different result.
### 5.5 `kdf` — keystream derivation
| Value | Meaning |
|---|---|
| `expand` | Hand-rolled mixing and rotation expansion |
| `shake256` | SHAKE256 extensible-output function |
### 5.6 `ksum` — key digest
A 16-character lowercase hexadecimal string: the first 16 characters of
`SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key)`, where `‖` is
concatenation of the ASCII encodings.
It exists so a reader can confirm that a key file and a ciphertext belong
together without exposing any usable secret. It is a **correlation aid, not a
security control**: it reveals nothing that helps an attacker, and its absence
of secrecy must not be relied upon as an integrity guarantee.
### 5.7 Legacy fields
Two optional fields exist for reading artefacts produced before the header was
introduced. They are never written by the current library except when wrapping a
legacy artefact, and they are **not** part of :data:`REQUIRED_FIELDS`.
| Field | Type | Meaning |
|---|---|---|
| `is_old` | boolean | The artefact predates this format |
| `old_is_key` | boolean | With `is_old`, whether it is a key file (`true`) or a ciphertext (`false`) |
> **Precedence rule.** When `is_old` is present and true, **every other field is
> ignored**, including `mode`, `mid`, `out` and `kind`. The artefact predates the
> field set, so there is nothing else meaningful to read. A reader must not
> reject such an artefact for missing `kind`, nor act on any field beside
> `is_old` and `old_is_key`.
An artefact that carries neither field is a normal artefact and is read
normally. Their absence is not an error.
### 5.8 Descriptive fields
The record also carries fields that describe the construction. They are
informational — a reader may use them for display or diagnostics — and they do
not modify the decryption path.
| Field | Meaning |
|---|---|
| `lib` | Library version that wrote the artefact |
| `cache` | Keystream cache strategy in effect; does not affect output |
| `subst` | Substitution kind, `"bijective"` |
| `flip` | Case-flip kind, `"conditional"` |
| `order` | The pipeline, as an ordered list of stage names |
| `xor_layers` | Number of XOR layers applied |
| `compact` | Whether the byte pipeline was used |
| `keyshort` | Length of the short key in bits |
| `flipbits` | Number of bits in the flip pattern |
| `b64` | Whether Base64 is used as a byte-safety layer |
| `utf8` | Whether text is transported as UTF-8 |
### 5.9 Example
```json
{
"alg": "chaoscrypt",
"b64": true,
"cache": "hash",
"compact": true,
"flip": "conditional",
"flipbits": 10,
"kdf": "shake256",
"keybits": 16384,
"keyshort": 2048,
"kind": "ciphertext",
"ksum": "3f9a1c7e5b2d4086",
"lib": "4.2.6",
"mid": "bt",
"mode": "turbo",
"order": "base64>subst>flip>reverse>xor>xor",
"out": "hex",
"subst": "bijective",
"utf8": true,
"v": 1,
"xor_layers": 2
}
```
Rendered between the markers, this is a single unbroken line.
---
## 6. Payload
### 6.1 Ciphertext payload
When `kind` is `"ciphertext"`, the payload is the ciphertext encoded per `out`:
- `out = "hex"`: lowercase hexadecimal, even length, alphabet `[0-9a-f]`.
- `out = "base"`: standard Base64 with `=` padding, alphabet
`[A-Za-z0-9+/]`.
The payload is the final output of the pipeline named in `mode`. Nothing is
appended after it.
### 6.2 Key payload
When `kind` is `"key"`, the payload is the password-wrapped key material, in the
same two encodings.
A caller reading a key file must check that `kind` is `"key"`. A key file is not
a ciphertext and must not be accepted where a ciphertext is expected, nor the
reverse. This is enforced, not advisory.
---
## 7. Reader conformance
A conforming reader performs these steps, in order, and fails at the first
that does not hold. It must not attempt recovery, interpretation or fallback.
| # | Check | Failure |
|---|---|---|
| 1 | Input is non-empty | `E017` |
| 2 | Input starts with `69CES` | `E017` |
| 3 | The 64 characters after the magic equal the begin marker | `E018` |
| 4 | The end marker occurs after the begin marker | `E018` |
| 5 | Text between the markers is valid UTF-8 JSON | `E018` |
| 6 | That JSON is an object | `E018` |
| 7 | Every required field of §5.1 is present | `E018` |
| 8 | `kind`, `mid` and `out` hold permitted values | `E018` |
| 9 | `kind` matches what the caller asked for | `E018` |
| 10 | The payload decodes under `out` | `E010` |
| 11 | Decryption succeeds under `mode` and the key | `E011` |
### 7.1 No fallback
There is no lenient path. An artefact without a header is not a ChaosCrypt
artefact and is rejected with `E017`. Artefacts written by revisions before the
header existed are not read by this path either: they are recovered only
through the explicit compatibility API, which wraps them in a header first
(see section 12). The ordinary reader never gains a lenient branch.
### 7.2 No partial trust
A header is either wholly valid or wholly rejected. A reader must not act on
fields it did manage to parse, because a partially parsed header may describe a
different construction than the one that produced the payload.
---
## 8. Error codes
| Code | Name | Raised when |
|---|---|---|
| `E010` | `MalformedCiphertext` | Payload does not decode under `out` |
| `E011` | `DecryptionFailed` | Decryption produced invalid data |
| `E017` | `MissingMagicHeader` | No magic present; input is not a CES artefact |
| `E018` | `MalformedHeader` | Marker missing, JSON invalid, field absent or out of range, or `kind` mismatch |
Codes `E001`–`E016` cover key handling, argument and I/O failures and are
unchanged.
---
## 9. Versioning
`v` is the format version and is independent of the library version in `lib`.
A reader must reject an artefact whose `v` it does not implement, with `E018`.
Adding optional descriptive fields does not change `v`. Changing the magic, the
markers, the region order, or the meaning of a required field does, and requires
a new major library release.
---
## 10. Conformance checklist
An implementation claims conformance when it:
- [ ] writes the magic, both markers and the configuration in the order of §4;
- [ ] sorts configuration keys and uses `,` and `:` as separators with no spaces;
- [ ] includes every required field of §5.1;
- [ ] rejects input without the magic with `E017`;
- [ ] rejects malformed headers with `E018`, never partially trusting them;
- [ ] enforces the `kind` match on both read and write paths;
- [ ] decodes the payload strictly under `out`, failing rather than guessing;
- [ ] implements no lenient or legacy fallback path.
---
---
## 11. Worked examples
### 11.1 A normal ciphertext
Text that is encrypted in `turbo` with `base64` output, then the layout read
back region by region.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="turbo", encoding="base64")
cc.load("my.key", "key-file-password")
token = cc.encrypt("meet me at the usual place", "encryption-password")
```
The token begins with the five magic octets, then the begin marker:
```
69CES
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
```
then the configuration, which states the parameters actually used:
```json
{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
"flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
"keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
"lib":"4.2.6","mid":"bt","mode":"turbo",
"order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
"utf8":true,"v":1,"xor_layers":2}
```
then the end marker, then the base64 payload:
```
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...
```
Decrypting needs no configuration at all — the record supplies it:
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt._header import read_header, strip_header
cfg = read_header(token)
print(cfg["mode"], cfg["out"]) # turbo base
# a fresh instance with no settings reads it correctly
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True).decrypt(token, "encryption-password")
```
### 11.2 Reading an artefact without knowing anything about it
Because the header is complete, a reader needs only the key file and the
password. The mode, the intermediate representation and the output encoding are
all taken from the artefact.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
cc.load("my.key", "key-file-password")
# whatever produced this, it will be read correctly
plain = cc.decrypt(open("message.ces").read(), "encryption-password")
```
### 11.3 Inspecting a key file
```python
from chaoscrypt._header import read_header, strip_header
text = open("my.key", encoding="utf-8").read()
cfg = read_header(text)
print(cfg["kind"]) # key
print(cfg["alg"]) # chaoscrypt
print(cfg["keybits"]) # 16384
print(cfg["ksum"]) # 16 hex characters
```
`ksum` lets a caller confirm that a key file and a ciphertext belong together
without exposing anything usable:
```python
cs = open("message.ces", encoding="utf-8").read()
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"
```
### 11.4 A rejection
```python
from chaoscrypt import ChaosCrypt
from chaoscrypt.errors import ChaosCryptError
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
try:
cc.decrypt("7a3f9c...", "encryption-password") # a bare payload
except ChaosCryptError as e:
print(e.code) # E017
print(e.detail) # MissingMagicHeader
```
There is no fallback path here. The input is not a CES artefact, and guessing
would risk returning garbage that looks like success.
---
## 12. Legacy artefacts
Artefacts written before this format existed have no header. They are handled by
**wrapping**, never by loosening the reader.
### 12.1 What a legacy artefact looks like
| Kind | Content |
|---|---|
| Legacy ciphertext | Hexadecimal text, no prefix of any kind |
| Legacy key file | Password-wrapped key material, no prefix of any kind |
Neither carries a magic value, a marker or a configuration record.
### 12.2 Wrapping
To read one, a minimal header is synthesised and prefixed. The payload is not
touched.
```json
{"is_old":true,"old_is_key":false}
```
That header is a normal, valid header. Wrapping therefore produces an artefact
that the ordinary path can read, and **the ordinary path never gains a lenient
branch**: it still rejects headerless input with `E017`.
```python
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
# a legacy ciphertext: no header
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
"encryption-password")
```
`decrypt_legacy` first tries the input as it stands. If that reports a missing
header, it wraps the input and tries again. `decrypt_legacy_auto` additionally
probes whether the artefact is a key file or a ciphertext, so the caller does
not have to say which.
A legacy key file is loaded the same way:
```python
legacy_text = open("old-encryption.key", encoding="utf-8").read()
cc = ChaosCrypt()
cc.load_legacy_keys(legacy_text, "key-file-password")
```
Or wrapped explicitly, when the artefact is to be stored in the new form:
```python
modernised = cc.wrap_legacy(legacy_text, is_key=True)
open("my.key", "w", encoding="utf-8").write(modernised)
```
### 12.3 Which algorithm reads a legacy artefact
`is_old` is true, so every other field is ignored — there is no `mode` to
consult. The legacy path tries the bulk-operation implementation first and
falls back to the original character pipeline:
| Attempt | Why |
|---|---|
| 1. `fast` | Same algorithm, faster; correct for anything the bulk path can read |
| 2. `reference` | The original pipeline, in case the artefact needs it |
Both implement the original algorithm, so either can read pre-header material.
The fallback costs a second attempt only when the first fails.
### 12.4 What is not supported
- **Writing legacy artefacts.** The library only reads them. Output always
carries a header.
- **Partial trust of a legacy artefact.** It is wrapped whole; nothing is
inferred from its content.
- **Reading a legacy artefact without wrapping.** The ordinary path will reject
it with `E017`. Use the compatibility API.
### 12.5 Conformance for readers
An implementation that supports legacy artefacts must:
- [ ] detect a headerless input and treat it as legacy, not as an error at the
API boundary;
- [ ] synthesise a header rather than bypass the parser;
- [ ] ignore every field except `is_old` and `old_is_key` when `is_old` is true;
- [ ] keep the ordinary path strict — a headerless input passed to `decrypt`
still raises `E017`;
- [ ] never write a legacy artefact.
---
## 13. Relation to the cipher
The header is metadata. It does not add secrecy, and it is not authenticated.
An attacker who modifies the header can cause a decryption to fail, which is a
denial of service and no more — the payload cannot be recovered without the key.
Where an application needs to detect tampering with the header as well as the
payload, it must compute a MAC over the **entire artefact**, header included,
and verify that MAC before parsing. The format does not do this for you.
See [SECURITY.md](../SECURITY.md) for what the cipher guarantees and what it
does not.