Every key file and ciphertext now carries a header that states how it was produced: magic, two fixed markers, and a complete JSON configuration record. A reader takes its parameters from the artefact rather than from the caller. This is an incompatible format change. Headerless input is rejected with E017; pre-4.2.6 material is read through a compatibility API that wraps it in a minimal header and then uses the ordinary path, so the strict path stays strict. Added - Self-describing header on all artefacts, with a normative specification in docs/FORMAT.md - E017 MissingMagicHeader and E018 MalformedHeader - Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising throughput about 4x. The old behaviour is available as mid="hex". - Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x) - Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto, load_legacy_keys - Frozen reference vectors covering every mode, encoding and intermediate - docs/benchmark.md with measured throughput and the profiling behind it Changed - Default mode is fast: byte-for-byte compatible with the original algorithm, roughly 30x quicker - Substitution, case flipping and the printable fold use bytes.translate over precomputed tables - XOR layers use one big-integer operation instead of a per-byte loop - Keystream derivation caches length-independent base material; expansion uses cached translation tables and list accumulation rather than str += - _get_dynamic_key repeats by multiplication. The loop version was quadratic: covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged. - _derive_final_key decodes only the prefix the output consumes - The decrypt path no longer copies or scans the whole artefact: a redundant strip(), a full-string int(payload, 16), and three whole-string slices in strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s. - Documentation rewritten for 4.2.6; all eleven markdown files are English Fixed 1. Key derivation is no longer quadratic (_get_dynamic_key and _expand_keystream_slow shared the str += pattern) 2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact written with mid="hex" could not be read back 3. change_key_password and export_key_text wrote artefacts without a header 4. The constructor accepted encoding="base64" while the header records out="base"; both spellings are now accepted 5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext) rather than E011 (DecryptionFailed) 6. The TUI encoding menu referenced a missing translation Security - Format break from pre-4.2.6 artefacts, documented rather than accidental - fast is not a weaker mode: the frozen vectors assert identical output - The header adds no secrecy and is not authenticated; an application that must detect header tampering should MAC the whole artefact Performance - raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s decryption, ciphertext 2.00x the payload - Throughput holds steady from 64 KiB to 1 GiB - The original implementation managed 0.08 MB/s
20 KiB
CES Artifact Format Specification
Normative reference for the ChaosCrypt (CES) container format
| Format version | 1 |
| Status | Normative |
| Applies to | Key files and ciphertexts produced from 4.2.6 onward |
| Library | chaoscrypt |
1. Scope
This document specifies the byte-level layout of every artefact the library produces. It is the authority for implementers: where this document and the code disagree, this document is wrong and the code is to be fixed.
It defines:
- the fixed magic and delimiter values;
- the layout of the self-describing header;
- the complete field set of the configuration record;
- the payload that follows;
- the conformance requirements a reader must enforce.
It does not describe the cipher itself. For the algorithm see SECURITY.md; for performance see benchmark.md.
2. Design rationale
Every artefact carries a header that states how it was produced. The reader never infers a parameter from context and never guesses.
Earlier revisions left the mode and encoding to the caller's memory. That is a silent-corruption hazard: a ciphertext decrypted under the wrong mode yields garbage that is indistinguishable from a wrong password. Recording the parameters alongside the artefact removes the ambiguity, and rejecting headerless input removes the guesswork entirely.
The consequence is deliberate and absolute: input without a valid header is rejected, not interpreted.
3. Fixed values
These byte sequences are constants of the format. They are reproduced in three representations for convenience; all three denote the same octets.
3.1 Magic
| Representation | Value |
|---|---|
| Bytes | b"69CES" |
| Hex | 3639434553 |
| Decimal byte list | [54, 57, 67, 69, 83] |
Length: 5 octets. Position: the first octet of every artefact.
3.2 Begin marker
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
64 characters of lowercase hexadecimal. Written verbatim; it is not decoded before comparison.
3.3 End marker
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
64 characters of lowercase hexadecimal. Written verbatim.
Note. Both markers are fixed constants of format version 1. They are delimiters, not integrity checks: they carry no signature semantics and do not authenticate the payload. Message integrity, where required, is provided by the cipher and by an external MAC.
4. Container layout
An artefact is a single character string with five regions:
+-------------------------------------------------------------------+
| MAGIC "69CES" 5 octets |
+-------------------------------------------------------------------+
| BEGIN MARKER 64 hex chars |
+-------------------------------------------------------------------+
| CONFIGURATION UTF-8 JSON object variable |
+-------------------------------------------------------------------+
| END MARKER 64 hex chars |
+-------------------------------------------------------------------+
| PAYLOAD key material or ciphertext variable |
+-------------------------------------------------------------------+
The three fixed regions are literals. Only the configuration and the payload vary between artefacts.
4.1 Formal grammar
artifact = MAGIC begin-marker config end-marker payload
MAGIC = "69CES"
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
config = <UTF-8 encoded JSON object, see section 5>
payload = <encoding-dependent text, see section 6>
There is no whitespace, newline or padding between regions. A reader must not skip optional characters.
5. Configuration record
The configuration is a JSON object encoded as UTF-8, with no line breaks.
Keys are sorted; separators are , and : with no surrounding spaces. The
canonical form is therefore deterministic: the same configuration always
renders as the same octets.
5.1 Required fields
A configuration missing any of these is invalid, and the artefact must be rejected rather than partially interpreted.
| Field | Type | Meaning |
|---|---|---|
kind |
string | "ciphertext" or "key" |
v |
integer | Format version; 1 for this specification |
mode |
string | Performance mode, see §5.2 |
mid |
string | Intermediate representation, see §5.3 |
out |
string | Output encoding, see §5.4 |
alg |
string | Algorithm family; "chaoscrypt" |
kdf |
string | Keystream derivation, see §5.5 |
keybits |
integer | Length of the long key in bits |
ksum |
string | Non-secret key digest, see §5.6 |
5.2 mode
| Value | Pipeline |
|---|---|
reference |
Original algorithm, character pipeline |
fast |
Original algorithm, bulk-operation implementation |
turbo |
SHAKE256 keystream, byte pipeline |
raw |
SHAKE256 keystream, byte pipeline with the long-key layer retained |
reference and fast produce identical ciphertexts. The distinction is
recorded so a reader can report which implementation wrote the artefact; it
does not affect decryption.
5.3 mid — intermediate representation
The mid field names how the two XOR layers of the byte pipeline exchange
their data. It is only meaningful for turbo and raw; for the character
pipelines it is descriptive.
| Value | Meaning |
|---|---|
bt |
Intermediate values carried as raw bytes (default) |
hex |
Intermediate values carried as hexadecimal text |
turbo and raw accept either value, and bt is the default. The choice is
recorded in the header and honoured when the artefact is read: a turbo or
raw artefact is decrypted with the mid it carries, and no fallback to the
other representation is attempted.
reference and fast are character pipelines and always use hex. They
ignore mid: whatever value is supplied, the header records hex for them.
Expansion. bt passes the bytes between the XOR layers unchanged, so the
only growth is the output encoding: 2.00x with hex output, 1.33x with
base output. hex re-encodes each layer's output as hexadecimal text
before the next layer consumes it; with the two layers of this format that
doubles the payload at each layer, so the byte payload handed to the output
encoder is about 4x the bt size (roughly 8 hexadecimal characters per
input octet with hex output).
Speed. bt is about 4x faster than hex: it skips the per-layer hex
encode and decode and XORs the shorter byte buffers directly. It is therefore
the default.
5.4 out — output encoding
| Value | Meaning | Expansion |
|---|---|---|
hex |
Two characters per octet | 2.00x |
base |
Base64, four characters per three octets | 1.33x |
reference and fast always emit hex and ignore any attempt to set it.
turbo and raw accept either.
Reader requirement. The payload must be decoded according to
out. Attempting to decode with the wrong encoding must fail, never silently produce a different result.
5.5 kdf — keystream derivation
| Value | Meaning |
|---|---|
expand |
Hand-rolled mixing and rotation expansion |
shake256 |
SHAKE256 extensible-output function |
5.6 ksum — key digest
A 16-character lowercase hexadecimal string: the first 16 characters of
SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key), where ‖ is
concatenation of the ASCII encodings.
It exists so a reader can confirm that a key file and a ciphertext belong together without exposing any usable secret. It is a correlation aid, not a security control: it reveals nothing that helps an attacker, and its absence of secrecy must not be relied upon as an integrity guarantee.
5.7 Legacy fields
Two optional fields exist for reading artefacts produced before the header was
introduced. They are never written by the current library except when wrapping a
legacy artefact, and they are not part of :data:REQUIRED_FIELDS.
| Field | Type | Meaning |
|---|---|---|
is_old |
boolean | The artefact predates this format |
old_is_key |
boolean | With is_old, whether it is a key file (true) or a ciphertext (false) |
Precedence rule. When
is_oldis present and true, every other field is ignored, includingmode,mid,outandkind. The artefact predates the field set, so there is nothing else meaningful to read. A reader must not reject such an artefact for missingkind, nor act on any field besideis_oldandold_is_key.
An artefact that carries neither field is a normal artefact and is read normally. Their absence is not an error.
5.8 Descriptive fields
The record also carries fields that describe the construction. They are informational — a reader may use them for display or diagnostics — and they do not modify the decryption path.
| Field | Meaning |
|---|---|
lib |
Library version that wrote the artefact |
cache |
Keystream cache strategy in effect; does not affect output |
subst |
Substitution kind, "bijective" |
flip |
Case-flip kind, "conditional" |
order |
The pipeline, as an ordered list of stage names |
xor_layers |
Number of XOR layers applied |
compact |
Whether the byte pipeline was used |
keyshort |
Length of the short key in bits |
flipbits |
Number of bits in the flip pattern |
b64 |
Whether Base64 is used as a byte-safety layer |
utf8 |
Whether text is transported as UTF-8 |
5.9 Example
{
"alg": "chaoscrypt",
"b64": true,
"cache": "hash",
"compact": true,
"flip": "conditional",
"flipbits": 10,
"kdf": "shake256",
"keybits": 16384,
"keyshort": 2048,
"kind": "ciphertext",
"ksum": "3f9a1c7e5b2d4086",
"lib": "4.2.6",
"mid": "bt",
"mode": "turbo",
"order": "base64>subst>flip>reverse>xor>xor",
"out": "hex",
"subst": "bijective",
"utf8": true,
"v": 1,
"xor_layers": 2
}
Rendered between the markers, this is a single unbroken line.
6. Payload
6.1 Ciphertext payload
When kind is "ciphertext", the payload is the ciphertext encoded per out:
out = "hex": lowercase hexadecimal, even length, alphabet[0-9a-f].out = "base": standard Base64 with=padding, alphabet[A-Za-z0-9+/].
The payload is the final output of the pipeline named in mode. Nothing is
appended after it.
6.2 Key payload
When kind is "key", the payload is the password-wrapped key material, in the
same two encodings.
A caller reading a key file must check that kind is "key". A key file is not
a ciphertext and must not be accepted where a ciphertext is expected, nor the
reverse. This is enforced, not advisory.
7. Reader conformance
A conforming reader performs these steps, in order, and fails at the first that does not hold. It must not attempt recovery, interpretation or fallback.
| # | Check | Failure |
|---|---|---|
| 1 | Input is non-empty | E017 |
| 2 | Input starts with 69CES |
E017 |
| 3 | The 64 characters after the magic equal the begin marker | E018 |
| 4 | The end marker occurs after the begin marker | E018 |
| 5 | Text between the markers is valid UTF-8 JSON | E018 |
| 6 | That JSON is an object | E018 |
| 7 | Every required field of §5.1 is present | E018 |
| 8 | kind, mid and out hold permitted values |
E018 |
| 9 | kind matches what the caller asked for |
E018 |
| 10 | The payload decodes under out |
E010 |
| 11 | Decryption succeeds under mode and the key |
E011 |
7.1 No fallback
There is no lenient path. An artefact without a header is not a ChaosCrypt
artefact and is rejected with E017. Artefacts written by revisions before the
header existed are not read by this path either: they are recovered only
through the explicit compatibility API, which wraps them in a header first
(see section 12). The ordinary reader never gains a lenient branch.
7.2 No partial trust
A header is either wholly valid or wholly rejected. A reader must not act on fields it did manage to parse, because a partially parsed header may describe a different construction than the one that produced the payload.
8. Error codes
| Code | Name | Raised when |
|---|---|---|
E010 |
MalformedCiphertext |
Payload does not decode under out |
E011 |
DecryptionFailed |
Decryption produced invalid data |
E017 |
MissingMagicHeader |
No magic present; input is not a CES artefact |
E018 |
MalformedHeader |
Marker missing, JSON invalid, field absent or out of range, or kind mismatch |
Codes E001–E016 cover key handling, argument and I/O failures and are
unchanged.
9. Versioning
v is the format version and is independent of the library version in lib.
A reader must reject an artefact whose v it does not implement, with E018.
Adding optional descriptive fields does not change v. Changing the magic, the
markers, the region order, or the meaning of a required field does, and requires
a new major library release.
10. Conformance checklist
An implementation claims conformance when it:
- writes the magic, both markers and the configuration in the order of §4;
- sorts configuration keys and uses
,and:as separators with no spaces; - includes every required field of §5.1;
- rejects input without the magic with
E017; - rejects malformed headers with
E018, never partially trusting them; - enforces the
kindmatch on both read and write paths; - decodes the payload strictly under
out, failing rather than guessing; - implements no lenient or legacy fallback path.
11. Worked examples
11.1 A normal ciphertext
Text that is encrypted in turbo with base64 output, then the layout read
back region by region.
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(mode="turbo", encoding="base64")
cc.load("my.key", "key-file-password")
token = cc.encrypt("meet me at the usual place", "encryption-password")
The token begins with the five magic octets, then the begin marker:
69CES
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
then the configuration, which states the parameters actually used:
{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
"flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
"keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
"lib":"4.2.6","mid":"bt","mode":"turbo",
"order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
"utf8":true,"v":1,"xor_layers":2}
then the end marker, then the base64 payload:
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...
Decrypting needs no configuration at all — the record supplies it:
from chaoscrypt import ChaosCrypt
from chaoscrypt._header import read_header, strip_header
cfg = read_header(token)
print(cfg["mode"], cfg["out"]) # turbo base
# a fresh instance with no settings reads it correctly
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True).decrypt(token, "encryption-password")
11.2 Reading an artefact without knowing anything about it
Because the header is complete, a reader needs only the key file and the password. The mode, the intermediate representation and the output encoding are all taken from the artefact.
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
cc.load("my.key", "key-file-password")
# whatever produced this, it will be read correctly
plain = cc.decrypt(open("message.ces").read(), "encryption-password")
11.3 Inspecting a key file
from chaoscrypt._header import read_header, strip_header
text = open("my.key", encoding="utf-8").read()
cfg = read_header(text)
print(cfg["kind"]) # key
print(cfg["alg"]) # chaoscrypt
print(cfg["keybits"]) # 16384
print(cfg["ksum"]) # 16 hex characters
ksum lets a caller confirm that a key file and a ciphertext belong together
without exposing anything usable:
cs = open("message.ces", encoding="utf-8").read()
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"
11.4 A rejection
from chaoscrypt import ChaosCrypt
from chaoscrypt.errors import ChaosCryptError
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
try:
cc.decrypt("7a3f9c...", "encryption-password") # a bare payload
except ChaosCryptError as e:
print(e.code) # E017
print(e.detail) # MissingMagicHeader
There is no fallback path here. The input is not a CES artefact, and guessing would risk returning garbage that looks like success.
12. Legacy artefacts
Artefacts written before this format existed have no header. They are handled by wrapping, never by loosening the reader.
12.1 What a legacy artefact looks like
| Kind | Content |
|---|---|
| Legacy ciphertext | Hexadecimal text, no prefix of any kind |
| Legacy key file | Password-wrapped key material, no prefix of any kind |
Neither carries a magic value, a marker or a configuration record.
12.2 Wrapping
To read one, a minimal header is synthesised and prefixed. The payload is not touched.
{"is_old":true,"old_is_key":false}
That header is a normal, valid header. Wrapping therefore produces an artefact
that the ordinary path can read, and the ordinary path never gains a lenient
branch: it still rejects headerless input with E017.
from chaoscrypt import ChaosCrypt
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
auto_load=True)
# a legacy ciphertext: no header
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
"encryption-password")
decrypt_legacy first tries the input as it stands. If that reports a missing
header, it wraps the input and tries again. decrypt_legacy_auto additionally
probes whether the artefact is a key file or a ciphertext, so the caller does
not have to say which.
A legacy key file is loaded the same way:
legacy_text = open("old-encryption.key", encoding="utf-8").read()
cc = ChaosCrypt()
cc.load_legacy_keys(legacy_text, "key-file-password")
Or wrapped explicitly, when the artefact is to be stored in the new form:
modernised = cc.wrap_legacy(legacy_text, is_key=True)
open("my.key", "w", encoding="utf-8").write(modernised)
12.3 Which algorithm reads a legacy artefact
is_old is true, so every other field is ignored — there is no mode to
consult. The legacy path tries the bulk-operation implementation first and
falls back to the original character pipeline:
| Attempt | Why |
|---|---|
1. fast |
Same algorithm, faster; correct for anything the bulk path can read |
2. reference |
The original pipeline, in case the artefact needs it |
Both implement the original algorithm, so either can read pre-header material. The fallback costs a second attempt only when the first fails.
12.4 What is not supported
- Writing legacy artefacts. The library only reads them. Output always carries a header.
- Partial trust of a legacy artefact. It is wrapped whole; nothing is inferred from its content.
- Reading a legacy artefact without wrapping. The ordinary path will reject
it with
E017. Use the compatibility API.
12.5 Conformance for readers
An implementation that supports legacy artefacts must:
- detect a headerless input and treat it as legacy, not as an error at the API boundary;
- synthesise a header rather than bypass the parser;
- ignore every field except
is_oldandold_is_keywhenis_oldis true; - keep the ordinary path strict — a headerless input passed to
decryptstill raisesE017; - never write a legacy artefact.
13. Relation to the cipher
The header is metadata. It does not add secrecy, and it is not authenticated. An attacker who modifies the header can cause a decryption to fail, which is a denial of service and no more — the payload cannot be recovered without the key.
Where an application needs to detect tampering with the header as well as the payload, it must compute a MAC over the entire artefact, header included, and verify that MAC before parsing. The format does not do this for you.
See SECURITY.md for what the cipher guarantees and what it does not.