Files
ChaosCrypt/docs/FORMAT.md
T
dvs dd49043171 v4.2.6: Self-describing format, byte pipeline, and a performance pass
Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
2026-09-27 22:06:34 +08:00

20 KiB
Raw Blame History

CES Artifact Format Specification

Normative reference for the ChaosCrypt (CES) container format

Format version 1
Status Normative
Applies to Key files and ciphertexts produced from 4.2.6 onward
Library chaoscrypt

1. Scope

This document specifies the byte-level layout of every artefact the library produces. It is the authority for implementers: where this document and the code disagree, this document is wrong and the code is to be fixed.

It defines:

  • the fixed magic and delimiter values;
  • the layout of the self-describing header;
  • the complete field set of the configuration record;
  • the payload that follows;
  • the conformance requirements a reader must enforce.

It does not describe the cipher itself. For the algorithm see SECURITY.md; for performance see benchmark.md.


2. Design rationale

Every artefact carries a header that states how it was produced. The reader never infers a parameter from context and never guesses.

Earlier revisions left the mode and encoding to the caller's memory. That is a silent-corruption hazard: a ciphertext decrypted under the wrong mode yields garbage that is indistinguishable from a wrong password. Recording the parameters alongside the artefact removes the ambiguity, and rejecting headerless input removes the guesswork entirely.

The consequence is deliberate and absolute: input without a valid header is rejected, not interpreted.


3. Fixed values

These byte sequences are constants of the format. They are reproduced in three representations for convenience; all three denote the same octets.

3.1 Magic

Representation Value
Bytes b"69CES"
Hex 3639434553
Decimal byte list [54, 57, 67, 69, 83]

Length: 5 octets. Position: the first octet of every artefact.

3.2 Begin marker

385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93

64 characters of lowercase hexadecimal. Written verbatim; it is not decoded before comparison.

3.3 End marker

2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2

64 characters of lowercase hexadecimal. Written verbatim.

Note. Both markers are fixed constants of format version 1. They are delimiters, not integrity checks: they carry no signature semantics and do not authenticate the payload. Message integrity, where required, is provided by the cipher and by an external MAC.


4. Container layout

An artefact is a single character string with five regions:

+-------------------------------------------------------------------+
|  MAGIC            "69CES"                         5 octets        |
+-------------------------------------------------------------------+
|  BEGIN MARKER     64 hex chars                                    |
+-------------------------------------------------------------------+
|  CONFIGURATION    UTF-8 JSON object              variable          |
+-------------------------------------------------------------------+
|  END MARKER       64 hex chars                                    |
+-------------------------------------------------------------------+
|  PAYLOAD          key material or ciphertext     variable          |
+-------------------------------------------------------------------+

The three fixed regions are literals. Only the configuration and the payload vary between artefacts.

4.1 Formal grammar

artifact    = MAGIC begin-marker config end-marker payload
MAGIC       = "69CES"
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
end-marker  = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
config      = <UTF-8 encoded JSON object, see section 5>
payload     = <encoding-dependent text, see section 6>

There is no whitespace, newline or padding between regions. A reader must not skip optional characters.


5. Configuration record

The configuration is a JSON object encoded as UTF-8, with no line breaks. Keys are sorted; separators are , and : with no surrounding spaces. The canonical form is therefore deterministic: the same configuration always renders as the same octets.

5.1 Required fields

A configuration missing any of these is invalid, and the artefact must be rejected rather than partially interpreted.

Field Type Meaning
kind string "ciphertext" or "key"
v integer Format version; 1 for this specification
mode string Performance mode, see §5.2
mid string Intermediate representation, see §5.3
out string Output encoding, see §5.4
alg string Algorithm family; "chaoscrypt"
kdf string Keystream derivation, see §5.5
keybits integer Length of the long key in bits
ksum string Non-secret key digest, see §5.6

5.2 mode

Value Pipeline
reference Original algorithm, character pipeline
fast Original algorithm, bulk-operation implementation
turbo SHAKE256 keystream, byte pipeline
raw SHAKE256 keystream, byte pipeline with the long-key layer retained

reference and fast produce identical ciphertexts. The distinction is recorded so a reader can report which implementation wrote the artefact; it does not affect decryption.

5.3 mid — intermediate representation

The mid field names how the two XOR layers of the byte pipeline exchange their data. It is only meaningful for turbo and raw; for the character pipelines it is descriptive.

Value Meaning
bt Intermediate values carried as raw bytes (default)
hex Intermediate values carried as hexadecimal text

turbo and raw accept either value, and bt is the default. The choice is recorded in the header and honoured when the artefact is read: a turbo or raw artefact is decrypted with the mid it carries, and no fallback to the other representation is attempted.

reference and fast are character pipelines and always use hex. They ignore mid: whatever value is supplied, the header records hex for them.

Expansion. bt passes the bytes between the XOR layers unchanged, so the only growth is the output encoding: 2.00x with hex output, 1.33x with base output. hex re-encodes each layer's output as hexadecimal text before the next layer consumes it; with the two layers of this format that doubles the payload at each layer, so the byte payload handed to the output encoder is about 4x the bt size (roughly 8 hexadecimal characters per input octet with hex output).

Speed. bt is about 4x faster than hex: it skips the per-layer hex encode and decode and XORs the shorter byte buffers directly. It is therefore the default.

5.4 out — output encoding

Value Meaning Expansion
hex Two characters per octet 2.00x
base Base64, four characters per three octets 1.33x

reference and fast always emit hex and ignore any attempt to set it. turbo and raw accept either.

Reader requirement. The payload must be decoded according to out. Attempting to decode with the wrong encoding must fail, never silently produce a different result.

5.5 kdf — keystream derivation

Value Meaning
expand Hand-rolled mixing and rotation expansion
shake256 SHAKE256 extensible-output function

5.6 ksum — key digest

A 16-character lowercase hexadecimal string: the first 16 characters of SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key), where ‖ is concatenation of the ASCII encodings.

It exists so a reader can confirm that a key file and a ciphertext belong together without exposing any usable secret. It is a correlation aid, not a security control: it reveals nothing that helps an attacker, and its absence of secrecy must not be relied upon as an integrity guarantee.

5.7 Legacy fields

Two optional fields exist for reading artefacts produced before the header was introduced. They are never written by the current library except when wrapping a legacy artefact, and they are not part of :data:REQUIRED_FIELDS.

Field Type Meaning
is_old boolean The artefact predates this format
old_is_key boolean With is_old, whether it is a key file (true) or a ciphertext (false)

Precedence rule. When is_old is present and true, every other field is ignored, including mode, mid, out and kind. The artefact predates the field set, so there is nothing else meaningful to read. A reader must not reject such an artefact for missing kind, nor act on any field beside is_old and old_is_key.

An artefact that carries neither field is a normal artefact and is read normally. Their absence is not an error.

5.8 Descriptive fields

The record also carries fields that describe the construction. They are informational — a reader may use them for display or diagnostics — and they do not modify the decryption path.

Field Meaning
lib Library version that wrote the artefact
cache Keystream cache strategy in effect; does not affect output
subst Substitution kind, "bijective"
flip Case-flip kind, "conditional"
order The pipeline, as an ordered list of stage names
xor_layers Number of XOR layers applied
compact Whether the byte pipeline was used
keyshort Length of the short key in bits
flipbits Number of bits in the flip pattern
b64 Whether Base64 is used as a byte-safety layer
utf8 Whether text is transported as UTF-8

5.9 Example

{
  "alg": "chaoscrypt",
  "b64": true,
  "cache": "hash",
  "compact": true,
  "flip": "conditional",
  "flipbits": 10,
  "kdf": "shake256",
  "keybits": 16384,
  "keyshort": 2048,
  "kind": "ciphertext",
  "ksum": "3f9a1c7e5b2d4086",
  "lib": "4.2.6",
  "mid": "bt",
  "mode": "turbo",
  "order": "base64>subst>flip>reverse>xor>xor",
  "out": "hex",
  "subst": "bijective",
  "utf8": true,
  "v": 1,
  "xor_layers": 2
}

Rendered between the markers, this is a single unbroken line.


6. Payload

6.1 Ciphertext payload

When kind is "ciphertext", the payload is the ciphertext encoded per out:

  • out = "hex": lowercase hexadecimal, even length, alphabet [0-9a-f].
  • out = "base": standard Base64 with = padding, alphabet [A-Za-z0-9+/].

The payload is the final output of the pipeline named in mode. Nothing is appended after it.

6.2 Key payload

When kind is "key", the payload is the password-wrapped key material, in the same two encodings.

A caller reading a key file must check that kind is "key". A key file is not a ciphertext and must not be accepted where a ciphertext is expected, nor the reverse. This is enforced, not advisory.


7. Reader conformance

A conforming reader performs these steps, in order, and fails at the first that does not hold. It must not attempt recovery, interpretation or fallback.

# Check Failure
1 Input is non-empty E017
2 Input starts with 69CES E017
3 The 64 characters after the magic equal the begin marker E018
4 The end marker occurs after the begin marker E018
5 Text between the markers is valid UTF-8 JSON E018
6 That JSON is an object E018
7 Every required field of §5.1 is present E018
8 kind, mid and out hold permitted values E018
9 kind matches what the caller asked for E018
10 The payload decodes under out E010
11 Decryption succeeds under mode and the key E011

7.1 No fallback

There is no lenient path. An artefact without a header is not a ChaosCrypt artefact and is rejected with E017. Artefacts written by revisions before the header existed are not read by this path either: they are recovered only through the explicit compatibility API, which wraps them in a header first (see section 12). The ordinary reader never gains a lenient branch.

7.2 No partial trust

A header is either wholly valid or wholly rejected. A reader must not act on fields it did manage to parse, because a partially parsed header may describe a different construction than the one that produced the payload.


8. Error codes

Code Name Raised when
E010 MalformedCiphertext Payload does not decode under out
E011 DecryptionFailed Decryption produced invalid data
E017 MissingMagicHeader No magic present; input is not a CES artefact
E018 MalformedHeader Marker missing, JSON invalid, field absent or out of range, or kind mismatch

Codes E001–E016 cover key handling, argument and I/O failures and are unchanged.


9. Versioning

v is the format version and is independent of the library version in lib.

A reader must reject an artefact whose v it does not implement, with E018. Adding optional descriptive fields does not change v. Changing the magic, the markers, the region order, or the meaning of a required field does, and requires a new major library release.


10. Conformance checklist

An implementation claims conformance when it:

  • writes the magic, both markers and the configuration in the order of §4;
  • sorts configuration keys and uses , and : as separators with no spaces;
  • includes every required field of §5.1;
  • rejects input without the magic with E017;
  • rejects malformed headers with E018, never partially trusting them;
  • enforces the kind match on both read and write paths;
  • decodes the payload strictly under out, failing rather than guessing;
  • implements no lenient or legacy fallback path.


11. Worked examples

11.1 A normal ciphertext

Text that is encrypted in turbo with base64 output, then the layout read back region by region.

from chaoscrypt import ChaosCrypt

cc = ChaosCrypt(mode="turbo", encoding="base64")
cc.load("my.key", "key-file-password")

token = cc.encrypt("meet me at the usual place", "encryption-password")

The token begins with the five magic octets, then the begin marker:

69CES
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93

then the configuration, which states the parameters actually used:

{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
 "flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
 "keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
 "lib":"4.2.6","mid":"bt","mode":"turbo",
 "order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
 "utf8":true,"v":1,"xor_layers":2}

then the end marker, then the base64 payload:

2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...

Decrypting needs no configuration at all — the record supplies it:

from chaoscrypt import ChaosCrypt
from chaoscrypt._header import read_header, strip_header

cfg = read_header(token)
print(cfg["mode"], cfg["out"])      # turbo base

# a fresh instance with no settings reads it correctly
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
                   auto_load=True).decrypt(token, "encryption-password")

11.2 Reading an artefact without knowing anything about it

Because the header is complete, a reader needs only the key file and the password. The mode, the intermediate representation and the output encoding are all taken from the artefact.

from chaoscrypt import ChaosCrypt

cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
                auto_load=True)
cc.load("my.key", "key-file-password")

# whatever produced this, it will be read correctly
plain = cc.decrypt(open("message.ces").read(), "encryption-password")

11.3 Inspecting a key file

from chaoscrypt._header import read_header, strip_header

text = open("my.key", encoding="utf-8").read()
cfg = read_header(text)

print(cfg["kind"])        # key
print(cfg["alg"])         # chaoscrypt
print(cfg["keybits"])     # 16384
print(cfg["ksum"])        # 16 hex characters

ksum lets a caller confirm that a key file and a ciphertext belong together without exposing anything usable:

cs = open("message.ces", encoding="utf-8").read()
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"

11.4 A rejection

from chaoscrypt import ChaosCrypt
from chaoscrypt.errors import ChaosCryptError

cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
                auto_load=True)

try:
    cc.decrypt("7a3f9c...", "encryption-password")   # a bare payload
except ChaosCryptError as e:
    print(e.code)     # E017
    print(e.detail)   # MissingMagicHeader

There is no fallback path here. The input is not a CES artefact, and guessing would risk returning garbage that looks like success.


12. Legacy artefacts

Artefacts written before this format existed have no header. They are handled by wrapping, never by loosening the reader.

12.1 What a legacy artefact looks like

Kind Content
Legacy ciphertext Hexadecimal text, no prefix of any kind
Legacy key file Password-wrapped key material, no prefix of any kind

Neither carries a magic value, a marker or a configuration record.

12.2 Wrapping

To read one, a minimal header is synthesised and prefixed. The payload is not touched.

{"is_old":true,"old_is_key":false}

That header is a normal, valid header. Wrapping therefore produces an artefact that the ordinary path can read, and the ordinary path never gains a lenient branch: it still rejects headerless input with E017.

from chaoscrypt import ChaosCrypt

cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
                auto_load=True)

# a legacy ciphertext: no header
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
                          "encryption-password")

decrypt_legacy first tries the input as it stands. If that reports a missing header, it wraps the input and tries again. decrypt_legacy_auto additionally probes whether the artefact is a key file or a ciphertext, so the caller does not have to say which.

A legacy key file is loaded the same way:

legacy_text = open("old-encryption.key", encoding="utf-8").read()
cc = ChaosCrypt()
cc.load_legacy_keys(legacy_text, "key-file-password")

Or wrapped explicitly, when the artefact is to be stored in the new form:

modernised = cc.wrap_legacy(legacy_text, is_key=True)
open("my.key", "w", encoding="utf-8").write(modernised)

12.3 Which algorithm reads a legacy artefact

is_old is true, so every other field is ignored — there is no mode to consult. The legacy path tries the bulk-operation implementation first and falls back to the original character pipeline:

Attempt Why
1. fast Same algorithm, faster; correct for anything the bulk path can read
2. reference The original pipeline, in case the artefact needs it

Both implement the original algorithm, so either can read pre-header material. The fallback costs a second attempt only when the first fails.

12.4 What is not supported

  • Writing legacy artefacts. The library only reads them. Output always carries a header.
  • Partial trust of a legacy artefact. It is wrapped whole; nothing is inferred from its content.
  • Reading a legacy artefact without wrapping. The ordinary path will reject it with E017. Use the compatibility API.

12.5 Conformance for readers

An implementation that supports legacy artefacts must:

  • detect a headerless input and treat it as legacy, not as an error at the API boundary;
  • synthesise a header rather than bypass the parser;
  • ignore every field except is_old and old_is_key when is_old is true;
  • keep the ordinary path strict — a headerless input passed to decrypt still raises E017;
  • never write a legacy artefact.

13. Relation to the cipher

The header is metadata. It does not add secrecy, and it is not authenticated. An attacker who modifies the header can cause a decryption to fail, which is a denial of service and no more — the payload cannot be recovered without the key.

Where an application needs to detect tampering with the header as well as the payload, it must compute a MAC over the entire artefact, header included, and verify that MAC before parsing. The format does not do this for you.

See SECURITY.md for what the cipher guarantees and what it does not.