# CES Artifact Format Specification **Normative reference for the ChaosCrypt (CES) container format** | | | |---|---| | Format version | 1 | | Status | Normative | | Applies to | Key files and ciphertexts produced from 4.2.6 onward | | Library | `chaoscrypt` | --- ## 1. Scope This document specifies the byte-level layout of every artefact the library produces. It is the authority for implementers: where this document and the code disagree, this document is wrong and the code is to be fixed. It defines: - the fixed magic and delimiter values; - the layout of the self-describing header; - the complete field set of the configuration record; - the payload that follows; - the conformance requirements a reader must enforce. It does **not** describe the cipher itself. For the algorithm see [SECURITY.md](../SECURITY.md); for performance see [benchmark.md](benchmark.md). --- ## 2. Design rationale Every artefact carries a header that states how it was produced. The reader never infers a parameter from context and never guesses. Earlier revisions left the mode and encoding to the caller's memory. That is a silent-corruption hazard: a ciphertext decrypted under the wrong mode yields garbage that is indistinguishable from a wrong password. Recording the parameters alongside the artefact removes the ambiguity, and rejecting headerless input removes the guesswork entirely. The consequence is deliberate and absolute: **input without a valid header is rejected, not interpreted.** --- ## 3. Fixed values These byte sequences are constants of the format. They are reproduced in three representations for convenience; all three denote the same octets. ### 3.1 Magic | Representation | Value | |---|---| | Bytes | `b"69CES"` | | Hex | `3639434553` | | Decimal byte list | `[54, 57, 67, 69, 83]` | Length: 5 octets. Position: the first octet of every artefact. ### 3.2 Begin marker ``` 385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93 ``` 64 characters of lowercase hexadecimal. Written verbatim; it is **not** decoded before comparison. ### 3.3 End marker ``` 2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2 ``` 64 characters of lowercase hexadecimal. Written verbatim. > **Note.** Both markers are fixed constants of format version 1. They are > delimiters, not integrity checks: they carry no signature semantics and do > not authenticate the payload. Message integrity, where required, is provided > by the cipher and by an external MAC. --- ## 4. Container layout An artefact is a single character string with five regions: ``` +-------------------------------------------------------------------+ | MAGIC "69CES" 5 octets | +-------------------------------------------------------------------+ | BEGIN MARKER 64 hex chars | +-------------------------------------------------------------------+ | CONFIGURATION UTF-8 JSON object variable | +-------------------------------------------------------------------+ | END MARKER 64 hex chars | +-------------------------------------------------------------------+ | PAYLOAD key material or ciphertext variable | +-------------------------------------------------------------------+ ``` The three fixed regions are literals. Only the configuration and the payload vary between artefacts. ### 4.1 Formal grammar ``` artifact = MAGIC begin-marker config end-marker payload MAGIC = "69CES" begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93" end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2" config = payload = ``` There is no whitespace, newline or padding between regions. A reader must not skip optional characters. --- ## 5. Configuration record The configuration is a JSON object encoded as UTF-8, with no line breaks. Keys are sorted; separators are `,` and `:` with no surrounding spaces. The canonical form is therefore deterministic: the same configuration always renders as the same octets. ### 5.1 Required fields A configuration missing any of these is invalid, and the artefact must be rejected rather than partially interpreted. | Field | Type | Meaning | |---|---|---| | `kind` | string | `"ciphertext"` or `"key"` | | `v` | integer | Format version; `1` for this specification | | `mode` | string | Performance mode, see §5.2 | | `mid` | string | Intermediate representation, see §5.3 | | `out` | string | Output encoding, see §5.4 | | `alg` | string | Algorithm family; `"chaoscrypt"` | | `kdf` | string | Keystream derivation, see §5.5 | | `keybits` | integer | Length of the long key in bits | | `ksum` | string | Non-secret key digest, see §5.6 | ### 5.2 `mode` | Value | Pipeline | |---|---| | `reference` | Original algorithm, character pipeline | | `fast` | Original algorithm, bulk-operation implementation | | `turbo` | SHAKE256 keystream, byte pipeline | | `raw` | SHAKE256 keystream, byte pipeline with the long-key layer retained | `reference` and `fast` produce identical ciphertexts. The distinction is recorded so a reader can report which implementation wrote the artefact; it does not affect decryption. ### 5.3 `mid` — intermediate representation The `mid` field names how the two XOR layers of the byte pipeline exchange their data. It is only meaningful for `turbo` and `raw`; for the character pipelines it is descriptive. | Value | Meaning | |---|---| | `bt` | Intermediate values carried as raw bytes (default) | | `hex` | Intermediate values carried as hexadecimal text | `turbo` and `raw` accept either value, and `bt` is the default. The choice is recorded in the header and honoured when the artefact is read: a `turbo` or `raw` artefact is decrypted with the `mid` it carries, and no fallback to the other representation is attempted. `reference` and `fast` are character pipelines and always use `hex`. They ignore `mid`: whatever value is supplied, the header records `hex` for them. **Expansion.** `bt` passes the bytes between the XOR layers unchanged, so the only growth is the output encoding: 2.00x with `hex` output, 1.33x with `base` output. `hex` re-encodes each layer's output as hexadecimal text before the next layer consumes it; with the two layers of this format that doubles the payload at each layer, so the byte payload handed to the output encoder is about **4x** the `bt` size (roughly 8 hexadecimal characters per input octet with `hex` output). **Speed.** `bt` is about **4x faster** than `hex`: it skips the per-layer hex encode and decode and XORs the shorter byte buffers directly. It is therefore the default. ### 5.4 `out` — output encoding | Value | Meaning | Expansion | |---|---|---| | `hex` | Two characters per octet | 2.00x | | `base` | Base64, four characters per three octets | 1.33x | `reference` and `fast` always emit `hex` and ignore any attempt to set it. `turbo` and `raw` accept either. > **Reader requirement.** The payload must be decoded according to `out`. > Attempting to decode with the wrong encoding must fail, never silently produce > a different result. ### 5.5 `kdf` — keystream derivation | Value | Meaning | |---|---| | `expand` | Hand-rolled mixing and rotation expansion | | `shake256` | SHAKE256 extensible-output function | ### 5.6 `ksum` — key digest A 16-character lowercase hexadecimal string: the first 16 characters of `SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key)`, where `‖` is concatenation of the ASCII encodings. It exists so a reader can confirm that a key file and a ciphertext belong together without exposing any usable secret. It is a **correlation aid, not a security control**: it reveals nothing that helps an attacker, and its absence of secrecy must not be relied upon as an integrity guarantee. ### 5.7 Legacy fields Two optional fields exist for reading artefacts produced before the header was introduced. They are never written by the current library except when wrapping a legacy artefact, and they are **not** part of :data:`REQUIRED_FIELDS`. | Field | Type | Meaning | |---|---|---| | `is_old` | boolean | The artefact predates this format | | `old_is_key` | boolean | With `is_old`, whether it is a key file (`true`) or a ciphertext (`false`) | > **Precedence rule.** When `is_old` is present and true, **every other field is > ignored**, including `mode`, `mid`, `out` and `kind`. The artefact predates the > field set, so there is nothing else meaningful to read. A reader must not > reject such an artefact for missing `kind`, nor act on any field beside > `is_old` and `old_is_key`. An artefact that carries neither field is a normal artefact and is read normally. Their absence is not an error. ### 5.8 Descriptive fields The record also carries fields that describe the construction. They are informational — a reader may use them for display or diagnostics — and they do not modify the decryption path. | Field | Meaning | |---|---| | `lib` | Library version that wrote the artefact | | `cache` | Keystream cache strategy in effect; does not affect output | | `subst` | Substitution kind, `"bijective"` | | `flip` | Case-flip kind, `"conditional"` | | `order` | The pipeline, as an ordered list of stage names | | `xor_layers` | Number of XOR layers applied | | `compact` | Whether the byte pipeline was used | | `keyshort` | Length of the short key in bits | | `flipbits` | Number of bits in the flip pattern | | `b64` | Whether Base64 is used as a byte-safety layer | | `utf8` | Whether text is transported as UTF-8 | ### 5.9 Example ```json { "alg": "chaoscrypt", "b64": true, "cache": "hash", "compact": true, "flip": "conditional", "flipbits": 10, "kdf": "shake256", "keybits": 16384, "keyshort": 2048, "kind": "ciphertext", "ksum": "3f9a1c7e5b2d4086", "lib": "4.2.6", "mid": "bt", "mode": "turbo", "order": "base64>subst>flip>reverse>xor>xor", "out": "hex", "subst": "bijective", "utf8": true, "v": 1, "xor_layers": 2 } ``` Rendered between the markers, this is a single unbroken line. --- ## 6. Payload ### 6.1 Ciphertext payload When `kind` is `"ciphertext"`, the payload is the ciphertext encoded per `out`: - `out = "hex"`: lowercase hexadecimal, even length, alphabet `[0-9a-f]`. - `out = "base"`: standard Base64 with `=` padding, alphabet `[A-Za-z0-9+/]`. The payload is the final output of the pipeline named in `mode`. Nothing is appended after it. ### 6.2 Key payload When `kind` is `"key"`, the payload is the password-wrapped key material, in the same two encodings. A caller reading a key file must check that `kind` is `"key"`. A key file is not a ciphertext and must not be accepted where a ciphertext is expected, nor the reverse. This is enforced, not advisory. --- ## 7. Reader conformance A conforming reader performs these steps, in order, and fails at the first that does not hold. It must not attempt recovery, interpretation or fallback. | # | Check | Failure | |---|---|---| | 1 | Input is non-empty | `E017` | | 2 | Input starts with `69CES` | `E017` | | 3 | The 64 characters after the magic equal the begin marker | `E018` | | 4 | The end marker occurs after the begin marker | `E018` | | 5 | Text between the markers is valid UTF-8 JSON | `E018` | | 6 | That JSON is an object | `E018` | | 7 | Every required field of §5.1 is present | `E018` | | 8 | `kind`, `mid` and `out` hold permitted values | `E018` | | 9 | `kind` matches what the caller asked for | `E018` | | 10 | The payload decodes under `out` | `E010` | | 11 | Decryption succeeds under `mode` and the key | `E011` | ### 7.1 No fallback There is no lenient path. An artefact without a header is not a ChaosCrypt artefact and is rejected with `E017`. Artefacts written by revisions before the header existed are not read by this path either: they are recovered only through the explicit compatibility API, which wraps them in a header first (see section 12). The ordinary reader never gains a lenient branch. ### 7.2 No partial trust A header is either wholly valid or wholly rejected. A reader must not act on fields it did manage to parse, because a partially parsed header may describe a different construction than the one that produced the payload. --- ## 8. Error codes | Code | Name | Raised when | |---|---|---| | `E010` | `MalformedCiphertext` | Payload does not decode under `out` | | `E011` | `DecryptionFailed` | Decryption produced invalid data | | `E017` | `MissingMagicHeader` | No magic present; input is not a CES artefact | | `E018` | `MalformedHeader` | Marker missing, JSON invalid, field absent or out of range, or `kind` mismatch | Codes `E001`–`E016` cover key handling, argument and I/O failures and are unchanged. --- ## 9. Versioning `v` is the format version and is independent of the library version in `lib`. A reader must reject an artefact whose `v` it does not implement, with `E018`. Adding optional descriptive fields does not change `v`. Changing the magic, the markers, the region order, or the meaning of a required field does, and requires a new major library release. --- ## 10. Conformance checklist An implementation claims conformance when it: - [ ] writes the magic, both markers and the configuration in the order of §4; - [ ] sorts configuration keys and uses `,` and `:` as separators with no spaces; - [ ] includes every required field of §5.1; - [ ] rejects input without the magic with `E017`; - [ ] rejects malformed headers with `E018`, never partially trusting them; - [ ] enforces the `kind` match on both read and write paths; - [ ] decodes the payload strictly under `out`, failing rather than guessing; - [ ] implements no lenient or legacy fallback path. --- --- ## 11. Worked examples ### 11.1 A normal ciphertext Text that is encrypted in `turbo` with `base64` output, then the layout read back region by region. ```python from chaoscrypt import ChaosCrypt cc = ChaosCrypt(mode="turbo", encoding="base64") cc.load("my.key", "key-file-password") token = cc.encrypt("meet me at the usual place", "encryption-password") ``` The token begins with the five magic octets, then the begin marker: ``` 69CES 385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93 ``` then the configuration, which states the parameters actually used: ```json {"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true, "flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384, "keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261", "lib":"4.2.6","mid":"bt","mode":"turbo", "order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective", "utf8":true,"v":1,"xor_layers":2} ``` then the end marker, then the base64 payload: ``` 2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2 IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla... ``` Decrypting needs no configuration at all — the record supplies it: ```python from chaoscrypt import ChaosCrypt from chaoscrypt._header import read_header, strip_header cfg = read_header(token) print(cfg["mode"], cfg["out"]) # turbo base # a fresh instance with no settings reads it correctly plain = ChaosCrypt(key_file="my.key", key_password="key-file-password", auto_load=True).decrypt(token, "encryption-password") ``` ### 11.2 Reading an artefact without knowing anything about it Because the header is complete, a reader needs only the key file and the password. The mode, the intermediate representation and the output encoding are all taken from the artefact. ```python from chaoscrypt import ChaosCrypt cc = ChaosCrypt(key_file="my.key", key_password="key-file-password", auto_load=True) cc.load("my.key", "key-file-password") # whatever produced this, it will be read correctly plain = cc.decrypt(open("message.ces").read(), "encryption-password") ``` ### 11.3 Inspecting a key file ```python from chaoscrypt._header import read_header, strip_header text = open("my.key", encoding="utf-8").read() cfg = read_header(text) print(cfg["kind"]) # key print(cfg["alg"]) # chaoscrypt print(cfg["keybits"]) # 16384 print(cfg["ksum"]) # 16 hex characters ``` `ksum` lets a caller confirm that a key file and a ciphertext belong together without exposing anything usable: ```python cs = open("message.ces", encoding="utf-8").read() assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file" ``` ### 11.4 A rejection ```python from chaoscrypt import ChaosCrypt from chaoscrypt.errors import ChaosCryptError cc = ChaosCrypt(key_file="my.key", key_password="key-file-password", auto_load=True) try: cc.decrypt("7a3f9c...", "encryption-password") # a bare payload except ChaosCryptError as e: print(e.code) # E017 print(e.detail) # MissingMagicHeader ``` There is no fallback path here. The input is not a CES artefact, and guessing would risk returning garbage that looks like success. --- ## 12. Legacy artefacts Artefacts written before this format existed have no header. They are handled by **wrapping**, never by loosening the reader. ### 12.1 What a legacy artefact looks like | Kind | Content | |---|---| | Legacy ciphertext | Hexadecimal text, no prefix of any kind | | Legacy key file | Password-wrapped key material, no prefix of any kind | Neither carries a magic value, a marker or a configuration record. ### 12.2 Wrapping To read one, a minimal header is synthesised and prefixed. The payload is not touched. ```json {"is_old":true,"old_is_key":false} ``` That header is a normal, valid header. Wrapping therefore produces an artefact that the ordinary path can read, and **the ordinary path never gains a lenient branch**: it still rejects headerless input with `E017`. ```python from chaoscrypt import ChaosCrypt cc = ChaosCrypt(key_file="my.key", key_password="key-file-password", auto_load=True) # a legacy ciphertext: no header plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69", "encryption-password") ``` `decrypt_legacy` first tries the input as it stands. If that reports a missing header, it wraps the input and tries again. `decrypt_legacy_auto` additionally probes whether the artefact is a key file or a ciphertext, so the caller does not have to say which. A legacy key file is loaded the same way: ```python legacy_text = open("old-encryption.key", encoding="utf-8").read() cc = ChaosCrypt() cc.load_legacy_keys(legacy_text, "key-file-password") ``` Or wrapped explicitly, when the artefact is to be stored in the new form: ```python modernised = cc.wrap_legacy(legacy_text, is_key=True) open("my.key", "w", encoding="utf-8").write(modernised) ``` ### 12.3 Which algorithm reads a legacy artefact `is_old` is true, so every other field is ignored — there is no `mode` to consult. The legacy path tries the bulk-operation implementation first and falls back to the original character pipeline: | Attempt | Why | |---|---| | 1. `fast` | Same algorithm, faster; correct for anything the bulk path can read | | 2. `reference` | The original pipeline, in case the artefact needs it | Both implement the original algorithm, so either can read pre-header material. The fallback costs a second attempt only when the first fails. ### 12.4 What is not supported - **Writing legacy artefacts.** The library only reads them. Output always carries a header. - **Partial trust of a legacy artefact.** It is wrapped whole; nothing is inferred from its content. - **Reading a legacy artefact without wrapping.** The ordinary path will reject it with `E017`. Use the compatibility API. ### 12.5 Conformance for readers An implementation that supports legacy artefacts must: - [ ] detect a headerless input and treat it as legacy, not as an error at the API boundary; - [ ] synthesise a header rather than bypass the parser; - [ ] ignore every field except `is_old` and `old_is_key` when `is_old` is true; - [ ] keep the ordinary path strict — a headerless input passed to `decrypt` still raises `E017`; - [ ] never write a legacy artefact. --- ## 13. Relation to the cipher The header is metadata. It does not add secrecy, and it is not authenticated. An attacker who modifies the header can cause a decryption to fail, which is a denial of service and no more — the payload cannot be recovered without the key. Where an application needs to detect tampering with the header as well as the payload, it must compute a MAC over the **entire artefact**, header included, and verify that MAC before parsing. The format does not do this for you. See [SECURITY.md](../SECURITY.md) for what the cipher guarantees and what it does not.