v4.2.6: Self-describing format, byte pipeline, and a performance pass
Every key file and ciphertext now carries a header that states how it was produced: magic, two fixed markers, and a complete JSON configuration record. A reader takes its parameters from the artefact rather than from the caller. This is an incompatible format change. Headerless input is rejected with E017; pre-4.2.6 material is read through a compatibility API that wraps it in a minimal header and then uses the ordinary path, so the strict path stays strict. Added - Self-describing header on all artefacts, with a normative specification in docs/FORMAT.md - E017 MissingMagicHeader and E018 MalformedHeader - Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising throughput about 4x. The old behaviour is available as mid="hex". - Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x) - Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto, load_legacy_keys - Frozen reference vectors covering every mode, encoding and intermediate - docs/benchmark.md with measured throughput and the profiling behind it Changed - Default mode is fast: byte-for-byte compatible with the original algorithm, roughly 30x quicker - Substitution, case flipping and the printable fold use bytes.translate over precomputed tables - XOR layers use one big-integer operation instead of a per-byte loop - Keystream derivation caches length-independent base material; expansion uses cached translation tables and list accumulation rather than str += - _get_dynamic_key repeats by multiplication. The loop version was quadratic: covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged. - _derive_final_key decodes only the prefix the output consumes - The decrypt path no longer copies or scans the whole artefact: a redundant strip(), a full-string int(payload, 16), and three whole-string slices in strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s. - Documentation rewritten for 4.2.6; all eleven markdown files are English Fixed 1. Key derivation is no longer quadratic (_get_dynamic_key and _expand_keystream_slow shared the str += pattern) 2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact written with mid="hex" could not be read back 3. change_key_password and export_key_text wrote artefacts without a header 4. The constructor accepted encoding="base64" while the header records out="base"; both spellings are now accepted 5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext) rather than E011 (DecryptionFailed) 6. The TUI encoding menu referenced a missing translation Security - Format break from pre-4.2.6 artefacts, documented rather than accidental - fast is not a weaker mode: the frozen vectors assert identical output - The header adds no secrecy and is not authenticated; an application that must detect header tampering should MAC the whole artefact Performance - raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s decryption, ciphertext 2.00x the payload - Throughput holds steady from 64 KiB to 1 GiB - The original implementation managed 0.08 MB/s
This commit is contained in:
+122
@@ -21,6 +21,12 @@ from chaoscrypt import ChaosCryptError # base error class
|
||||
| `decrypt` | function | One-shot decrypt using a key file |
|
||||
| `create_key` | function | One-shot key file generation |
|
||||
| `DEFAULT_KEY_FILE` | constant | `"encryption.key"` |
|
||||
| `MODES` | constant | Tuple of the four mode names |
|
||||
| `DEFAULT_MODE` | constant | `"fast"` |
|
||||
| `MODE_REFERENCE` | constant | `"reference"` |
|
||||
| `MODE_FAST` | constant | `"fast"` |
|
||||
| `MODE_TURBO` | constant | `"turbo"` |
|
||||
| `MODE_RAW` | constant | `"raw"` |
|
||||
| `KEYFILE_VERSION` | constant | On-disk key file format version (`1`) |
|
||||
| `__version__` | constant | Library version string |
|
||||
| `__license__` | constant | `"MIT"` |
|
||||
@@ -50,6 +56,9 @@ ChaosCrypt(
|
||||
| `key_password` | `str \| None` | `None` | Password protecting that key file |
|
||||
| `auto_load` | `bool` | `False` | Load the key file during construction |
|
||||
| `default_key_file` | `str` | `"encryption.key"` | File name used when `key_file` is omitted |
|
||||
| `mode` | `str` | `"fast"` | Performance mode: `reference`, `fast`, `turbo` or `raw` |
|
||||
| `encoding` | `str` | `"hex"` | Output encoding for `turbo`/`raw`: `hex` or `base64` (also accepted as `base`) |
|
||||
| `mid` | `str` | `"bt"` | Intermediate representation for `turbo`/`raw`: `bt` or `hex` (also accepted as `bytes`) |
|
||||
|
||||
When `auto_load=True` and the file cannot be loaded, the corresponding
|
||||
exception is raised from the constructor.
|
||||
@@ -292,6 +301,117 @@ compatibility guarantee.
|
||||
|
||||
---
|
||||
|
||||
## Performance modes
|
||||
|
||||
Four modes, plus two settings that apply to the two byte-pipeline modes.
|
||||
|
||||
| Constant | Value | Pipeline | Ciphertext |
|
||||
|---|---|---|---|
|
||||
| `MODE_REFERENCE` | `"reference"` | original algorithm, character pipeline | hex, 5.33x |
|
||||
| `MODE_FAST` | `"fast"` | same algorithm, bulk operations | hex, 5.33x |
|
||||
| `MODE_TURBO` | `"turbo"` | SHAKE256 keystream, byte pipeline | 2.00x / 1.33x |
|
||||
| `MODE_RAW` | `"raw"` | SHAKE256 plus the long-key layer | 2.00x / 1.33x |
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt, MODES, DEFAULT_MODE
|
||||
|
||||
print(MODES) # ('reference', 'fast', 'turbo', 'raw')
|
||||
print(DEFAULT_MODE) # 'fast'
|
||||
```
|
||||
|
||||
### Settings for the byte-pipeline modes
|
||||
|
||||
| Constant | Value | Applies to | Meaning |
|
||||
|---|---|---|---|
|
||||
| `ENC_HEX` | `"hex"` | `turbo`, `raw` | Two characters per octet (2.00x) |
|
||||
| `ENC_BASE64` | `"base64"` | `turbo`, `raw` | Four characters per three octets (1.33x) |
|
||||
| `MID_BYTES` | `"bt"` | `turbo`, `raw` | XOR layers exchange raw bytes |
|
||||
| `MID_HEX` | `"hex"` | `turbo`, `raw` | XOR layers exchange hexadecimal text |
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
|
||||
cc = ChaosCrypt(mode="raw", encoding="base64", mid="bt")
|
||||
cc.load("my.key", "key-password")
|
||||
|
||||
token = cc.encrypt("hello", "user-password")
|
||||
```
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCryptCore, MODE_TURBO, ENC_BASE64, MID_HEX
|
||||
|
||||
core = ChaosCryptCore(MODE_TURBO, "hash", ENC_BASE64, MID_HEX)
|
||||
core.generate_keymaterial()
|
||||
token = core.encrypt_text("hello", "password")
|
||||
assert core.decrypt_text(token, "password") == "hello"
|
||||
```
|
||||
|
||||
### Which mode to choose
|
||||
|
||||
| If you… | Use |
|
||||
|---|---|
|
||||
| Are not sure | `fast` — the default |
|
||||
| Need maximum speed from a standardised keystream primitive | `raw` with `mid="bt"` |
|
||||
| Need the smallest ciphertext | `turbo` or `raw` with `encoding="base64"` |
|
||||
| Need to interoperate with the character pipeline's shape | `mid="hex"` |
|
||||
| Need a payload byte-compatible with pre-4.2.6 material | `reference` or `fast` |
|
||||
|
||||
`fast` performs the same arithmetic as `reference` and produces the same bytes;
|
||||
only the implementation differs. `turbo` and `raw` change the algorithm and
|
||||
define their own formats.
|
||||
|
||||
> **Artefacts are not interchangeable between mode families**, and `turbo`/`raw`
|
||||
> artefacts are also specific to their `encoding` and `mid`. The header records
|
||||
> all three, so a reader configured differently still reads the artefact
|
||||
> correctly — the ordinary decrypt takes its parameters from the header.
|
||||
|
||||
An unrecognised `mode`, `encoding` or `mid` raises `ValueError` at construction.
|
||||
|
||||
## Reading pre-4.2.6 artefacts
|
||||
|
||||
Artefacts written before the header existed have none, and are rejected by the
|
||||
ordinary path with `E017`. They are read through the compatibility API, which
|
||||
synthesises a minimal header and then uses the ordinary path.
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
|
||||
cc = ChaosCrypt(key_file="my.key", key_password="kf-pwd", auto_load=True)
|
||||
|
||||
# a ciphertext with no header
|
||||
plain = cc.decrypt_legacy(old_ciphertext, "user-password")
|
||||
|
||||
# if you know it is a legacy key file
|
||||
cc.load_legacy_keys(old_key_text, "key-password")
|
||||
|
||||
# let the library work out which it is
|
||||
plain = cc.decrypt_legacy_auto(old_ciphertext, "user-password")
|
||||
|
||||
# migrate once, then use the modern path
|
||||
open("new.key", "w", encoding="utf-8").write(
|
||||
cc.wrap_legacy(old_key_text, is_key=True)
|
||||
)
|
||||
```
|
||||
|
||||
| Method | Purpose |
|
||||
|---|---|
|
||||
| `wrap_legacy(text, is_key=False)` | Prefix a minimal header; the payload is untouched. Idempotent. |
|
||||
| `decrypt_legacy(text, password, is_key=False)` | Read an artefact that may predate the header |
|
||||
| `decrypt_legacy_auto(text, password)` | As above, probing whether the input is a key file or a ciphertext |
|
||||
| `load_legacy_keys(text, key_password)` | Load a pre-header key file |
|
||||
|
||||
`is_old` in the synthesised header means every other field is ignored; the
|
||||
legacy path tries `fast` and falls back to `reference`. Both implement the
|
||||
original algorithm.
|
||||
|
||||
> The compatibility API is a **wrapper** around the ordinary path, never a
|
||||
> lenient branch inside it. Passing headerless input to `decrypt` still raises
|
||||
> `E017`.
|
||||
|
||||
See [FORMAT.md §12](FORMAT.md) for the legacy standard.
|
||||
|
||||
---
|
||||
|
||||
## Error hierarchy
|
||||
|
||||
Every error is a subclass of `ChaosCryptError`, so a single `except` clause
|
||||
@@ -326,6 +446,8 @@ except ChaosCryptError as e:
|
||||
| `E014` | `ChaosCrypt_ERR_FileReadFailed_E014` | Reading a file failed |
|
||||
| `E015` | `ChaosCrypt_ERR_FileWriteFailed_E015` | Writing a file failed |
|
||||
| `E016` | `ChaosCrypt_ERR_UnsupportedKeyVersion_E016` | Key file version not supported |
|
||||
| `E017` | `ChaosCrypt_ERR_MissingMagicHeader_E017` | Input is not a CES artefact |
|
||||
| `E018` | `ChaosCrypt_ERR_MalformedHeader_E018` | Header unusable, or `kind` mismatch |
|
||||
|
||||
Codes are stable and safe to branch on. The human-readable portion of the
|
||||
message may be reworded between minor releases.
|
||||
|
||||
+618
@@ -0,0 +1,618 @@
|
||||
# CES Artifact Format Specification
|
||||
|
||||
**Normative reference for the ChaosCrypt (CES) container format**
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Format version | 1 |
|
||||
| Status | Normative |
|
||||
| Applies to | Key files and ciphertexts produced from 4.2.6 onward |
|
||||
| Library | `chaoscrypt` |
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
This document specifies the byte-level layout of every artefact the library
|
||||
produces. It is the authority for implementers: where this document and the
|
||||
code disagree, this document is wrong and the code is to be fixed.
|
||||
|
||||
It defines:
|
||||
|
||||
- the fixed magic and delimiter values;
|
||||
- the layout of the self-describing header;
|
||||
- the complete field set of the configuration record;
|
||||
- the payload that follows;
|
||||
- the conformance requirements a reader must enforce.
|
||||
|
||||
It does **not** describe the cipher itself. For the algorithm see
|
||||
[SECURITY.md](../SECURITY.md); for performance see [benchmark.md](benchmark.md).
|
||||
|
||||
---
|
||||
|
||||
## 2. Design rationale
|
||||
|
||||
Every artefact carries a header that states how it was produced. The reader
|
||||
never infers a parameter from context and never guesses.
|
||||
|
||||
Earlier revisions left the mode and encoding to the caller's memory. That is a
|
||||
silent-corruption hazard: a ciphertext decrypted under the wrong mode yields
|
||||
garbage that is indistinguishable from a wrong password. Recording the
|
||||
parameters alongside the artefact removes the ambiguity, and rejecting
|
||||
headerless input removes the guesswork entirely.
|
||||
|
||||
The consequence is deliberate and absolute: **input without a valid header is
|
||||
rejected, not interpreted.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Fixed values
|
||||
|
||||
These byte sequences are constants of the format. They are reproduced in three
|
||||
representations for convenience; all three denote the same octets.
|
||||
|
||||
### 3.1 Magic
|
||||
|
||||
| Representation | Value |
|
||||
|---|---|
|
||||
| Bytes | `b"69CES"` |
|
||||
| Hex | `3639434553` |
|
||||
| Decimal byte list | `[54, 57, 67, 69, 83]` |
|
||||
|
||||
Length: 5 octets. Position: the first octet of every artefact.
|
||||
|
||||
### 3.2 Begin marker
|
||||
|
||||
```
|
||||
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
|
||||
```
|
||||
|
||||
64 characters of lowercase hexadecimal. Written verbatim; it is **not** decoded
|
||||
before comparison.
|
||||
|
||||
### 3.3 End marker
|
||||
|
||||
```
|
||||
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
|
||||
```
|
||||
|
||||
64 characters of lowercase hexadecimal. Written verbatim.
|
||||
|
||||
> **Note.** Both markers are fixed constants of format version 1. They are
|
||||
> delimiters, not integrity checks: they carry no signature semantics and do
|
||||
> not authenticate the payload. Message integrity, where required, is provided
|
||||
> by the cipher and by an external MAC.
|
||||
|
||||
---
|
||||
|
||||
## 4. Container layout
|
||||
|
||||
An artefact is a single character string with five regions:
|
||||
|
||||
```
|
||||
+-------------------------------------------------------------------+
|
||||
| MAGIC "69CES" 5 octets |
|
||||
+-------------------------------------------------------------------+
|
||||
| BEGIN MARKER 64 hex chars |
|
||||
+-------------------------------------------------------------------+
|
||||
| CONFIGURATION UTF-8 JSON object variable |
|
||||
+-------------------------------------------------------------------+
|
||||
| END MARKER 64 hex chars |
|
||||
+-------------------------------------------------------------------+
|
||||
| PAYLOAD key material or ciphertext variable |
|
||||
+-------------------------------------------------------------------+
|
||||
```
|
||||
|
||||
The three fixed regions are literals. Only the configuration and the payload
|
||||
vary between artefacts.
|
||||
|
||||
### 4.1 Formal grammar
|
||||
|
||||
```
|
||||
artifact = MAGIC begin-marker config end-marker payload
|
||||
MAGIC = "69CES"
|
||||
begin-marker= %s"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
|
||||
end-marker = %s"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
|
||||
config = <UTF-8 encoded JSON object, see section 5>
|
||||
payload = <encoding-dependent text, see section 6>
|
||||
```
|
||||
|
||||
There is no whitespace, newline or padding between regions. A reader must not
|
||||
skip optional characters.
|
||||
|
||||
---
|
||||
|
||||
## 5. Configuration record
|
||||
|
||||
The configuration is a JSON object encoded as UTF-8, with no line breaks.
|
||||
Keys are sorted; separators are `,` and `:` with no surrounding spaces. The
|
||||
canonical form is therefore deterministic: the same configuration always
|
||||
renders as the same octets.
|
||||
|
||||
### 5.1 Required fields
|
||||
|
||||
A configuration missing any of these is invalid, and the artefact must be
|
||||
rejected rather than partially interpreted.
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `kind` | string | `"ciphertext"` or `"key"` |
|
||||
| `v` | integer | Format version; `1` for this specification |
|
||||
| `mode` | string | Performance mode, see §5.2 |
|
||||
| `mid` | string | Intermediate representation, see §5.3 |
|
||||
| `out` | string | Output encoding, see §5.4 |
|
||||
| `alg` | string | Algorithm family; `"chaoscrypt"` |
|
||||
| `kdf` | string | Keystream derivation, see §5.5 |
|
||||
| `keybits` | integer | Length of the long key in bits |
|
||||
| `ksum` | string | Non-secret key digest, see §5.6 |
|
||||
|
||||
### 5.2 `mode`
|
||||
|
||||
| Value | Pipeline |
|
||||
|---|---|
|
||||
| `reference` | Original algorithm, character pipeline |
|
||||
| `fast` | Original algorithm, bulk-operation implementation |
|
||||
| `turbo` | SHAKE256 keystream, byte pipeline |
|
||||
| `raw` | SHAKE256 keystream, byte pipeline with the long-key layer retained |
|
||||
|
||||
`reference` and `fast` produce identical ciphertexts. The distinction is
|
||||
recorded so a reader can report which implementation wrote the artefact; it
|
||||
does not affect decryption.
|
||||
|
||||
### 5.3 `mid` — intermediate representation
|
||||
|
||||
The `mid` field names how the two XOR layers of the byte pipeline exchange
|
||||
their data. It is only meaningful for `turbo` and `raw`; for the character
|
||||
pipelines it is descriptive.
|
||||
|
||||
| Value | Meaning |
|
||||
|---|---|
|
||||
| `bt` | Intermediate values carried as raw bytes (default) |
|
||||
| `hex` | Intermediate values carried as hexadecimal text |
|
||||
|
||||
`turbo` and `raw` accept either value, and `bt` is the default. The choice is
|
||||
recorded in the header and honoured when the artefact is read: a `turbo` or
|
||||
`raw` artefact is decrypted with the `mid` it carries, and no fallback to the
|
||||
other representation is attempted.
|
||||
|
||||
`reference` and `fast` are character pipelines and always use `hex`. They
|
||||
ignore `mid`: whatever value is supplied, the header records `hex` for them.
|
||||
|
||||
**Expansion.** `bt` passes the bytes between the XOR layers unchanged, so the
|
||||
only growth is the output encoding: 2.00x with `hex` output, 1.33x with
|
||||
`base` output. `hex` re-encodes each layer's output as hexadecimal text
|
||||
before the next layer consumes it; with the two layers of this format that
|
||||
doubles the payload at each layer, so the byte payload handed to the output
|
||||
encoder is about **4x** the `bt` size (roughly 8 hexadecimal characters per
|
||||
input octet with `hex` output).
|
||||
|
||||
**Speed.** `bt` is about **4x faster** than `hex`: it skips the per-layer hex
|
||||
encode and decode and XORs the shorter byte buffers directly. It is therefore
|
||||
the default.
|
||||
|
||||
### 5.4 `out` — output encoding
|
||||
|
||||
| Value | Meaning | Expansion |
|
||||
|---|---|---|
|
||||
| `hex` | Two characters per octet | 2.00x |
|
||||
| `base` | Base64, four characters per three octets | 1.33x |
|
||||
|
||||
`reference` and `fast` always emit `hex` and ignore any attempt to set it.
|
||||
`turbo` and `raw` accept either.
|
||||
|
||||
> **Reader requirement.** The payload must be decoded according to `out`.
|
||||
> Attempting to decode with the wrong encoding must fail, never silently produce
|
||||
> a different result.
|
||||
|
||||
### 5.5 `kdf` — keystream derivation
|
||||
|
||||
| Value | Meaning |
|
||||
|---|---|
|
||||
| `expand` | Hand-rolled mixing and rotation expansion |
|
||||
| `shake256` | SHAKE256 extensible-output function |
|
||||
|
||||
### 5.6 `ksum` — key digest
|
||||
|
||||
A 16-character lowercase hexadecimal string: the first 16 characters of
|
||||
`SHA256(upper_mapping ‖ lower_mapping ‖ long_key ‖ short_key)`, where `‖` is
|
||||
concatenation of the ASCII encodings.
|
||||
|
||||
It exists so a reader can confirm that a key file and a ciphertext belong
|
||||
together without exposing any usable secret. It is a **correlation aid, not a
|
||||
security control**: it reveals nothing that helps an attacker, and its absence
|
||||
of secrecy must not be relied upon as an integrity guarantee.
|
||||
|
||||
### 5.7 Legacy fields
|
||||
|
||||
Two optional fields exist for reading artefacts produced before the header was
|
||||
introduced. They are never written by the current library except when wrapping a
|
||||
legacy artefact, and they are **not** part of :data:`REQUIRED_FIELDS`.
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `is_old` | boolean | The artefact predates this format |
|
||||
| `old_is_key` | boolean | With `is_old`, whether it is a key file (`true`) or a ciphertext (`false`) |
|
||||
|
||||
> **Precedence rule.** When `is_old` is present and true, **every other field is
|
||||
> ignored**, including `mode`, `mid`, `out` and `kind`. The artefact predates the
|
||||
> field set, so there is nothing else meaningful to read. A reader must not
|
||||
> reject such an artefact for missing `kind`, nor act on any field beside
|
||||
> `is_old` and `old_is_key`.
|
||||
|
||||
An artefact that carries neither field is a normal artefact and is read
|
||||
normally. Their absence is not an error.
|
||||
|
||||
### 5.8 Descriptive fields
|
||||
|
||||
The record also carries fields that describe the construction. They are
|
||||
informational — a reader may use them for display or diagnostics — and they do
|
||||
not modify the decryption path.
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `lib` | Library version that wrote the artefact |
|
||||
| `cache` | Keystream cache strategy in effect; does not affect output |
|
||||
| `subst` | Substitution kind, `"bijective"` |
|
||||
| `flip` | Case-flip kind, `"conditional"` |
|
||||
| `order` | The pipeline, as an ordered list of stage names |
|
||||
| `xor_layers` | Number of XOR layers applied |
|
||||
| `compact` | Whether the byte pipeline was used |
|
||||
| `keyshort` | Length of the short key in bits |
|
||||
| `flipbits` | Number of bits in the flip pattern |
|
||||
| `b64` | Whether Base64 is used as a byte-safety layer |
|
||||
| `utf8` | Whether text is transported as UTF-8 |
|
||||
|
||||
### 5.9 Example
|
||||
|
||||
```json
|
||||
{
|
||||
"alg": "chaoscrypt",
|
||||
"b64": true,
|
||||
"cache": "hash",
|
||||
"compact": true,
|
||||
"flip": "conditional",
|
||||
"flipbits": 10,
|
||||
"kdf": "shake256",
|
||||
"keybits": 16384,
|
||||
"keyshort": 2048,
|
||||
"kind": "ciphertext",
|
||||
"ksum": "3f9a1c7e5b2d4086",
|
||||
"lib": "4.2.6",
|
||||
"mid": "bt",
|
||||
"mode": "turbo",
|
||||
"order": "base64>subst>flip>reverse>xor>xor",
|
||||
"out": "hex",
|
||||
"subst": "bijective",
|
||||
"utf8": true,
|
||||
"v": 1,
|
||||
"xor_layers": 2
|
||||
}
|
||||
```
|
||||
|
||||
Rendered between the markers, this is a single unbroken line.
|
||||
|
||||
---
|
||||
|
||||
## 6. Payload
|
||||
|
||||
### 6.1 Ciphertext payload
|
||||
|
||||
When `kind` is `"ciphertext"`, the payload is the ciphertext encoded per `out`:
|
||||
|
||||
- `out = "hex"`: lowercase hexadecimal, even length, alphabet `[0-9a-f]`.
|
||||
- `out = "base"`: standard Base64 with `=` padding, alphabet
|
||||
`[A-Za-z0-9+/]`.
|
||||
|
||||
The payload is the final output of the pipeline named in `mode`. Nothing is
|
||||
appended after it.
|
||||
|
||||
### 6.2 Key payload
|
||||
|
||||
When `kind` is `"key"`, the payload is the password-wrapped key material, in the
|
||||
same two encodings.
|
||||
|
||||
A caller reading a key file must check that `kind` is `"key"`. A key file is not
|
||||
a ciphertext and must not be accepted where a ciphertext is expected, nor the
|
||||
reverse. This is enforced, not advisory.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reader conformance
|
||||
|
||||
A conforming reader performs these steps, in order, and fails at the first
|
||||
that does not hold. It must not attempt recovery, interpretation or fallback.
|
||||
|
||||
| # | Check | Failure |
|
||||
|---|---|---|
|
||||
| 1 | Input is non-empty | `E017` |
|
||||
| 2 | Input starts with `69CES` | `E017` |
|
||||
| 3 | The 64 characters after the magic equal the begin marker | `E018` |
|
||||
| 4 | The end marker occurs after the begin marker | `E018` |
|
||||
| 5 | Text between the markers is valid UTF-8 JSON | `E018` |
|
||||
| 6 | That JSON is an object | `E018` |
|
||||
| 7 | Every required field of §5.1 is present | `E018` |
|
||||
| 8 | `kind`, `mid` and `out` hold permitted values | `E018` |
|
||||
| 9 | `kind` matches what the caller asked for | `E018` |
|
||||
| 10 | The payload decodes under `out` | `E010` |
|
||||
| 11 | Decryption succeeds under `mode` and the key | `E011` |
|
||||
|
||||
### 7.1 No fallback
|
||||
|
||||
There is no lenient path. An artefact without a header is not a ChaosCrypt
|
||||
artefact and is rejected with `E017`. Artefacts written by revisions before the
|
||||
header existed are not read by this path either: they are recovered only
|
||||
through the explicit compatibility API, which wraps them in a header first
|
||||
(see section 12). The ordinary reader never gains a lenient branch.
|
||||
|
||||
### 7.2 No partial trust
|
||||
|
||||
A header is either wholly valid or wholly rejected. A reader must not act on
|
||||
fields it did manage to parse, because a partially parsed header may describe a
|
||||
different construction than the one that produced the payload.
|
||||
|
||||
---
|
||||
|
||||
## 8. Error codes
|
||||
|
||||
| Code | Name | Raised when |
|
||||
|---|---|---|
|
||||
| `E010` | `MalformedCiphertext` | Payload does not decode under `out` |
|
||||
| `E011` | `DecryptionFailed` | Decryption produced invalid data |
|
||||
| `E017` | `MissingMagicHeader` | No magic present; input is not a CES artefact |
|
||||
| `E018` | `MalformedHeader` | Marker missing, JSON invalid, field absent or out of range, or `kind` mismatch |
|
||||
|
||||
Codes `E001`–`E016` cover key handling, argument and I/O failures and are
|
||||
unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 9. Versioning
|
||||
|
||||
`v` is the format version and is independent of the library version in `lib`.
|
||||
|
||||
A reader must reject an artefact whose `v` it does not implement, with `E018`.
|
||||
Adding optional descriptive fields does not change `v`. Changing the magic, the
|
||||
markers, the region order, or the meaning of a required field does, and requires
|
||||
a new major library release.
|
||||
|
||||
---
|
||||
|
||||
## 10. Conformance checklist
|
||||
|
||||
An implementation claims conformance when it:
|
||||
|
||||
- [ ] writes the magic, both markers and the configuration in the order of §4;
|
||||
- [ ] sorts configuration keys and uses `,` and `:` as separators with no spaces;
|
||||
- [ ] includes every required field of §5.1;
|
||||
- [ ] rejects input without the magic with `E017`;
|
||||
- [ ] rejects malformed headers with `E018`, never partially trusting them;
|
||||
- [ ] enforces the `kind` match on both read and write paths;
|
||||
- [ ] decodes the payload strictly under `out`, failing rather than guessing;
|
||||
- [ ] implements no lenient or legacy fallback path.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 11. Worked examples
|
||||
|
||||
### 11.1 A normal ciphertext
|
||||
|
||||
Text that is encrypted in `turbo` with `base64` output, then the layout read
|
||||
back region by region.
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
|
||||
cc = ChaosCrypt(mode="turbo", encoding="base64")
|
||||
cc.load("my.key", "key-file-password")
|
||||
|
||||
token = cc.encrypt("meet me at the usual place", "encryption-password")
|
||||
```
|
||||
|
||||
The token begins with the five magic octets, then the begin marker:
|
||||
|
||||
```
|
||||
69CES
|
||||
385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93
|
||||
```
|
||||
|
||||
then the configuration, which states the parameters actually used:
|
||||
|
||||
```json
|
||||
{"alg":"chaoscrypt","b64":true,"cache":"hash","compact":true,
|
||||
"flip":"conditional","flipbits":10,"kdf":"shake256","keybits":16384,
|
||||
"keyshort":2048,"kind":"ciphertext","ksum":"8007e271aea2b261",
|
||||
"lib":"4.2.6","mid":"bt","mode":"turbo",
|
||||
"order":"base64>subst>flip>reverse>xor>xor","out":"base","subst":"bijective",
|
||||
"utf8":true,"v":1,"xor_layers":2}
|
||||
```
|
||||
|
||||
then the end marker, then the base64 payload:
|
||||
|
||||
```
|
||||
2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2
|
||||
IUhCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla...
|
||||
```
|
||||
|
||||
Decrypting needs no configuration at all — the record supplies it:
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
from chaoscrypt._header import read_header, strip_header
|
||||
|
||||
cfg = read_header(token)
|
||||
print(cfg["mode"], cfg["out"]) # turbo base
|
||||
|
||||
# a fresh instance with no settings reads it correctly
|
||||
plain = ChaosCrypt(key_file="my.key", key_password="key-file-password",
|
||||
auto_load=True).decrypt(token, "encryption-password")
|
||||
```
|
||||
|
||||
### 11.2 Reading an artefact without knowing anything about it
|
||||
|
||||
Because the header is complete, a reader needs only the key file and the
|
||||
password. The mode, the intermediate representation and the output encoding are
|
||||
all taken from the artefact.
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
|
||||
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
|
||||
auto_load=True)
|
||||
cc.load("my.key", "key-file-password")
|
||||
|
||||
# whatever produced this, it will be read correctly
|
||||
plain = cc.decrypt(open("message.ces").read(), "encryption-password")
|
||||
```
|
||||
|
||||
### 11.3 Inspecting a key file
|
||||
|
||||
```python
|
||||
from chaoscrypt._header import read_header, strip_header
|
||||
|
||||
text = open("my.key", encoding="utf-8").read()
|
||||
cfg = read_header(text)
|
||||
|
||||
print(cfg["kind"]) # key
|
||||
print(cfg["alg"]) # chaoscrypt
|
||||
print(cfg["keybits"]) # 16384
|
||||
print(cfg["ksum"]) # 16 hex characters
|
||||
```
|
||||
|
||||
`ksum` lets a caller confirm that a key file and a ciphertext belong together
|
||||
without exposing anything usable:
|
||||
|
||||
```python
|
||||
cs = open("message.ces", encoding="utf-8").read()
|
||||
assert read_header(cs)["ksum"] == read_header(text)["ksum"], "wrong key file"
|
||||
```
|
||||
|
||||
### 11.4 A rejection
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
from chaoscrypt.errors import ChaosCryptError
|
||||
|
||||
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
|
||||
auto_load=True)
|
||||
|
||||
try:
|
||||
cc.decrypt("7a3f9c...", "encryption-password") # a bare payload
|
||||
except ChaosCryptError as e:
|
||||
print(e.code) # E017
|
||||
print(e.detail) # MissingMagicHeader
|
||||
```
|
||||
|
||||
There is no fallback path here. The input is not a CES artefact, and guessing
|
||||
would risk returning garbage that looks like success.
|
||||
|
||||
---
|
||||
|
||||
## 12. Legacy artefacts
|
||||
|
||||
Artefacts written before this format existed have no header. They are handled by
|
||||
**wrapping**, never by loosening the reader.
|
||||
|
||||
### 12.1 What a legacy artefact looks like
|
||||
|
||||
| Kind | Content |
|
||||
|---|---|
|
||||
| Legacy ciphertext | Hexadecimal text, no prefix of any kind |
|
||||
| Legacy key file | Password-wrapped key material, no prefix of any kind |
|
||||
|
||||
Neither carries a magic value, a marker or a configuration record.
|
||||
|
||||
### 12.2 Wrapping
|
||||
|
||||
To read one, a minimal header is synthesised and prefixed. The payload is not
|
||||
touched.
|
||||
|
||||
```json
|
||||
{"is_old":true,"old_is_key":false}
|
||||
```
|
||||
|
||||
That header is a normal, valid header. Wrapping therefore produces an artefact
|
||||
that the ordinary path can read, and **the ordinary path never gains a lenient
|
||||
branch**: it still rejects headerless input with `E017`.
|
||||
|
||||
```python
|
||||
from chaoscrypt import ChaosCrypt
|
||||
|
||||
cc = ChaosCrypt(key_file="my.key", key_password="key-file-password",
|
||||
auto_load=True)
|
||||
|
||||
# a legacy ciphertext: no header
|
||||
plain = cc.decrypt_legacy("72370655031311011c111f261365172716020b69",
|
||||
"encryption-password")
|
||||
```
|
||||
|
||||
`decrypt_legacy` first tries the input as it stands. If that reports a missing
|
||||
header, it wraps the input and tries again. `decrypt_legacy_auto` additionally
|
||||
probes whether the artefact is a key file or a ciphertext, so the caller does
|
||||
not have to say which.
|
||||
|
||||
A legacy key file is loaded the same way:
|
||||
|
||||
```python
|
||||
legacy_text = open("old-encryption.key", encoding="utf-8").read()
|
||||
cc = ChaosCrypt()
|
||||
cc.load_legacy_keys(legacy_text, "key-file-password")
|
||||
```
|
||||
|
||||
Or wrapped explicitly, when the artefact is to be stored in the new form:
|
||||
|
||||
```python
|
||||
modernised = cc.wrap_legacy(legacy_text, is_key=True)
|
||||
open("my.key", "w", encoding="utf-8").write(modernised)
|
||||
```
|
||||
|
||||
### 12.3 Which algorithm reads a legacy artefact
|
||||
|
||||
`is_old` is true, so every other field is ignored — there is no `mode` to
|
||||
consult. The legacy path tries the bulk-operation implementation first and
|
||||
falls back to the original character pipeline:
|
||||
|
||||
| Attempt | Why |
|
||||
|---|---|
|
||||
| 1. `fast` | Same algorithm, faster; correct for anything the bulk path can read |
|
||||
| 2. `reference` | The original pipeline, in case the artefact needs it |
|
||||
|
||||
Both implement the original algorithm, so either can read pre-header material.
|
||||
The fallback costs a second attempt only when the first fails.
|
||||
|
||||
### 12.4 What is not supported
|
||||
|
||||
- **Writing legacy artefacts.** The library only reads them. Output always
|
||||
carries a header.
|
||||
- **Partial trust of a legacy artefact.** It is wrapped whole; nothing is
|
||||
inferred from its content.
|
||||
- **Reading a legacy artefact without wrapping.** The ordinary path will reject
|
||||
it with `E017`. Use the compatibility API.
|
||||
|
||||
### 12.5 Conformance for readers
|
||||
|
||||
An implementation that supports legacy artefacts must:
|
||||
|
||||
- [ ] detect a headerless input and treat it as legacy, not as an error at the
|
||||
API boundary;
|
||||
- [ ] synthesise a header rather than bypass the parser;
|
||||
- [ ] ignore every field except `is_old` and `old_is_key` when `is_old` is true;
|
||||
- [ ] keep the ordinary path strict — a headerless input passed to `decrypt`
|
||||
still raises `E017`;
|
||||
- [ ] never write a legacy artefact.
|
||||
|
||||
---
|
||||
|
||||
## 13. Relation to the cipher
|
||||
|
||||
The header is metadata. It does not add secrecy, and it is not authenticated.
|
||||
An attacker who modifies the header can cause a decryption to fail, which is a
|
||||
denial of service and no more — the payload cannot be recovered without the key.
|
||||
|
||||
Where an application needs to detect tampering with the header as well as the
|
||||
payload, it must compute a MAC over the **entire artefact**, header included,
|
||||
and verify that MAC before parsing. The format does not do this for you.
|
||||
|
||||
See [SECURITY.md](../SECURITY.md) for what the cipher guarantees and what it
|
||||
does not.
|
||||
@@ -7,6 +7,7 @@ Reference material for ChaosCrypt.
|
||||
| [API.md](API.md) | Complete Python API reference |
|
||||
| [CLI.md](CLI.md) | Command line reference, pipelines, troubleshooting |
|
||||
| [ARCHITECTURE.md](ARCHITECTURE.md) | Module map, data flow, design decisions |
|
||||
| [benchmark.md](benchmark.md) | Performance measurements and mode comparison |
|
||||
| [BUILDING.md](BUILDING.md) | Building, testing and releasing |
|
||||
| [../SECURITY.md](../SECURITY.md) | Design, key space, threat model, attack analysis |
|
||||
| [../README.md](../README.md) | Project overview and quick start |
|
||||
@@ -26,6 +27,10 @@ Read the [Scope](../README.md#scope) section of the README, then
|
||||
**Using the command line**
|
||||
[CLI.md](CLI.md) — covers every subcommand, plus pipelines and shell aliases.
|
||||
|
||||
**Choosing a performance mode**
|
||||
[benchmark.md](benchmark.md) for the numbers, [SECURITY.md §4](../SECURITY.md)
|
||||
for what each mode costs in cryptographic margin.
|
||||
|
||||
**Understanding how it works**
|
||||
[ARCHITECTURE.md](ARCHITECTURE.md) for the code structure, and
|
||||
[SECURITY.md §3](../SECURITY.md) for the algorithm itself.
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
# Benchmark
|
||||
|
||||
Performance reference for `chaoscrypt` **4.2.6**.
|
||||
|
||||
> **Read this first.** These numbers are not a claim that ChaosCrypt competes
|
||||
> with a C-backed cipher. It is pure Python. AES-GCM in `cryptography` runs at
|
||||
> hundreds of megabytes per second; ChaosCrypt's fastest mode manages tens.
|
||||
> What the tables below show is what the implementation costs, how the modes
|
||||
> differ, and which parameters are realistic for the workloads this library is
|
||||
> meant for.
|
||||
|
||||
---
|
||||
|
||||
## 1. Headline figures
|
||||
|
||||
`raw` mode, hex output, keystream cache warm, best of several runs.
|
||||
|
||||
| Payload | `bt` encrypt | `bt` decrypt | `hex` encrypt | `hex` decrypt |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 64 KiB | 26.9 | 60.6 | 20.7 | 20.2 |
|
||||
| 128 KiB | 54.9 | 85.5 | 15.8 | 14.3 |
|
||||
| 1 MiB | 57.3 | 61.0 | 19.3 | 21.8 |
|
||||
| 16 MiB | 63.4 | 69.0 | 18.9 | 22.7 |
|
||||
| 128 MiB | 59.4 | 64.5 | 18.2 | 18.0 |
|
||||
| 1024 MiB | 47.9 | 31.7 | — | — |
|
||||
|
||||
MB/s. Ciphertext expansion: `bt` **2.00x**, `hex` **8.00x**.
|
||||
|
||||
**The one-line summary:** `raw` with the default `bt` intermediate runs at
|
||||
roughly **60 MB/s encryption and 70 MB/s decryption**, with ciphertext twice the
|
||||
payload size. Throughput holds steady across four orders of magnitude, which is
|
||||
what a linear algorithm looks like.
|
||||
|
||||
### Against the original implementation
|
||||
|
||||
| | Original | `raw` bt | Improvement |
|
||||
|---|---:|---:|---:|
|
||||
| 1 MiB encrypt | 12.050 s (0.08 MB/s) | 0.017 s (57 MB/s) | **≈700x** |
|
||||
| Ciphertext | 5.33x | 2.00x | 62% smaller |
|
||||
|
||||
---
|
||||
|
||||
## 2. Test environment
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| CPU | 16 logical cores |
|
||||
| Python | CPython 3.13.4 |
|
||||
| OS | Windows 10 (10.0.26200) |
|
||||
| Library | `chaoscrypt` 4.2.6, pure Python, zero dependencies |
|
||||
| Payload | ASCII, `"A" * n` |
|
||||
| Measurement | Best of several runs; each size warmed once so the keystream cache is populated |
|
||||
|
||||
Best-of-N is used because the machine is not otherwise idle. That makes the
|
||||
figures optimistic; a loaded system will be slower, and the ordering between
|
||||
modes is unaffected.
|
||||
|
||||
> **1024 MiB with `hex`** needs an 8 GB ciphertext as a single Python string,
|
||||
> which is not addressable. That cell is left blank rather than measured in a
|
||||
> way that would only report the allocator's behaviour.
|
||||
|
||||
---
|
||||
|
||||
## 3. Throughput across four orders of magnitude
|
||||
|
||||
`raw`, `bt` intermediate, encryption:
|
||||
|
||||
| Size | MB/s | Ratio to previous |
|
||||
|---|---:|---:|
|
||||
| 64 KiB | 26.9 | — |
|
||||
| 128 KiB | 54.9 | 2.04x |
|
||||
| 1 MiB | 57.3 | 1.04x |
|
||||
| 16 MiB | 63.4 | 1.11x |
|
||||
| 128 MiB | 59.4 | 0.94x |
|
||||
| 1024 MiB | 47.9 | 0.81x |
|
||||
|
||||
From 128 KiB upward the figure is flat within ±15%, which is measurement noise
|
||||
plus cache effects rather than algorithmic behaviour. **64 KiB reads low
|
||||
because fixed setup costs are a larger share of a short operation**, not because
|
||||
something is quadratic.
|
||||
|
||||
The 1024 MiB figure is lower because a 2 GB ciphertext string puts the process
|
||||
under memory pressure. It is an observation about the machine, not the
|
||||
algorithm.
|
||||
|
||||
### The quadratic bug that used to be here
|
||||
|
||||
An earlier revision was genuinely super-linear, and it showed up exactly this
|
||||
way: throughput fell as payload grew. The cause was `_get_dynamic_key`:
|
||||
|
||||
```python
|
||||
key = self.long_key
|
||||
while len(key) < text_length:
|
||||
key += self.long_key # reallocates the whole accumulator every round
|
||||
```
|
||||
|
||||
Covering a 5.6 MB intermediate took **866 ms**. Multiplication instead:
|
||||
|
||||
```python
|
||||
return (lk * (text_length // ll + 1))[:text_length]
|
||||
```
|
||||
|
||||
The same work takes **3 ms**, with byte-identical output. `_expand_keystream_slow`
|
||||
shared the pattern and was fixed the same way.
|
||||
|
||||
---
|
||||
|
||||
## 4. Intermediate representation
|
||||
|
||||
The setting with the largest effect on both speed and size.
|
||||
|
||||
| Payload | `bt` enc | `bt` dec | `hex` enc | `hex` dec | Ciphertext `bt` / `hex` |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 64 KiB | 26.9 | 60.6 | 20.7 | 20.2 | 0.1 / 0.5 MB |
|
||||
| 1 MiB | 57.3 | 61.0 | 19.3 | 21.8 | 2 / 8 MB |
|
||||
| 16 MiB | 63.4 | 69.0 | 18.9 | 22.7 | 32 / 128 MB |
|
||||
| 128 MiB | 59.4 | 64.5 | 18.2 | 18.0 | 256 / 1024 MB |
|
||||
|
||||
`hex` is about **3x slower** and produces **4x more** ciphertext. It re-encodes
|
||||
the payload between the XOR layers, which doubles it twice. It exists for
|
||||
interoperability with the character pipeline's shape, not for performance.
|
||||
|
||||
---
|
||||
|
||||
## 5. Output encoding
|
||||
|
||||
| Encoding | Expansion | Notes |
|
||||
|---|---:|---|
|
||||
| `hex` | 2.00x | Two characters per octet |
|
||||
| `base64` | 1.33x | Four characters per three octets |
|
||||
|
||||
Base64 produces a third less output at broadly the same speed. The choice
|
||||
affects only the final encode.
|
||||
|
||||
---
|
||||
|
||||
## 6. How the modes compare
|
||||
|
||||
| | `reference` | `fast` | `turbo` | `raw` |
|
||||
|---|---|---|---|---|
|
||||
| 1 MiB encrypt | 12.05 s | 0.35 s | 0.05 s | 0.017 s |
|
||||
| Ciphertext | 5.33x | 5.33x | 2.00x | 2.00x |
|
||||
| Payload compatible with pre-4.2.6 | yes | yes | no | no |
|
||||
| Keystream primitive | hand-rolled expansion | hand-rolled expansion | SHAKE256 | SHAKE256 |
|
||||
|
||||
`fast` is `reference` with the same arithmetic performed by C-level bulk
|
||||
operations; its payload is byte-identical. The frozen vectors assert this, so
|
||||
the equivalence cannot silently stop holding.
|
||||
|
||||
`turbo` and `raw` replace the keystream derivation with SHAKE256 and stop
|
||||
re-encoding between layers. `raw` keeps the long-key XOR layer, `turbo` does not.
|
||||
|
||||
---
|
||||
|
||||
## 7. Where the time goes
|
||||
|
||||
Profiling `raw` with the `bt` intermediate, 10 MiB payload, encryption:
|
||||
|
||||
| Stage | Time | Share |
|
||||
|---|---:|---:|
|
||||
| `_bytes_stages` (substitution, flip, reversal) | 29 ms | 24% |
|
||||
| XOR with the long key | 38 ms | 32% |
|
||||
| XOR with the SHAKE keystream | 38 ms | 32% |
|
||||
| Keystream derivation (cached) | ~0 ms | 0% |
|
||||
| Output encode (`.hex()`) | 9 ms | 8% |
|
||||
| **Total** | **119 ms** | |
|
||||
|
||||
Decryption is the mirror image, at 122 ms, and is if anything slightly quicker
|
||||
end to end because it does not encode its output.
|
||||
|
||||
### The changes that mattered
|
||||
|
||||
**1. Keeping the payload in bytes between XOR layers (4x).**
|
||||
The original re-encoded to hexadecimal after each XOR, doubling the payload at
|
||||
every layer. At 1 MiB that meant a 5.6 MB intermediate. Bytes between layers
|
||||
means a 1.4 MB intermediate instead.
|
||||
|
||||
**2. Per-character loops to `bytes.translate` (61x on substitution alone).**
|
||||
Substitution was a dictionary lookup per character; it is now one translation
|
||||
over a 256-entry table. The case flip is periodic, so it became ten strided
|
||||
slice translations instead of a per-character branch.
|
||||
|
||||
**3. Per-byte XOR to one big-integer XOR (43x).**
|
||||
```python
|
||||
# before
|
||||
for i, char in enumerate(text):
|
||||
result.append(f"{ord(char) ^ ord(key[i % key_len]):02x}")
|
||||
|
||||
# after
|
||||
xored = int.from_bytes(data, 'big') ^ int.from_bytes(key, 'big')
|
||||
return xored.to_bytes(n, 'big').hex()
|
||||
```
|
||||
|
||||
**4. Not copying or scanning the whole artefact on decrypt (2x).**
|
||||
The decrypt path used to call `strip()`, then `int(payload, 16)`, and
|
||||
`strip_header` sliced the whole string three times. On a 20 MB artefact that was
|
||||
about 65 ms of pure copying. Removing it took decryption at 512 KiB from
|
||||
34 MB/s to 87 MB/s.
|
||||
|
||||
---
|
||||
|
||||
## 8. What did not work
|
||||
|
||||
An attempt to build all expansion chunks with `numpy` as a two-dimensional
|
||||
array reached only 2.9x and added a dependency. Cached translation tables were
|
||||
faster and dependency-free.
|
||||
|
||||
A later attempt to fold the hex round trip in `_derive_final_key` by processing
|
||||
only the prefix the output consumes gained 1.6x — worth keeping, but far less
|
||||
than the byte-pipeline change that made the whole stage cheaper.
|
||||
|
||||
---
|
||||
|
||||
## 9. Reproducing these numbers
|
||||
|
||||
```bash
|
||||
python bench/benchmark.py
|
||||
python bench/benchmark.py --sizes 64,1024,65536 --budget 0.05 --repeats 3
|
||||
python bench/benchmark.py --groups Text,Binary
|
||||
```
|
||||
|
||||
The harness dispatches every task to a process pool, one worker per core, and
|
||||
reports the best of N rounds per task.
|
||||
|
||||
### A note on large payloads
|
||||
|
||||
Beyond roughly 128 MiB the ciphertext itself becomes the practical limit: at
|
||||
1024 MiB inputs, `bt` needs a 2 GB string and `hex` needs 8 GB. For payloads
|
||||
that large, encrypt in blocks. The per-block throughput is the same, as §3
|
||||
shows.
|
||||
|
||||
---
|
||||
|
||||
## 10. Summary
|
||||
|
||||
| Dimension | Result |
|
||||
|---|---|
|
||||
| Fastest throughput | **~63 MB/s encryption, ~85 MB/s decryption** (`raw`, `bt`, hex) |
|
||||
| Ciphertext overhead | 2.00x (`bt`) or 1.33x (`base64`) |
|
||||
| Speed with a payload compatible with pre-4.2.6 | `fast`, ~30x the original |
|
||||
| Linear across | 64 KiB to 1 GiB |
|
||||
| Dependencies | none, in every mode |
|
||||
| Best suited for | Notes, credentials, documents, personal backups |
|
||||
| Not suited for | Bulk media at gigabyte scale, high-throughput services, KDF stretching |
|
||||
|
||||
> **Bottom line.** The original implementation's cost was almost entirely
|
||||
> Python-level per-byte work, not the cipher's arithmetic. Keeping the payload
|
||||
> in bytes between the XOR layers removed the largest remaining factor, and the
|
||||
> result is a library that runs at tens of megabytes per second rather than
|
||||
> fractions of one — with the same construction, and with the modes and their
|
||||
> trade-offs documented in [SECURITY.md](../SECURITY.md).
|
||||
Reference in New Issue
Block a user