v4.2.6: Self-describing format, byte pipeline, and a performance pass
Every key file and ciphertext now carries a header that states how it was produced: magic, two fixed markers, and a complete JSON configuration record. A reader takes its parameters from the artefact rather than from the caller. This is an incompatible format change. Headerless input is rejected with E017; pre-4.2.6 material is read through a compatibility API that wraps it in a minimal header and then uses the ordinary path, so the strict path stays strict. Added - Self-describing header on all artefacts, with a normative specification in docs/FORMAT.md - E017 MissingMagicHeader and E018 MalformedHeader - Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising throughput about 4x. The old behaviour is available as mid="hex". - Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x) - Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto, load_legacy_keys - Frozen reference vectors covering every mode, encoding and intermediate - docs/benchmark.md with measured throughput and the profiling behind it Changed - Default mode is fast: byte-for-byte compatible with the original algorithm, roughly 30x quicker - Substitution, case flipping and the printable fold use bytes.translate over precomputed tables - XOR layers use one big-integer operation instead of a per-byte loop - Keystream derivation caches length-independent base material; expansion uses cached translation tables and list accumulation rather than str += - _get_dynamic_key repeats by multiplication. The loop version was quadratic: covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged. - _derive_final_key decodes only the prefix the output consumes - The decrypt path no longer copies or scans the whole artefact: a redundant strip(), a full-string int(payload, 16), and three whole-string slices in strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s. - Documentation rewritten for 4.2.6; all eleven markdown files are English Fixed 1. Key derivation is no longer quadratic (_get_dynamic_key and _expand_keystream_slow shared the str += pattern) 2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact written with mid="hex" could not be read back 3. change_key_password and export_key_text wrote artefacts without a header 4. The constructor accepted encoding="base64" while the header records out="base"; both spellings are now accepted 5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext) rather than E011 (DecryptionFailed) 6. The TUI encoding menu referenced a missing translation Security - Format break from pre-4.2.6 artefacts, documented rather than accidental - fast is not a weaker mode: the frozen vectors assert identical output - The header adds no secrecy and is not authenticated; an application that must detect header tampering should MAC the whole artefact Performance - raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s decryption, ciphertext 2.00x the payload - Throughput holds steady from 64 KiB to 1 GiB - The original implementation managed 0.08 MB/s
This commit is contained in:
@@ -0,0 +1,348 @@
|
||||
"""
|
||||
ChaosCrypt self-describing header.
|
||||
|
||||
Every key file and every ciphertext produced by this system begins with a
|
||||
header that states exactly how the payload was produced. Nothing is inferred
|
||||
from context, and nothing is accepted without a header.
|
||||
|
||||
Layout
|
||||
------
|
||||
|
||||
b"69CES" magic bytes (hex 3639434553)
|
||||
<BEGIN_MARKER> fixed hex marker (64 bytes)
|
||||
{"kind": ..., ...} UTF-8 JSON configuration
|
||||
<END_MARKER> fixed hex marker (64 bytes)
|
||||
<payload> the key material or the ciphertext
|
||||
|
||||
The configuration JSON is complete on its own: it records the kind of payload,
|
||||
the performance mode, whether the intermediate representation was bytes or hex,
|
||||
the output encoding, the algorithm version, and the parameters needed to
|
||||
reproduce the decryption path. A payload whose header is missing, truncated or
|
||||
self-inconsistent is rejected rather than guessed at.
|
||||
|
||||
Rationale
|
||||
---------
|
||||
Earlier releases relied on the caller to remember which mode and encoding had
|
||||
been used. That is a silent-corruption hazard: a ciphertext decrypted with the
|
||||
wrong mode produces garbage that is indistinguishable from a wrong password.
|
||||
Recording the parameters next to the ciphertext removes the ambiguity, and
|
||||
rejecting headerless input removes the guesswork.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from .errors import (
|
||||
ChaosCrypt_ERR_MissingMagicHeader_E017,
|
||||
ChaosCrypt_ERR_MalformedHeader_E018,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ENCODING_TO_OUT",
|
||||
"OUT_TO_ENCODING",
|
||||
"DEFAULT_MID",
|
||||
"FIELD_IS_OLD",
|
||||
"FIELD_OLD_IS_KEY",
|
||||
"build_legacy_header",
|
||||
"is_legacy",
|
||||
"MAGIC",
|
||||
"MAGIC_HEX",
|
||||
"MAGIC_LIST",
|
||||
"BEGIN_MARKER",
|
||||
"END_MARKER",
|
||||
"HEADER_VERSION",
|
||||
"build_header",
|
||||
"read_header",
|
||||
"strip_header",
|
||||
"has_magic",
|
||||
]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixed values
|
||||
# ---------------------------------------------------------------------------
|
||||
#: Magic bytes at the very start of every ChaosCrypt artefact.
|
||||
MAGIC = b"69CES"
|
||||
|
||||
#: The same magic as hex text and as a list of byte values, for callers that
|
||||
#: find one of those representations more convenient.
|
||||
MAGIC_HEX = "3639434553"
|
||||
MAGIC_LIST = [54, 57, 67, 69, 83]
|
||||
|
||||
#: Marker that opens the configuration block. Fixed, recorded verbatim.
|
||||
BEGIN_MARKER = (
|
||||
"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
|
||||
)
|
||||
|
||||
#: Marker that closes the configuration block.
|
||||
END_MARKER = (
|
||||
"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
|
||||
)
|
||||
|
||||
#: Version of the header format itself, independent of the algorithm version.
|
||||
HEADER_VERSION = 1
|
||||
|
||||
#: Values accepted for the ``kind`` field.
|
||||
KIND_CIPHERTEXT = "ciphertext"
|
||||
KIND_KEY = "key"
|
||||
KINDS = (KIND_CIPHERTEXT, KIND_KEY)
|
||||
|
||||
#: Values accepted for the ``mid`` (intermediate representation) field.
|
||||
MID_BYTES = "bt"
|
||||
MID_HEX = "hex"
|
||||
MIDS = (MID_BYTES, MID_HEX)
|
||||
|
||||
#: Default intermediate representation for the byte-pipeline modes.
|
||||
DEFAULT_MID = MID_BYTES
|
||||
|
||||
#: Values accepted for the ``out`` (output encoding) field.
|
||||
OUT_HEX = "hex"
|
||||
OUT_BASE = "base"
|
||||
OUTS = (OUT_HEX, OUT_BASE)
|
||||
|
||||
#: Mapping from the constructor's encoding names to the header's vocabulary.
|
||||
#: The constructor accepts ENC_HEX/ENC_BASE64; the header records OUT_HEX/OUT_BASE.
|
||||
ENCODING_TO_OUT = {
|
||||
"hex": OUT_HEX,
|
||||
"base64": OUT_BASE,
|
||||
}
|
||||
#: And the reverse, for reading a header back into a constructor value.
|
||||
OUT_TO_ENCODING = {v: k for k, v in ENCODING_TO_OUT.items()}
|
||||
|
||||
#: Required fields in every configuration block. Missing any one of these makes
|
||||
#: the header invalid; the header is never partially trusted.
|
||||
#: Field marking an artefact as produced by the pre-header implementation.
|
||||
#: When present and true, **every other configuration field is ignored**; the
|
||||
#: artefact is decrypted by the legacy path (see ``_core``). It is optional: a
|
||||
#: normal artefact does not carry it at all.
|
||||
FIELD_IS_OLD = "is_old"
|
||||
|
||||
#: Field naming what a legacy artefact is: true for a key file, false for a
|
||||
#: ciphertext. Only meaningful alongside ``is_old``. Optional.
|
||||
FIELD_OLD_IS_KEY = "old_is_key"
|
||||
|
||||
REQUIRED_FIELDS = (
|
||||
"kind",
|
||||
"v",
|
||||
"mode",
|
||||
"mid",
|
||||
"out",
|
||||
"alg",
|
||||
"kdf",
|
||||
"keybits",
|
||||
"ksum",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Building
|
||||
# ---------------------------------------------------------------------------
|
||||
def build_header(config: Dict[str, Any]) -> str:
|
||||
"""Render a full header for ``config``.
|
||||
|
||||
Args:
|
||||
config: The configuration dictionary. It must contain every name in
|
||||
:data:`REQUIRED_FIELDS`.
|
||||
|
||||
Returns:
|
||||
The header as text, ready to be prefixed to a payload.
|
||||
|
||||
Raises:
|
||||
ChaosCrypt_ERR_MalformedHeader_E018: a required field is absent, or a
|
||||
field holds a value outside its permitted set.
|
||||
"""
|
||||
# Legacy records carry only is_old and old_is_key; the strict field checks
|
||||
# below do not apply to them.
|
||||
legacy = bool(config.get(FIELD_IS_OLD))
|
||||
|
||||
if not legacy:
|
||||
missing = [f for f in REQUIRED_FIELDS if f not in config]
|
||||
if missing:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="MissingConfigFields:" + ",".join(missing), code="E018"
|
||||
)
|
||||
|
||||
if legacy:
|
||||
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
|
||||
separators=(",", ":"))
|
||||
return (MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER)
|
||||
|
||||
if config["kind"] not in KINDS:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="BadKind:" + str(config["kind"]), code="E018"
|
||||
)
|
||||
if config["mid"] not in MIDS:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="BadMid:" + str(config["mid"]), code="E018"
|
||||
)
|
||||
if config["out"] not in OUTS:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="BadOut:" + str(config["out"]), code="E018"
|
||||
)
|
||||
|
||||
# Deterministic key order so identical configurations render identically.
|
||||
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
|
||||
separators=(",", ":"))
|
||||
|
||||
return (
|
||||
MAGIC.decode("ascii")
|
||||
+ BEGIN_MARKER
|
||||
+ body
|
||||
+ END_MARKER
|
||||
)
|
||||
|
||||
|
||||
def build_legacy_header(is_key: bool = False) -> str:
|
||||
"""Render the minimal header used to wrap a pre-header artefact.
|
||||
|
||||
A legacy artefact has no header of its own, so one is synthesised before it
|
||||
enters the normal decryption path. The synthesised header carries only the
|
||||
two fields that matter -- ``is_old`` and ``old_is_key`` -- because once
|
||||
``is_old`` is true every other field is ignored.
|
||||
|
||||
This keeps a single code path: the library always decrypts something that
|
||||
has a header. Legacy support is a wrapper around that path, not a second
|
||||
implementation of it.
|
||||
|
||||
Args:
|
||||
is_key: Whether the artefact is a key file rather than a ciphertext.
|
||||
|
||||
Returns:
|
||||
A complete header, ready to be prefixed to a legacy payload.
|
||||
"""
|
||||
body = json.dumps(
|
||||
{FIELD_IS_OLD: True, FIELD_OLD_IS_KEY: bool(is_key)},
|
||||
ensure_ascii=False, sort_keys=True, separators=(",", ":"),
|
||||
)
|
||||
return MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER
|
||||
|
||||
|
||||
def is_legacy(text: str) -> bool:
|
||||
"""Whether ``text`` is a headed legacy artefact.
|
||||
|
||||
Returns ``False`` for anything that is not a parseable header, so callers
|
||||
can use this as a test rather than a validator.
|
||||
"""
|
||||
try:
|
||||
cfg = read_header(text)
|
||||
except Exception:
|
||||
return False
|
||||
return bool(cfg.get(FIELD_IS_OLD))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
def has_magic(text: str) -> bool:
|
||||
"""Whether ``text`` begins with the ChaosCrypt magic bytes."""
|
||||
return text.startswith(MAGIC.decode("ascii"))
|
||||
|
||||
|
||||
def read_header(text: str) -> Dict[str, Any]:
|
||||
"""Parse the header from ``text`` and return its configuration.
|
||||
|
||||
Raises:
|
||||
ChaosCrypt_ERR_MissingMagicHeader_E017: ``text`` does not start with the
|
||||
magic. Callers must not attempt to guess the format.
|
||||
ChaosCrypt_ERR_MalformedHeader_E018: a marker is missing or the JSON is
|
||||
invalid or incomplete.
|
||||
"""
|
||||
if not text:
|
||||
raise ChaosCrypt_ERR_MissingMagicHeader_E017(
|
||||
detail="EmptyInput", code="E017"
|
||||
)
|
||||
|
||||
magic = MAGIC.decode("ascii")
|
||||
if not text.startswith(magic):
|
||||
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
|
||||
|
||||
body_at = len(magic)
|
||||
|
||||
if not text.startswith(BEGIN_MARKER, body_at):
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="MissingBeginMarker", code="E018"
|
||||
)
|
||||
body_at += len(BEGIN_MARKER)
|
||||
|
||||
end_at = text.find(END_MARKER, body_at)
|
||||
if end_at < 0:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="MissingEndMarker", code="E018"
|
||||
)
|
||||
|
||||
body = text[body_at:end_at]
|
||||
try:
|
||||
config = json.loads(body)
|
||||
except Exception:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="InvalidHeaderJson", code="E018"
|
||||
)
|
||||
|
||||
if not isinstance(config, dict):
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="HeaderJsonNotObject", code="E018"
|
||||
)
|
||||
|
||||
# A legacy artefact carries only is_old and old_is_key. Everything else is
|
||||
# meaningless for it, so the required-field check is skipped entirely --
|
||||
# requiring kind/mode/... here would reject the very artefacts this branch
|
||||
# exists to read.
|
||||
if config.get(FIELD_IS_OLD):
|
||||
if FIELD_OLD_IS_KEY in config and not isinstance(
|
||||
config[FIELD_OLD_IS_KEY], bool
|
||||
):
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="OldIsKeyNotBoolean", code="E018"
|
||||
)
|
||||
return config
|
||||
|
||||
missing = [f for f in REQUIRED_FIELDS if f not in config]
|
||||
if missing:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="MissingConfigFields:" + ",".join(missing), code="E018"
|
||||
)
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def _payload_offset(text: str) -> int:
|
||||
"""Index at which the payload begins, or ``-1`` if the header is unusable.
|
||||
|
||||
Markers are located by index so no intermediate copy of the payload is
|
||||
made. The payload can be tens of megabytes; slicing the whole string three
|
||||
times, as an earlier version did, costs three full copies per call and shows
|
||||
up directly in throughput.
|
||||
"""
|
||||
magic = MAGIC.decode("ascii")
|
||||
if not text.startswith(magic):
|
||||
return -1
|
||||
body_at = len(magic)
|
||||
if not text.startswith(BEGIN_MARKER, body_at):
|
||||
return -1
|
||||
body_at += len(BEGIN_MARKER)
|
||||
end_at = text.find(END_MARKER, body_at)
|
||||
if end_at < 0:
|
||||
return -1
|
||||
return end_at + len(END_MARKER)
|
||||
|
||||
|
||||
def strip_header(text: str) -> str:
|
||||
"""Return the payload that follows the header.
|
||||
|
||||
The header is validated on the way through, so a malformed header raises
|
||||
rather than yielding a payload.
|
||||
|
||||
Raises:
|
||||
ChaosCrypt_ERR_MissingMagicHeader_E017: no magic present.
|
||||
ChaosCrypt_ERR_MalformedHeader_E018: header could not be parsed.
|
||||
"""
|
||||
if not has_magic(text):
|
||||
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
|
||||
read_header(text) # validate the record itself
|
||||
offset = _payload_offset(text)
|
||||
if offset < 0:
|
||||
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
||||
detail="MissingEndMarker", code="E018"
|
||||
)
|
||||
return text[offset:] # a single slice
|
||||
Reference in New Issue
Block a user