Files
dvs dd49043171 v4.2.6: Self-describing format, byte pipeline, and a performance pass
Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
2026-09-27 22:06:34 +08:00

349 lines
11 KiB
Python

"""
ChaosCrypt self-describing header.
Every key file and every ciphertext produced by this system begins with a
header that states exactly how the payload was produced. Nothing is inferred
from context, and nothing is accepted without a header.
Layout
------
b"69CES" magic bytes (hex 3639434553)
<BEGIN_MARKER> fixed hex marker (64 bytes)
{"kind": ..., ...} UTF-8 JSON configuration
<END_MARKER> fixed hex marker (64 bytes)
<payload> the key material or the ciphertext
The configuration JSON is complete on its own: it records the kind of payload,
the performance mode, whether the intermediate representation was bytes or hex,
the output encoding, the algorithm version, and the parameters needed to
reproduce the decryption path. A payload whose header is missing, truncated or
self-inconsistent is rejected rather than guessed at.
Rationale
---------
Earlier releases relied on the caller to remember which mode and encoding had
been used. That is a silent-corruption hazard: a ciphertext decrypted with the
wrong mode produces garbage that is indistinguishable from a wrong password.
Recording the parameters next to the ciphertext removes the ambiguity, and
rejecting headerless input removes the guesswork.
"""
from __future__ import annotations
import json
from typing import Any, Dict, Optional
from .errors import (
ChaosCrypt_ERR_MissingMagicHeader_E017,
ChaosCrypt_ERR_MalformedHeader_E018,
)
__all__ = [
"ENCODING_TO_OUT",
"OUT_TO_ENCODING",
"DEFAULT_MID",
"FIELD_IS_OLD",
"FIELD_OLD_IS_KEY",
"build_legacy_header",
"is_legacy",
"MAGIC",
"MAGIC_HEX",
"MAGIC_LIST",
"BEGIN_MARKER",
"END_MARKER",
"HEADER_VERSION",
"build_header",
"read_header",
"strip_header",
"has_magic",
]
# ---------------------------------------------------------------------------
# Fixed values
# ---------------------------------------------------------------------------
#: Magic bytes at the very start of every ChaosCrypt artefact.
MAGIC = b"69CES"
#: The same magic as hex text and as a list of byte values, for callers that
#: find one of those representations more convenient.
MAGIC_HEX = "3639434553"
MAGIC_LIST = [54, 57, 67, 69, 83]
#: Marker that opens the configuration block. Fixed, recorded verbatim.
BEGIN_MARKER = (
"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
)
#: Marker that closes the configuration block.
END_MARKER = (
"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
)
#: Version of the header format itself, independent of the algorithm version.
HEADER_VERSION = 1
#: Values accepted for the ``kind`` field.
KIND_CIPHERTEXT = "ciphertext"
KIND_KEY = "key"
KINDS = (KIND_CIPHERTEXT, KIND_KEY)
#: Values accepted for the ``mid`` (intermediate representation) field.
MID_BYTES = "bt"
MID_HEX = "hex"
MIDS = (MID_BYTES, MID_HEX)
#: Default intermediate representation for the byte-pipeline modes.
DEFAULT_MID = MID_BYTES
#: Values accepted for the ``out`` (output encoding) field.
OUT_HEX = "hex"
OUT_BASE = "base"
OUTS = (OUT_HEX, OUT_BASE)
#: Mapping from the constructor's encoding names to the header's vocabulary.
#: The constructor accepts ENC_HEX/ENC_BASE64; the header records OUT_HEX/OUT_BASE.
ENCODING_TO_OUT = {
"hex": OUT_HEX,
"base64": OUT_BASE,
}
#: And the reverse, for reading a header back into a constructor value.
OUT_TO_ENCODING = {v: k for k, v in ENCODING_TO_OUT.items()}
#: Required fields in every configuration block. Missing any one of these makes
#: the header invalid; the header is never partially trusted.
#: Field marking an artefact as produced by the pre-header implementation.
#: When present and true, **every other configuration field is ignored**; the
#: artefact is decrypted by the legacy path (see ``_core``). It is optional: a
#: normal artefact does not carry it at all.
FIELD_IS_OLD = "is_old"
#: Field naming what a legacy artefact is: true for a key file, false for a
#: ciphertext. Only meaningful alongside ``is_old``. Optional.
FIELD_OLD_IS_KEY = "old_is_key"
REQUIRED_FIELDS = (
"kind",
"v",
"mode",
"mid",
"out",
"alg",
"kdf",
"keybits",
"ksum",
)
# ---------------------------------------------------------------------------
# Building
# ---------------------------------------------------------------------------
def build_header(config: Dict[str, Any]) -> str:
"""Render a full header for ``config``.
Args:
config: The configuration dictionary. It must contain every name in
:data:`REQUIRED_FIELDS`.
Returns:
The header as text, ready to be prefixed to a payload.
Raises:
ChaosCrypt_ERR_MalformedHeader_E018: a required field is absent, or a
field holds a value outside its permitted set.
"""
# Legacy records carry only is_old and old_is_key; the strict field checks
# below do not apply to them.
legacy = bool(config.get(FIELD_IS_OLD))
if not legacy:
missing = [f for f in REQUIRED_FIELDS if f not in config]
if missing:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="MissingConfigFields:" + ",".join(missing), code="E018"
)
if legacy:
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
separators=(",", ":"))
return (MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER)
if config["kind"] not in KINDS:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="BadKind:" + str(config["kind"]), code="E018"
)
if config["mid"] not in MIDS:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="BadMid:" + str(config["mid"]), code="E018"
)
if config["out"] not in OUTS:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="BadOut:" + str(config["out"]), code="E018"
)
# Deterministic key order so identical configurations render identically.
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
separators=(",", ":"))
return (
MAGIC.decode("ascii")
+ BEGIN_MARKER
+ body
+ END_MARKER
)
def build_legacy_header(is_key: bool = False) -> str:
"""Render the minimal header used to wrap a pre-header artefact.
A legacy artefact has no header of its own, so one is synthesised before it
enters the normal decryption path. The synthesised header carries only the
two fields that matter -- ``is_old`` and ``old_is_key`` -- because once
``is_old`` is true every other field is ignored.
This keeps a single code path: the library always decrypts something that
has a header. Legacy support is a wrapper around that path, not a second
implementation of it.
Args:
is_key: Whether the artefact is a key file rather than a ciphertext.
Returns:
A complete header, ready to be prefixed to a legacy payload.
"""
body = json.dumps(
{FIELD_IS_OLD: True, FIELD_OLD_IS_KEY: bool(is_key)},
ensure_ascii=False, sort_keys=True, separators=(",", ":"),
)
return MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER
def is_legacy(text: str) -> bool:
"""Whether ``text`` is a headed legacy artefact.
Returns ``False`` for anything that is not a parseable header, so callers
can use this as a test rather than a validator.
"""
try:
cfg = read_header(text)
except Exception:
return False
return bool(cfg.get(FIELD_IS_OLD))
# ---------------------------------------------------------------------------
# Parsing
# ---------------------------------------------------------------------------
def has_magic(text: str) -> bool:
"""Whether ``text`` begins with the ChaosCrypt magic bytes."""
return text.startswith(MAGIC.decode("ascii"))
def read_header(text: str) -> Dict[str, Any]:
"""Parse the header from ``text`` and return its configuration.
Raises:
ChaosCrypt_ERR_MissingMagicHeader_E017: ``text`` does not start with the
magic. Callers must not attempt to guess the format.
ChaosCrypt_ERR_MalformedHeader_E018: a marker is missing or the JSON is
invalid or incomplete.
"""
if not text:
raise ChaosCrypt_ERR_MissingMagicHeader_E017(
detail="EmptyInput", code="E017"
)
magic = MAGIC.decode("ascii")
if not text.startswith(magic):
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
body_at = len(magic)
if not text.startswith(BEGIN_MARKER, body_at):
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="MissingBeginMarker", code="E018"
)
body_at += len(BEGIN_MARKER)
end_at = text.find(END_MARKER, body_at)
if end_at < 0:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="MissingEndMarker", code="E018"
)
body = text[body_at:end_at]
try:
config = json.loads(body)
except Exception:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="InvalidHeaderJson", code="E018"
)
if not isinstance(config, dict):
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="HeaderJsonNotObject", code="E018"
)
# A legacy artefact carries only is_old and old_is_key. Everything else is
# meaningless for it, so the required-field check is skipped entirely --
# requiring kind/mode/... here would reject the very artefacts this branch
# exists to read.
if config.get(FIELD_IS_OLD):
if FIELD_OLD_IS_KEY in config and not isinstance(
config[FIELD_OLD_IS_KEY], bool
):
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="OldIsKeyNotBoolean", code="E018"
)
return config
missing = [f for f in REQUIRED_FIELDS if f not in config]
if missing:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="MissingConfigFields:" + ",".join(missing), code="E018"
)
return config
def _payload_offset(text: str) -> int:
"""Index at which the payload begins, or ``-1`` if the header is unusable.
Markers are located by index so no intermediate copy of the payload is
made. The payload can be tens of megabytes; slicing the whole string three
times, as an earlier version did, costs three full copies per call and shows
up directly in throughput.
"""
magic = MAGIC.decode("ascii")
if not text.startswith(magic):
return -1
body_at = len(magic)
if not text.startswith(BEGIN_MARKER, body_at):
return -1
body_at += len(BEGIN_MARKER)
end_at = text.find(END_MARKER, body_at)
if end_at < 0:
return -1
return end_at + len(END_MARKER)
def strip_header(text: str) -> str:
"""Return the payload that follows the header.
The header is validated on the way through, so a malformed header raises
rather than yielding a payload.
Raises:
ChaosCrypt_ERR_MissingMagicHeader_E017: no magic present.
ChaosCrypt_ERR_MalformedHeader_E018: header could not be parsed.
"""
if not has_magic(text):
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
read_header(text) # validate the record itself
offset = _payload_offset(text)
if offset < 0:
raise ChaosCrypt_ERR_MalformedHeader_E018(
detail="MissingEndMarker", code="E018"
)
return text[offset:] # a single slice