Every key file and ciphertext now carries a header that states how it was produced: magic, two fixed markers, and a complete JSON configuration record. A reader takes its parameters from the artefact rather than from the caller. This is an incompatible format change. Headerless input is rejected with E017; pre-4.2.6 material is read through a compatibility API that wraps it in a minimal header and then uses the ordinary path, so the strict path stays strict. Added - Self-describing header on all artefacts, with a normative specification in docs/FORMAT.md - E017 MissingMagicHeader and E018 MalformedHeader - Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising throughput about 4x. The old behaviour is available as mid="hex". - Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x) - Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto, load_legacy_keys - Frozen reference vectors covering every mode, encoding and intermediate - docs/benchmark.md with measured throughput and the profiling behind it Changed - Default mode is fast: byte-for-byte compatible with the original algorithm, roughly 30x quicker - Substitution, case flipping and the printable fold use bytes.translate over precomputed tables - XOR layers use one big-integer operation instead of a per-byte loop - Keystream derivation caches length-independent base material; expansion uses cached translation tables and list accumulation rather than str += - _get_dynamic_key repeats by multiplication. The loop version was quadratic: covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged. - _derive_final_key decodes only the prefix the output consumes - The decrypt path no longer copies or scans the whole artefact: a redundant strip(), a full-string int(payload, 16), and three whole-string slices in strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s. - Documentation rewritten for 4.2.6; all eleven markdown files are English Fixed 1. Key derivation is no longer quadratic (_get_dynamic_key and _expand_keystream_slow shared the str += pattern) 2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact written with mid="hex" could not be read back 3. change_key_password and export_key_text wrote artefacts without a header 4. The constructor accepted encoding="base64" while the header records out="base"; both spellings are now accepted 5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext) rather than E011 (DecryptionFailed) 6. The TUI encoding menu referenced a missing translation Security - Format break from pre-4.2.6 artefacts, documented rather than accidental - fast is not a weaker mode: the frozen vectors assert identical output - The header adds no secrecy and is not authenticated; an application that must detect header tampering should MAC the whole artefact Performance - raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s decryption, ciphertext 2.00x the payload - Throughput holds steady from 64 KiB to 1 GiB - The original implementation managed 0.08 MB/s
349 lines
11 KiB
Python
349 lines
11 KiB
Python
"""
|
|
ChaosCrypt self-describing header.
|
|
|
|
Every key file and every ciphertext produced by this system begins with a
|
|
header that states exactly how the payload was produced. Nothing is inferred
|
|
from context, and nothing is accepted without a header.
|
|
|
|
Layout
|
|
------
|
|
|
|
b"69CES" magic bytes (hex 3639434553)
|
|
<BEGIN_MARKER> fixed hex marker (64 bytes)
|
|
{"kind": ..., ...} UTF-8 JSON configuration
|
|
<END_MARKER> fixed hex marker (64 bytes)
|
|
<payload> the key material or the ciphertext
|
|
|
|
The configuration JSON is complete on its own: it records the kind of payload,
|
|
the performance mode, whether the intermediate representation was bytes or hex,
|
|
the output encoding, the algorithm version, and the parameters needed to
|
|
reproduce the decryption path. A payload whose header is missing, truncated or
|
|
self-inconsistent is rejected rather than guessed at.
|
|
|
|
Rationale
|
|
---------
|
|
Earlier releases relied on the caller to remember which mode and encoding had
|
|
been used. That is a silent-corruption hazard: a ciphertext decrypted with the
|
|
wrong mode produces garbage that is indistinguishable from a wrong password.
|
|
Recording the parameters next to the ciphertext removes the ambiguity, and
|
|
rejecting headerless input removes the guesswork.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from typing import Any, Dict, Optional
|
|
|
|
from .errors import (
|
|
ChaosCrypt_ERR_MissingMagicHeader_E017,
|
|
ChaosCrypt_ERR_MalformedHeader_E018,
|
|
)
|
|
|
|
__all__ = [
|
|
"ENCODING_TO_OUT",
|
|
"OUT_TO_ENCODING",
|
|
"DEFAULT_MID",
|
|
"FIELD_IS_OLD",
|
|
"FIELD_OLD_IS_KEY",
|
|
"build_legacy_header",
|
|
"is_legacy",
|
|
"MAGIC",
|
|
"MAGIC_HEX",
|
|
"MAGIC_LIST",
|
|
"BEGIN_MARKER",
|
|
"END_MARKER",
|
|
"HEADER_VERSION",
|
|
"build_header",
|
|
"read_header",
|
|
"strip_header",
|
|
"has_magic",
|
|
]
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixed values
|
|
# ---------------------------------------------------------------------------
|
|
#: Magic bytes at the very start of every ChaosCrypt artefact.
|
|
MAGIC = b"69CES"
|
|
|
|
#: The same magic as hex text and as a list of byte values, for callers that
|
|
#: find one of those representations more convenient.
|
|
MAGIC_HEX = "3639434553"
|
|
MAGIC_LIST = [54, 57, 67, 69, 83]
|
|
|
|
#: Marker that opens the configuration block. Fixed, recorded verbatim.
|
|
BEGIN_MARKER = (
|
|
"385aeff1f4179a744c8fb0c5ae4702d325f6230d059f4251135c2536409dee93"
|
|
)
|
|
|
|
#: Marker that closes the configuration block.
|
|
END_MARKER = (
|
|
"2025ea4e4b4b93803f2ccadeaf318d1957ce42ff5409410f660348be96a4c0f2"
|
|
)
|
|
|
|
#: Version of the header format itself, independent of the algorithm version.
|
|
HEADER_VERSION = 1
|
|
|
|
#: Values accepted for the ``kind`` field.
|
|
KIND_CIPHERTEXT = "ciphertext"
|
|
KIND_KEY = "key"
|
|
KINDS = (KIND_CIPHERTEXT, KIND_KEY)
|
|
|
|
#: Values accepted for the ``mid`` (intermediate representation) field.
|
|
MID_BYTES = "bt"
|
|
MID_HEX = "hex"
|
|
MIDS = (MID_BYTES, MID_HEX)
|
|
|
|
#: Default intermediate representation for the byte-pipeline modes.
|
|
DEFAULT_MID = MID_BYTES
|
|
|
|
#: Values accepted for the ``out`` (output encoding) field.
|
|
OUT_HEX = "hex"
|
|
OUT_BASE = "base"
|
|
OUTS = (OUT_HEX, OUT_BASE)
|
|
|
|
#: Mapping from the constructor's encoding names to the header's vocabulary.
|
|
#: The constructor accepts ENC_HEX/ENC_BASE64; the header records OUT_HEX/OUT_BASE.
|
|
ENCODING_TO_OUT = {
|
|
"hex": OUT_HEX,
|
|
"base64": OUT_BASE,
|
|
}
|
|
#: And the reverse, for reading a header back into a constructor value.
|
|
OUT_TO_ENCODING = {v: k for k, v in ENCODING_TO_OUT.items()}
|
|
|
|
#: Required fields in every configuration block. Missing any one of these makes
|
|
#: the header invalid; the header is never partially trusted.
|
|
#: Field marking an artefact as produced by the pre-header implementation.
|
|
#: When present and true, **every other configuration field is ignored**; the
|
|
#: artefact is decrypted by the legacy path (see ``_core``). It is optional: a
|
|
#: normal artefact does not carry it at all.
|
|
FIELD_IS_OLD = "is_old"
|
|
|
|
#: Field naming what a legacy artefact is: true for a key file, false for a
|
|
#: ciphertext. Only meaningful alongside ``is_old``. Optional.
|
|
FIELD_OLD_IS_KEY = "old_is_key"
|
|
|
|
REQUIRED_FIELDS = (
|
|
"kind",
|
|
"v",
|
|
"mode",
|
|
"mid",
|
|
"out",
|
|
"alg",
|
|
"kdf",
|
|
"keybits",
|
|
"ksum",
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Building
|
|
# ---------------------------------------------------------------------------
|
|
def build_header(config: Dict[str, Any]) -> str:
|
|
"""Render a full header for ``config``.
|
|
|
|
Args:
|
|
config: The configuration dictionary. It must contain every name in
|
|
:data:`REQUIRED_FIELDS`.
|
|
|
|
Returns:
|
|
The header as text, ready to be prefixed to a payload.
|
|
|
|
Raises:
|
|
ChaosCrypt_ERR_MalformedHeader_E018: a required field is absent, or a
|
|
field holds a value outside its permitted set.
|
|
"""
|
|
# Legacy records carry only is_old and old_is_key; the strict field checks
|
|
# below do not apply to them.
|
|
legacy = bool(config.get(FIELD_IS_OLD))
|
|
|
|
if not legacy:
|
|
missing = [f for f in REQUIRED_FIELDS if f not in config]
|
|
if missing:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="MissingConfigFields:" + ",".join(missing), code="E018"
|
|
)
|
|
|
|
if legacy:
|
|
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
|
|
separators=(",", ":"))
|
|
return (MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER)
|
|
|
|
if config["kind"] not in KINDS:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="BadKind:" + str(config["kind"]), code="E018"
|
|
)
|
|
if config["mid"] not in MIDS:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="BadMid:" + str(config["mid"]), code="E018"
|
|
)
|
|
if config["out"] not in OUTS:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="BadOut:" + str(config["out"]), code="E018"
|
|
)
|
|
|
|
# Deterministic key order so identical configurations render identically.
|
|
body = json.dumps(config, ensure_ascii=False, sort_keys=True,
|
|
separators=(",", ":"))
|
|
|
|
return (
|
|
MAGIC.decode("ascii")
|
|
+ BEGIN_MARKER
|
|
+ body
|
|
+ END_MARKER
|
|
)
|
|
|
|
|
|
def build_legacy_header(is_key: bool = False) -> str:
|
|
"""Render the minimal header used to wrap a pre-header artefact.
|
|
|
|
A legacy artefact has no header of its own, so one is synthesised before it
|
|
enters the normal decryption path. The synthesised header carries only the
|
|
two fields that matter -- ``is_old`` and ``old_is_key`` -- because once
|
|
``is_old`` is true every other field is ignored.
|
|
|
|
This keeps a single code path: the library always decrypts something that
|
|
has a header. Legacy support is a wrapper around that path, not a second
|
|
implementation of it.
|
|
|
|
Args:
|
|
is_key: Whether the artefact is a key file rather than a ciphertext.
|
|
|
|
Returns:
|
|
A complete header, ready to be prefixed to a legacy payload.
|
|
"""
|
|
body = json.dumps(
|
|
{FIELD_IS_OLD: True, FIELD_OLD_IS_KEY: bool(is_key)},
|
|
ensure_ascii=False, sort_keys=True, separators=(",", ":"),
|
|
)
|
|
return MAGIC.decode("ascii") + BEGIN_MARKER + body + END_MARKER
|
|
|
|
|
|
def is_legacy(text: str) -> bool:
|
|
"""Whether ``text`` is a headed legacy artefact.
|
|
|
|
Returns ``False`` for anything that is not a parseable header, so callers
|
|
can use this as a test rather than a validator.
|
|
"""
|
|
try:
|
|
cfg = read_header(text)
|
|
except Exception:
|
|
return False
|
|
return bool(cfg.get(FIELD_IS_OLD))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Parsing
|
|
# ---------------------------------------------------------------------------
|
|
def has_magic(text: str) -> bool:
|
|
"""Whether ``text`` begins with the ChaosCrypt magic bytes."""
|
|
return text.startswith(MAGIC.decode("ascii"))
|
|
|
|
|
|
def read_header(text: str) -> Dict[str, Any]:
|
|
"""Parse the header from ``text`` and return its configuration.
|
|
|
|
Raises:
|
|
ChaosCrypt_ERR_MissingMagicHeader_E017: ``text`` does not start with the
|
|
magic. Callers must not attempt to guess the format.
|
|
ChaosCrypt_ERR_MalformedHeader_E018: a marker is missing or the JSON is
|
|
invalid or incomplete.
|
|
"""
|
|
if not text:
|
|
raise ChaosCrypt_ERR_MissingMagicHeader_E017(
|
|
detail="EmptyInput", code="E017"
|
|
)
|
|
|
|
magic = MAGIC.decode("ascii")
|
|
if not text.startswith(magic):
|
|
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
|
|
|
|
body_at = len(magic)
|
|
|
|
if not text.startswith(BEGIN_MARKER, body_at):
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="MissingBeginMarker", code="E018"
|
|
)
|
|
body_at += len(BEGIN_MARKER)
|
|
|
|
end_at = text.find(END_MARKER, body_at)
|
|
if end_at < 0:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="MissingEndMarker", code="E018"
|
|
)
|
|
|
|
body = text[body_at:end_at]
|
|
try:
|
|
config = json.loads(body)
|
|
except Exception:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="InvalidHeaderJson", code="E018"
|
|
)
|
|
|
|
if not isinstance(config, dict):
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="HeaderJsonNotObject", code="E018"
|
|
)
|
|
|
|
# A legacy artefact carries only is_old and old_is_key. Everything else is
|
|
# meaningless for it, so the required-field check is skipped entirely --
|
|
# requiring kind/mode/... here would reject the very artefacts this branch
|
|
# exists to read.
|
|
if config.get(FIELD_IS_OLD):
|
|
if FIELD_OLD_IS_KEY in config and not isinstance(
|
|
config[FIELD_OLD_IS_KEY], bool
|
|
):
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="OldIsKeyNotBoolean", code="E018"
|
|
)
|
|
return config
|
|
|
|
missing = [f for f in REQUIRED_FIELDS if f not in config]
|
|
if missing:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="MissingConfigFields:" + ",".join(missing), code="E018"
|
|
)
|
|
|
|
return config
|
|
|
|
|
|
def _payload_offset(text: str) -> int:
|
|
"""Index at which the payload begins, or ``-1`` if the header is unusable.
|
|
|
|
Markers are located by index so no intermediate copy of the payload is
|
|
made. The payload can be tens of megabytes; slicing the whole string three
|
|
times, as an earlier version did, costs three full copies per call and shows
|
|
up directly in throughput.
|
|
"""
|
|
magic = MAGIC.decode("ascii")
|
|
if not text.startswith(magic):
|
|
return -1
|
|
body_at = len(magic)
|
|
if not text.startswith(BEGIN_MARKER, body_at):
|
|
return -1
|
|
body_at += len(BEGIN_MARKER)
|
|
end_at = text.find(END_MARKER, body_at)
|
|
if end_at < 0:
|
|
return -1
|
|
return end_at + len(END_MARKER)
|
|
|
|
|
|
def strip_header(text: str) -> str:
|
|
"""Return the payload that follows the header.
|
|
|
|
The header is validated on the way through, so a malformed header raises
|
|
rather than yielding a payload.
|
|
|
|
Raises:
|
|
ChaosCrypt_ERR_MissingMagicHeader_E017: no magic present.
|
|
ChaosCrypt_ERR_MalformedHeader_E018: header could not be parsed.
|
|
"""
|
|
if not has_magic(text):
|
|
raise ChaosCrypt_ERR_MissingMagicHeader_E017()
|
|
read_header(text) # validate the record itself
|
|
offset = _payload_offset(text)
|
|
if offset < 0:
|
|
raise ChaosCrypt_ERR_MalformedHeader_E018(
|
|
detail="MissingEndMarker", code="E018"
|
|
)
|
|
return text[offset:] # a single slice
|