v4.2.6: Self-describing format, byte pipeline, and a performance pass

Every key file and ciphertext now carries a header that states how it was
produced: magic, two fixed markers, and a complete JSON configuration record.
A reader takes its parameters from the artefact rather than from the caller.

This is an incompatible format change. Headerless input is rejected with E017;
pre-4.2.6 material is read through a compatibility API that wraps it in a
minimal header and then uses the ordinary path, so the strict path stays strict.

Added
- Self-describing header on all artefacts, with a normative specification in
  docs/FORMAT.md
- E017 MissingMagicHeader and E018 MalformedHeader
- Byte pipeline for turbo and raw: the XOR layers exchange raw bytes rather
  than hexadecimal text, cutting expansion from 5.33x to 2.00x and raising
  throughput about 4x. The old behaviour is available as mid="hex".
- Selectable output encoding for turbo and raw: hex (2.00x) or base64 (1.33x)
- Legacy compatibility API: wrap_legacy, decrypt_legacy, decrypt_legacy_auto,
  load_legacy_keys
- Frozen reference vectors covering every mode, encoding and intermediate
- docs/benchmark.md with measured throughput and the profiling behind it

Changed
- Default mode is fast: byte-for-byte compatible with the original algorithm,
  roughly 30x quicker
- Substitution, case flipping and the printable fold use bytes.translate over
  precomputed tables
- XOR layers use one big-integer operation instead of a per-byte loop
- Keystream derivation caches length-independent base material; expansion uses
  cached translation tables and list accumulation rather than str +=
- _get_dynamic_key repeats by multiplication. The loop version was quadratic:
  covering a 5.6 MB intermediate took 866 ms, now 3 ms, output unchanged.
- _derive_final_key decodes only the prefix the output consumes
- The decrypt path no longer copies or scans the whole artefact: a redundant
  strip(), a full-string int(payload, 16), and three whole-string slices in
  strip_header were removed. Decryption at 512 KiB went from 34 to 87 MB/s.
- Documentation rewritten for 4.2.6; all eleven markdown files are English

Fixed
1. Key derivation is no longer quadratic (_get_dynamic_key and
   _expand_keystream_slow shared the str += pattern)
2. Two hex-intermediate pipelines were asymmetric, so a turbo or raw artefact
   written with mid="hex" could not be read back
3. change_key_password and export_key_text wrote artefacts without a header
4. The constructor accepted encoding="base64" while the header records
   out="base"; both spellings are now accepted
5. A payload that is not valid hexadecimal reports E010 (MalformedCiphertext)
   rather than E011 (DecryptionFailed)
6. The TUI encoding menu referenced a missing translation

Security
- Format break from pre-4.2.6 artefacts, documented rather than accidental
- fast is not a weaker mode: the frozen vectors assert identical output
- The header adds no secrecy and is not authenticated; an application that must
  detect header tampering should MAC the whole artefact

Performance
- raw with the default bt intermediate: ~60 MB/s encryption, ~70 MB/s
  decryption, ciphertext 2.00x the payload
- Throughput holds steady from 64 KiB to 1 GiB
- The original implementation managed 0.08 MB/s
This commit is contained in:
dvs
2026-09-27 22:06:34 +08:00
parent 24a36d6fb4
commit dd49043171
25 changed files with 5348 additions and 852 deletions
+55 -6
View File
@@ -22,7 +22,8 @@ import os
import sys
from typing import List, Optional
from . import DEFAULT_KEY_FILE, __version__
from . import (DEFAULT_KEY_FILE, __version__, MODES, DEFAULT_MODE,
ENCODINGS, DEFAULT_ENCODING, MIDS, DEFAULT_MID)
from .errors import ChaosCryptError, ChaosCrypt_ERR_InvalidArgument_E013
__all__ = ["build_parser", "main", "cli_main"]
@@ -102,6 +103,18 @@ def _add_global_flags(parser: argparse.ArgumentParser) -> None:
dest="quiet_local",
help="suppress non-essential human-readable output",
)
parser.add_argument(
"--mode", choices=MODES, default=None, dest="mode_local", metavar="MODE",
help="performance mode: reference / fast / turbo / raw",
)
parser.add_argument(
"--encoding", choices=ENCODINGS, default=None, dest="encoding_local",
metavar="ENC", help="output encoding for turbo/raw: hex / base64",
)
parser.add_argument(
"--mid", choices=MIDS, default=None, dest="mid_local", metavar="MID",
help="intermediate representation for turbo/raw: bt / hex",
)
def _add_password_opts(
@@ -179,6 +192,33 @@ def build_parser() -> argparse.ArgumentParser:
"-q", "--quiet", action="store_true",
help="suppress non-essential human-readable output",
)
parser.add_argument(
"--mid", choices=MIDS, default=None, metavar="MID",
help=(
"intermediate representation for turbo/raw: 'bt' (bytes, default, "
"2.00x expansion) or 'hex' (hexadecimal text, ~4x). Ignored by "
"reference/fast."
),
)
parser.add_argument(
"--encoding", choices=ENCODINGS, default=None, metavar="ENC",
help=(
"output encoding for turbo/raw: 'hex' (2.00x expansion) or "
"'base64' (1.33x). Ignored by reference/fast, which always emit "
f"hex. Default: {DEFAULT_ENCODING}"
),
)
parser.add_argument(
"--mode", choices=MODES, default=None, metavar="MODE",
help=(
"performance mode. "
"'reference' is the original algorithm (slowest, baseline); "
"'fast' is bit-for-bit identical to it but uses C-level bulk "
"operations; 'turbo' derives the keystream with SHAKE256 and is "
"far faster, but its ciphertexts are not interchangeable with the "
f"other two. Default: {DEFAULT_MODE}"
),
)
sub = parser.add_subparsers(dest="command", metavar="<command>")
@@ -400,7 +440,7 @@ def _cmd_keygen(args: argparse.Namespace) -> int:
args,
)
cc = ChaosCrypt()
cc = ChaosCrypt(mode=args.mode, encoding=args.encoding, mid=args.mid)
path = cc.create_key(pwd, args.key_file, overwrite=args.force)
_emit(
{"ok": True, "key_file": os.path.abspath(path)},
@@ -434,7 +474,7 @@ def _cmd_text(args: argparse.Namespace, mode: str) -> int:
args.text, args.in_file, "plaintext" if mode == "encrypt" else "ciphertext"
)
cc = ChaosCrypt()
cc = ChaosCrypt(mode=args.mode, encoding=args.encoding, mid=args.mid)
cc.load(args.key_file, key_pwd)
if mode == "encrypt":
@@ -483,7 +523,7 @@ def _cmd_file(args: argparse.Namespace, mode: str) -> int:
args,
)
cc = ChaosCrypt()
cc = ChaosCrypt(mode=args.mode, encoding=args.encoding, mid=args.mid)
cc.load(args.key_file, key_pwd)
if mode == "encrypt":
@@ -520,7 +560,7 @@ def _cmd_info(args: argparse.Namespace) -> int:
args,
)
cc = ChaosCrypt()
cc = ChaosCrypt(mode=args.mode, encoding=args.encoding, mid=args.mid)
cc.load(args.key_file, key_pwd)
info = cc.info()
@@ -560,7 +600,7 @@ def _cmd_chpwd(args: argparse.Namespace) -> int:
args,
)
cc = ChaosCrypt()
cc = ChaosCrypt(mode=args.mode, encoding=args.encoding, mid=args.mid)
path = cc.change_key_password(old, new, args.key_file)
_emit(
{"ok": True, "key_file": os.path.abspath(path)},
@@ -596,6 +636,15 @@ def main(argv: Optional[List[str]] = None) -> int:
or getattr(args, "json_local", False))
args.quiet = bool(getattr(args, "quiet", False)
or getattr(args, "quiet_local", False))
args.mode = (getattr(args, "mode", None)
or getattr(args, "mode_local", None)
or DEFAULT_MODE)
args.encoding = (getattr(args, "encoding", None)
or getattr(args, "encoding_local", None)
or DEFAULT_ENCODING)
args.mid = (getattr(args, "mid", None)
or getattr(args, "mid_local", None)
or DEFAULT_MID)
if args.command is None or args.command == "tui":
from .tui import run_tui