Files
Aidem-Media-DLL-Analysis/docs/batch.md
T
Patryk GenschandClaude Opus 5.5 e03037647e data: extract 33 scene modules and the Wojna Trojańska engine (68 builds)
The DLL collection gains every WORLD/KOLOROWANKA variant, MATRIX, QMAN
(TEST), INERTIA and BOARD, plus the later PIKLIB72 build from Wojna
Trojańska, taken from the game discs. All 68 builds re-extracted; engine
snapshots are unchanged except the Piklib 6.1 switch dispatch. Roadmap,
batch results and docs follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 21:38:11 +01:00

4.3 KiB

Resumable engine and module extraction

Generate an inventory with python -m ams.inventory ROOT --out inventory.json, then run:

python -m ams.acquire.batch --manifest inventory.json --out-dir snapshots/batch \
  --ghidra-home /path/to/ghidra --timeout 1800 --limit 2

engine and plugin entries run: the extractor recognises a scene module (SCENE:DLLS=) by its CMC_ExternObject::resolve when the DLL has no engine factory. Native dependencies (Sekai) and earlier engines are skipped. Remove --limit for the full inventory. This queue measures successful extraction, not completeness or functional correctness.

Snapshots are named <sha256>.snapshot.json. Each attempt runs in a fresh Python process and process group; the timeout kills that group, including native children. A supplied Ghidra home selects PyGhidra. Otherwise the current wrapper environment chooses the backend.

State records live in states/, separate attempt logs in logs/, and the last invocation's summary/failures in failure-report.json. Every attempt verifies source content before and after analysis. A successful snapshot must contain the expected binary SHA and nonempty types and methods. Writes are atomic, and failed refreshes preserve the prior snapshot.

Re-running skips validated successes with the same extractor fingerprint. The fingerprint includes the queue, wrapper, Python extractor scripts, selected provider, configured Ghidra installation and application properties, PyGhidra version when installed, Python version and Java home. Changes trigger refreshes. Snapshot byte hashes detect edits even when the embedded binary identity remains unchanged. Successful state records include raw type, method and dispatch record counts (not unique API names) and the input filename. Failures and interrupted attempts require --retry-failed. A POSIX advisory lock prevents simultaneous writers to one output directory. Avoid external edits to queue state or snapshots while extraction runs.

Failure reports include skipped failed attempts, and the CLI exits 1 if failures remain. --limit limits attempted builds in that invocation; scope exclusions and resumed successes do not consume it. A limit of zero performs no extraction. The report counts unattempted engine builds as pending when the limit is reached. A prior snapshot alone, without matching successful queue state, is not trusted as a completed queue attempt.

Local macOS setup

The API environment and Ghidra worker use separate Python environments. The tested Ghidra 12.1.2 distribution supplies PyGhidra and JPype wheels compatible with Python 3.12; the API environment uses Python 3.14. From this repository:

.venv-ghidra/bin/python -m ams.acquire.batch \
  --manifest data/binary-inventory.json --out-dir snapshots/batch \
  --ghidra-home /Users/genszu/Downloads/GhidraMacOS/ghidra_install/ghidra/ghidra_12.1.2_PUBLIC \
  --timeout 600 --retry-failed

No interactive Ghidra window or Hopper license is needed. --ghidra-home configures both GHIDRA_HOME and GHIDRA_INSTALL_DIR for the worker; setting GHIDRA_HOME was not a mistake. Each worker gets a temporary project and runtime home, and the original DLL is only read. Internal Ghidra program names are sanitized to ASCII because repository filenames can contain Polish characters; snapshots retain the original filename and SHA.

The bundled inventory (python -m ams.inventory /Users/genszu/_dll_repository --snapshots snapshots --out data/binary-inventory.json) records 99 file paths / 84 distinct SHA-256 values: 35 engine builds, 33 scene modules (KOLOROWANKA 17, WORLD 9, MATRIX 3, QMAN 2, INERTIA 1, BOARD 1), 9 native Sekai dependencies and 7 earlier engines. The modules and the Wojna Trojańska engine were added on 2026-10-28 from the game discs (7-Zip; bsdtar and the original collector missed files on some images). Scope classification is based on filenames, so review newly added or renamed files.

Snapshot metadata retains all source paths and game aliases, extraction fingerprint and provider version. The original files must remain available to verify resumed successes. The queue is sequential; independent manifests can be processed in separate output directories. Do not edit extractor code while a queue is running.

Wynik pełnego przebiegu i kolejność dalszych prac: batch-results.md.