Inventory classifies the DLL repository by SHA; the batch queue runs each engine build in a fresh PyGhidra process with a timeout, fingerprints the extractor environment and keeps per-attempt state and logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
38 lines
3.9 KiB
Markdown
38 lines
3.9 KiB
Markdown
# Resumable engine extraction
|
|
|
|
Generate an inventory with `python -m ams.inventory ROOT --out inventory.json`, then run:
|
|
|
|
```sh
|
|
python -m ams.acquire.batch --manifest inventory.json --out-dir snapshots/batch \
|
|
--ghidra-home /path/to/ghidra --timeout 1800 --limit 2
|
|
```
|
|
|
|
Only `engine` entries run. Plugins and native dependencies require their own extraction paths. Remove `--limit` for the full engine inventory. This queue measures successful extraction, not completeness or functional correctness.
|
|
|
|
Snapshots are named `<sha256>.snapshot.json`. Each attempt runs in a fresh Python process and process group; the timeout kills that group, including native children. A supplied Ghidra home selects PyGhidra. Otherwise the current wrapper environment chooses the backend.
|
|
|
|
State records live in `states/`, separate attempt logs in `logs/`, and the last invocation's summary/failures in `failure-report.json`. Every attempt verifies source content before and after analysis. A successful snapshot must contain the expected binary SHA and nonempty types and methods. Writes are atomic, and failed refreshes preserve the prior snapshot.
|
|
|
|
Re-running skips validated successes with the same extractor fingerprint. The fingerprint includes the queue, wrapper, Python extractor scripts, selected provider, configured Ghidra installation and application properties, PyGhidra version when installed, Python version and Java home. Changes trigger refreshes. Snapshot byte hashes detect edits even when the embedded binary identity remains unchanged. Successful state records include raw type, method and dispatch record counts (not unique API names) and the input filename. Failures and interrupted attempts require `--retry-failed`. A POSIX advisory lock prevents simultaneous writers to one output directory. Avoid external edits to queue state or snapshots while extraction runs.
|
|
|
|
Failure reports include skipped failed attempts, and the CLI exits 1 if failures remain. `--limit` limits attempted builds in that invocation; scope exclusions and resumed successes do not consume it. A limit of zero performs no extraction. The report counts unattempted engine builds as `pending` when the limit is reached. A prior snapshot alone, without matching successful queue state, is not trusted as a completed queue attempt.
|
|
|
|
## Local macOS setup
|
|
|
|
The API environment and Ghidra worker use separate Python environments. The tested Ghidra 12.1.2 distribution supplies PyGhidra and JPype wheels compatible with Python 3.12; the API environment uses Python 3.14. From this repository:
|
|
|
|
```sh
|
|
.venv-ghidra/bin/python -m ams.acquire.batch \
|
|
--manifest data/binary-inventory.json --out-dir snapshots/batch \
|
|
--ghidra-home /Users/genszu/Downloads/GhidraMacOS/ghidra_install/ghidra/ghidra_12.1.2_PUBLIC \
|
|
--timeout 600 --retry-failed
|
|
```
|
|
|
|
No interactive Ghidra window or Hopper license is needed. `--ghidra-home` configures both `GHIDRA_HOME` and `GHIDRA_INSTALL_DIR` for the worker; setting `GHIDRA_HOME` was not a mistake. Each worker gets a temporary project and runtime home, and the original DLL is only read. Internal Ghidra program names are sanitized to ASCII because repository filenames can contain Polish characters; snapshots retain the original filename and SHA.
|
|
|
|
The bundled inventory records 64 file paths / 51 distinct SHA-256 values: 34 engine builds in scope, 2 script plugins, 8 native Sekai dependencies and 7 earlier engines excluded from this batch. WORLD and KOLOROWANKA have separately documented registration snapshots. Scope classification is based on filenames, so review newly added or renamed files.
|
|
|
|
Snapshot metadata retains all source paths and game aliases, extraction fingerprint and provider version. The original files must remain available to verify resumed successes. The queue is sequential; independent manifests can be processed in separate output directories. Do not edit extractor code while a queue is running.
|
|
|
|
Wynik pełnego przebiegu i kolejność dalszych prac: [batch-results.md](batch-results.md).
|