Commit Graph
22 Commits
Author SHA1 Message Date
Patryk GenschandClaude Opus 5.5 b16c8ecf33 docs: generated emulator roadmap and schema 7 results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:29:24 +02:00
Patryk GenschandClaude Opus 5.5 aee8f17417 feat: type_fields in diff and tracker, joined with emulator property reads
The tracker lists each scope's original properties (constructor chain,
or class fields up the chain for Piklib 6.1 and singletons such as
SCENE) and where the emulator reads them: a CNVParser.createVariable
case arm, an attribute read in the type's class or Variable.java, or
other code. The switch-arm parser is shared with the @ statements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 20:30:36 +02:00
Patryk GenschandClaude Opus 5.5 597733d8d7 feat: type_events in diff and tracker, joined with emulator signal literals
Diff gains a type_events axis keyed by (script type, event). The tracker
lists each type's original events (vftable list, or the nearest class
list up the constructor chain for Piklib 6.1) and where the emulator names
them: the type's class, Variable.java or other engine code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 19:49:26 +02:00
Patryk GenschandClaude Opus 5.5 f8178a08f3 docs: runner_map and method facts; tools/ghidra_probe.py for fast iteration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 19:01:41 +02:00
Patryk GenschandClaude Opus 5.5 10c81c4390 feat(tracker): Atlas joining DLL snapshots with emulator registrations
Matches original script APIs against MethodSpec registrations in
Rex-EMoolator and keeps review evidence separate from registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 18:24:15 +02:00
Patryk Gensch ce44e30c60 feat: add full supported API details for variants and enhance tests 2026-06-19 00:20:46 +02:00
Patryk Gensch 0666692198 feat: add detailed method changes in compatibility analysis and update tests 2026-06-19 00:15:04 +02:00
Patryk Gensch 429aafb6ce feat: enhance compatibility table with file name handling and UI updates 2026-06-19 00:07:18 +02:00
Patryk Gensch 2c785dc87c feat: add compatibility table endpoint and rendering for game libraries 2026-06-18 23:32:31 +02:00
Patryk GenschandClaude Opus 4.8 67cbc32a2c Support Piklib 6.1/7.1: CMC_Scene::resolve factory + tag-based types
Earlier text-script engines (Piklib 6.1/7.1, added text scripts in 6.1) keep the
type factory on CMC_Scene::resolve, not CMC_ObjectsContainer::resolve — so the
extractor bailed with "resolve not found". find_factory() now tries both anchors.

6.1's factory is also tag-based: each branch is operator==(NAME) -> new(0x74) ->
store tag -> jmp, with the ctor in a separate tag switch (no inline ctor). extract_types
gains a pre-emit: when the next operator== arrives still armed, it records the pending
type by name (size known, ctor/cpp_class not). The 8.x inline-ctor factory clears `armed`
first, so it's untouched (golden pair unchanged).

Per-version reality: 6.1 = 23 types / 0 methods (no prepareMthHashSet yet) / 103 events
/ 80 fields; 7.1 = 26 / 322 / 102 / 86 / 288 dispatch (full); type names line up across
6.1->7.1->8.x so version diffs work.

- snapshots/PIKLib61 + PIKLIB71 added as golden fixtures (evolution chain)
- tests/test_versions.py: 6.1 partial surface, 7.1 full, 61->71 diff -> 38/38

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 19:53:47 +02:00
Patryk GenschandClaude Opus 4.8 ba9db82a4c worker: optional PyGhidra back-end for Ghidra 11.4+/12.x (no Jython)
The .py extractor runs fine under PyGhidra in the GUI; only `analyzeHeadless`
doesn't init PyGhidra. Add an env-gated CPython path so modern Ghidra works headless:

- ghidra.run_extractor_pyghidra(): runs the same GhidraScript via pyghidra.run_script
  (boots Ghidra in-process, imports+analyses, getScriptArgs()=[out_path]); run_extractor
  dispatches to it when AMS_USE_PYGHIDRA is set. No script changes needed.
- worker image installs pyghidra + sets GHIDRA_INSTALL_DIR; compose exposes
  AMS_USE_PYGHIDRA (default off). Jython path stays the default and untouched.
- README documents both variants (Jython <=11.3.x vs PyGhidra 11.4+/12.x).
- test: AMS_USE_PYGHIDRA routes to the PyGhidra back-end (clear error if pkg missing).

35/35 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 18:03:04 +02:00
Patryk GenschandClaude Opus 4.8 aa65beb7c1 worker: pin Jython-era Ghidra (11.2.1) for headless .py post-script
Ghidra 11.4+/12.x dropped the bundled Jython, so the .py extractor fails headless
with "Ghidra was not started with PyGhidra. Python is not available" — analysis
succeeds but the post-script never runs, so no snapshot is produced. Default
GHIDRA_URL now points at 11.2.1 (Jython); README documents the constraint and the
PyGhidra path for staying on 12.x. Keeps the local Dockerfile fixes (pip upgrade,
non-editable install).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 17:57:17 +02:00
Patryk GenschandClaude Opus 4.8 b77419f4d2 docs: add Quickstart (local, docker compose + Ghidra worker) to README
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 17:00:14 +02:00
Patryk GenschandClaude Opus 4.8 b0d3d22445 Body normalisation: per-method similarity score + leaf delta
Turns the dispatch axis from a binary changed/unchanged into a "how much" measure
of code change — the original goal. ams.normalize compares two body fingerprints
(the ordered leaf-call anchors) with difflib after collapsing consecutive-duplicate
anchors (a load-twice codegen artefact), yielding a 0-100 similarity and the exact
leaves that appeared/vanished.

Every dispatch `changed` entry now carries body={similarity, added, removed}, and the
block carries a summary={shared, identical, changed, mean_similarity}.

Golden pair (cross-compiler): 470 shared bodies, 131 identical, mean 66% similar;
Animo SHOW/HIDE/PAUSE/RESUME come out 100% despite MSVC6 vs MSVC8, LOAD 50% with the
swapped leaves spelled out.

- normalize.py: canonical / body_similarity / body_delta
- diff: _dispatch_diff enriches changed with body + adds summary
- render: METHOD BODIES shows %, leaf delta, summary line
- UI: similarity % + leaf delta + axis summary
- tests: 5 new -> 34/34

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 13:23:15 +02:00
Patryk GenschandClaude Opus 4.8 27399a52b1 Method dispatch axis: map id -> body via Runner::run switch
Recovers how a script method id maps to its implementation, the foundation for
body-level normalisation. Each CMC_*_Runner::run is a switch(id) (vtable slot 17);
every case is the method body — inline (MSVC6) or a tail-call to a separate
show()/load() (MSVC8). The extractor parses the jump table at the disassembly
level (Ghidra's decompiler jump-table recovery silently dropped the big runners),
fingerprints each case by its ordered CALL anchors (Class::method / vtbl+0xNN),
and expands thin wrappers one level so MSVC8 lines up with MSVC6.

Validated on the golden pair: Animo SHOW..RESUME (id 1-4) yield identical leaves
(getAnimo + vtbl+0xa0/0xa4/0x4c/0x50) across both compilers. Coverage 30/32
runners; Piklib 475 / BlooMoo 619 dispatch rows.

- extract_engine_surface.py: extract_method_dispatch (schema_version -> 4)
- snapshots regenerated with the method_dispatch axis
- ams: Snapshot.method_dispatch; diff axis keyed (owner,id) on [impl,calls] with
  method-name join; render METHOD BODIES section; cli --only dispatch; owner filter
- UI: "Ciała metod" diff axis + browse tab
- tests: body-change unit + cross-compiler vtbl assertion -> 29/29

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 13:15:58 +02:00
Patryk GenschandClaude Opus 4.8 38be932abc Similar versions: surface-overlap metric + endpoint + UI panel
Ranks catalogued engine versions by how much of their CMC_* surface they share,
which (unlike a binary fuzzy hash) stays meaningful across compilers — the golden
pair PIKLIB8/MSVC6 vs bloomoodll/MSVC8 scores 85%.

- similarity.py: jaccard, surface_similarity (per-axis + pooled overall),
  fuzzy_similarity (ssdeep via ppdeep, secondary signal)
- service.similar_snapshots + GET /snapshots/{id}/similar?min=N (SimilarHit)
- UI: "Podobne wersje" panel in the snapshot browser (overlap bar + ⇄ diff)
- tests: 6 new (jaccard, identical/disjoint, golden pair 0<x<100, fuzzy,
  endpoint + min filter) -> 28/28

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 12:33:50 +02:00
Patryk GenschandClaude Opus 4.8 30b2b1011e UI: upload panel — POST /jobs with live status polling
Adds a "+ wgraj" control to the sidebar that uploads an ISO/ZIP/DLL to the
acquisition endpoint and tracks the job to completion, then refreshes the
version list so the new snapshot appears without a reload.

- index.html: upload form + #jobs panel in the sidebar
- app.js: submitUpload() (FormData → POST /jobs), pollJobs() (2.5s while any
  job is queued/started; finished → load(); failed → inline error)
- style.css: mini-btn / upload form / job rows + queued/started badges

Verified: node --check clean; uvicorn serves /ui assets 200 and GET /jobs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 12:28:35 +02:00
Patryk GenschandClaude Opus 4.8 f4aa7caaa9 Containerise: Postgres + Redis/RQ + API + Ghidra worker
Brings up the documented target architecture as a docker-compose stack — a
modular monolith with the Ghidra step split into its own async worker.

- worker/: RQ queue (lazy redis import) + run_acquisition task (Job status
  queued→started→finished/failed, drives ams.acquire with sink=db)
- Job model + JobOut schema; Snapshot.data is JSONB on Postgres
- POST/GET /jobs: stream an upload to a shared volume, enqueue, poll status
- docker/api.Dockerfile (slim) + docker/worker.Dockerfile (JDK21 + Ghidra
  fetched at build, overridable via GHIDRA_URL) + docker-compose.yml
- ghidra.py: AMS_GHIDRA_SCRIPTS override for in-container script path
- pyproject: [worker] extra (rq/redis/psycopg), python-multipart in [api]
- tests: 4 new (task success/failure + endpoint enqueue/503) -> 22/22

Verified: API image builds, container serves /health + /ui + /jobs; compose
config validates. Worker image (downloads ~1 GB Ghidra) not built here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 12:24:47 +02:00
Patryk GenschandClaude Opus 4.8 6797ad5ddb Add ISO/ZIP acquisition pipeline (ams.acquire worker)
Closes the chain from a game file to a catalog entry: unpack an ISO/ZIP,
content-identify the engine DLL (CMC_ObjectsContainer marker in RTTI, so a
renamed file is still found), hash it (sha256 + md5 + optional ssdeep via
ppdeep), run Ghidra headless with the extractor, enrich and import the snapshot.

- unpack.py: bsdtar (ISO9660 + ZIP) with a pure-Python zipfile fallback
- identify.py: content-based engine-DLL picker + hashing
- ghidra.py: analyzeHeadless launcher discovery + post-script run
- pipeline.py: orchestration with injectable extract_fn; sink db|http|none
- cli.py: python -m ams.acquire (incl. --identify-only dry run)
- tests: 7 new (forged PE markers + stubbed extractor) -> 18/18

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 12:11:56 +02:00
Patryk GenschandClaude Opus 4.8 4542763936 Add Command Center web UI (no build step)
Static HTML/CSS/JS served by FastAPI (mounted at /ui, / redirects there),
talking to the existing JSON API — no node/npm, no bundler.

- games/versions sidebar with A/B version selectors
- visual 4-axis diff (types/methods/events/fields, +/- struct_layout) with
  +/-/~ rows, per-axis counts, class (owner) filter, moved-methods section
- single-snapshot browser (tabs + live filter)
- app.py mounts StaticFiles(html=True) last so API routes win; / -> /ui/

Smoke-tested live on uvicorn: /, /ui/ and assets serve 200; UI wiring drives
the same /games and /diff endpoints verified end-to-end. app.js passes
`node --check`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 23:37:03 +02:00
Patryk GenschandClaude Opus 4.8 bd03c56e98 docs: document diff CLI and FastAPI backend in README
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 22:28:04 +02:00
Patryk GenschandClaude Opus 4.8 91c7a11ba8 Engine-surface extractor for Piklib/BlooMoo + golden-pair snapshots
Ghidra headless post-script (pyghidra/Jython) that extracts the scripting
"surface" of Aidem Media engine DLLs into a versionable snapshot.json, for
diffing engine versions. All four axes validated on the golden pair
(PIKLIB8.dll / MSVC6 vs bloomoodll.dll / MSVC8):

- types   : CMC_ObjectsContainer::resolve factory ladder
            (script name -> C++ class, ctor, object size; + dispatch_addr,
            via_module_iface for the dual MULTIARRAY branch)
- methods : CMC_*_Runner::prepareMthHashSet (name -> id) + inheritance chain
- events  : CMC_*::getBehavioursList (ordered per-class list)
- fields  : CMC_* ctor -> CMElement::getProperty<T>Value (name + type)
            (+ bonus struct_layout: this+offset stores via decompiler P-code)

Extraction rests on semantic anchors (call targets, referenced string
literals, push/immediate operands), never decompiled-C text, so the same
script works across both compilers despite ILT stubs, undefined string
literals, unnamed FUN_ ctors and an MSVC6 inline-strcpy off-by-one.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 22:11:41 +02:00