Compare commits

..
11 Commits
Author SHA1 Message Date
Patryk GenschandClaude Opus 5 20ce26364c Add relocate, and a CUDA image for transcribing on an NVIDIA box
The database records absolute paths to the disc images, because audio, animation
frames and transcription all read straight out of them. Carrying catalog.sqlite
to another machine therefore leaves a catalogue that displays everything and can
play nothing — and the same already applied between host and container, where the
collection is /Users/... on one side and /media on the other.

`relocate <directory>` re-points every copy, matching on filename and confirming
by size, with --verify adding a full SHA-256 and --dry-run showing the outcome
first. Copies that cannot be matched keep their old path and are reported rather
than quietly rewritten.

Names are compared after Unicode normalisation, which turned out to be the whole
problem in practice: macOS stores "Wojna Trojańska.iso" decomposed (n + combining
acute) while the database held it composed. Two of thirteen copies failed to match
until that was fixed — and the same mismatch is exactly what would happen carrying
a collection between macOS and Windows.

Verified by rewriting the paths in a copy of the database to a Windows-shaped
D:\Kolekcja, relocating, and then reading a voice line and an animation frame out
of the discs through the relocated database.

Dockerfile.cuda builds whisper.cpp with GGML_CUDA for compute capability 8.6
(GeForce RTX 30), on nvidia/cuda for both stages, with the JRE installed on top of
the CUDA runtime. It is a separate file because both base images differ; folding it
into the main Dockerfile would be more conditionals than content. Not verified
beyond `docker build --check`: there is no NVIDIA GPU here, and the images are
amd64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 17:10:36 +02:00
Patryk GenschandClaude Opus 5 b99f239553 Splice short lines into 30-second windows before transcribing
Whisper always encodes a full 30-second window, whatever the clip length, and the
average line here is 4.4 seconds. Fed one at a time, 19.5 hours of speech costs
what 135 hours of continuous audio would. Lines are now concatenated with a
second of silence between them, filling a window with about five, and the result
is cut back apart using the timestamps from whisper's JSON output. A segment goes
to the line it overlaps most, so a line split across several segments is
reassembled and a segment straying into the silence still lands correctly.

Measured on 48 real lines, same set through both paths:

  speech (39)      similarity 0.92, no text landed under the wrong file
  non-speech (9)   similarity 0.54 — both modes invent music annotations

Splicing turns out to be slightly *more* accurate on speech, because the model
sees context: "Jeden raz czułem" becomes "Niejeden raz czułem", "muszę to pościć"
becomes "puścić", "SOO?" becomes "Co?".

On the whole collection this projects to 5.8 h → 3.5 h, a 1.7× gain rather than
the 5× I claimed earlier from counting encoder windows alone. Encoding a window
and decoding a line cost about the same, 0.65 s each, and decoding is per line no
matter how the audio is packed — so only half the work can be folded away.

Window size, gap and batch size are all settable, and catalog.whisper.concat=false
restores the file-at-a-time path, which is how the two were compared. Output moved
from -otxt to -oj because only the JSON carries the timestamps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 16:59:55 +02:00
Patryk GenschandClaude Opus 5 a773d791a7 Drop Whisper's invented subtitle credits from transcripts
Measured on 64 real clips with ggml-medium: 54 came back as genuine Polish, 9 as
bracketed non-speech markers that are accurate ([muzyka] on music), and one as
"Napisy stworzone przez społeczność Amara.org" — a credit line Whisper learned
from training data and emits over silence. That is not a record of what is on the
disc, so it does not belong in an archival catalogue. Bracketed markers stay:
they describe the recording truthfully.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 16:30:14 +02:00
Patryk GenschandClaude Opus 5 930cfb3462 Transcribe in batches instead of one file per whisper run
Every whisper.cpp invocation loads the model from scratch, which for a large
model takes longer than recognising a few seconds of speech. Running it once per
file meant that across seventeen thousand recordings the loading would dominate
the work entirely. Files now go in batches of sixteen — 40 files took 3 runs
instead of 40 in a stub test, with each file still getting its own result.

A batch has to be one language, since -l applies to the whole invocation, so work
is grouped by the language derived from the wavs/<code>/ path. Results come back
as files next to the inputs (-otxt) rather than on stdout, because with several
files in one run stdout cannot be split per file. Cancellation now lands between
batches rather than between files, which at sixteen files is close enough.

Thread count is left at whisper's own default and exposed as
catalog.whisper.threads: raising it buys speed at the cost of heat, and on a
fanless machine that turns into throttling anyway.

Docker: whisper.cpp bumped to v1.9.2 to match what Homebrew installs. The library
naming changed there — versioned sonames like libggml.so.0 — and the copy pattern
had to widen to match, otherwise the binary could not start.

Verified with a stub in place of whisper-cli, on the host and inside the
container: batching holds, each file gets its own text, and the temp directory
works as uid 10001. Nothing was left in the transcript table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 16:06:08 +02:00
Patryk GenschandClaude Opus 5 cf6fd4e55e Build whisper.cpp into the image and mount the model from outside
Baking the model in was the wrong call: it is the part that weighs gigabytes,
everyone keeps a different one, and it has no business inside an image. The
binary is the opposite — whisper.cpp plus its libraries come to 2 MB. So the
tools go in and the model is mounted under /models, pointed at by WHISPER_MODEL.

Two things had to be worked out to build it. ggml tunes for the building
machine's CPU by default, which on arm64 emits -mcpu=native+nodotprod+noi8mm+nosve
and GCC 12 rejects outright; GGML_NATIVE=OFF fixes that and is what a portable
image wants anyway. And `cmake --install` insists on installing every example,
including binaries we deliberately did not build, so the artefacts are copied
straight out of the build tree.

A missing model is now reported before any work starts, not hit halfway through:
in a container the path is supplied from outside and the file behind it may
simply not be there.

Verified on the running daemon: image builds, and inside the container the full
pipeline reproduces the host run exactly — 13 copies, 880 scripts, 13 884
recordings, 3531 images and 6928 animations (81 515 frames). Graphics decode in
the container too, since the JRE image carries java.desktop. Thumbnails and
on-demand frame rendering answer over the published port, MCP lists 9 tools, the
collection mount rejects writes, the process runs as uid 10001, and data survives
a restart. The transcription chain was exercised with a stub binary in place of
whisper-cli: ffmpeg hands it exactly 16 kHz mono and results reach the database.
Only a real model run remains untried, since no model was downloaded.

Note on size: the image goes from 516 MB to 1.09 GB, and ffmpeg alone accounts
for 410 MB of that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 15:11:31 +02:00
Patryk GenschandClaude Opus 5 f1b36f0e29 Decode IMG and ANN graphics, and follow the engine's file lookup rules
10 459 images and animations were listed but unviewable. They are now decoded to
PNG without OpenGL, so a script reference like IMGOVERLAY:FILENAME=NAKLADKA.IMG
shows the actual picture.

Decompression comes from :core (CLZW2Compression, CRLECompression) — that is the
hard part and there is no reason to have two copies of it. The header parsing had
to be written here, and not by choice: ImageLoader keeps its parser private, and
AnimoLoader, despite a signature that looks headless-friendly, builds Image
objects whose constructor creates a Texture straight away. Field layout mirrors
those loaders one for one, so catalogue and emulator read the same bytes the same
way. Pixels are RGB565/RGB555 plus a separate alpha byte, composed with ImageIO.

One quirk needed care: ImageLoader maps compression 4 to "none" for IMG files,
but in animation frames the same 4 means real CRLE and AnimoLoader passes it
through. Applying the IMG quirk to ANN turned whole animations into noise.

3531 images and 6928 animations decode (81 515 frames, 29 019 named events);
30 files fail and are recorded with the reason. Previews are thumbnails only —
340 MB of cache instead of decoding everything to disk — and full frames are
rendered from the disc on demand. Animations carry their author: 6513 of them
are signed Piotr Maciejewski.

File lookup now follows the engine instead of guessing. A bare FILENAME means
next to the script; $ is the game root, so $COMMON\X and $WAVS\X resolve there;
WAV files live in wavs/. This matters because a name alone does not identify a
file — Wojna Trojańska ships seventeen different bkg.img, one per scene, and
matching on the name showed the wrong picture for all of them. Resolution is now
98% overall and 97% for WAV with nothing uncertain; the 900 matches that still
fall back to name-only are flagged in the UI rather than passed off as fact.
What stays unresolved is mostly save-state written at runtime.

Two extraction bugs fixed along the way: fields ending in ^N (VARIWST:ONCHANGED^2)
were skipped entirely, hiding 176 real references; and blocking a match at an
underscore made the regex restart one character later, cutting HIST0.ARR out of
+"_HIST0.ARR". Matches must now begin at a token boundary.

Resolution used to run as correlated subqueries over a CTE, re-evaluated per row:
36 s for a script with 745 references. Parameters are now bound directly and
file.basename is a generated, indexed column — 0.19 s.

Docker gains whisper.cpp and ffmpeg (tens of MB); the model is mounted under
/models instead, since it is the part that weighs gigabytes and everyone keeps a
different one. Not verified: the Docker daemon is not running on this machine, so
the image was not built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 14:40:13 +02:00
Patryk GenschandClaude Opus 5 6feb2254b2 Index audio, dialogue tables and script→asset references
Turns the catalogue from a file listing into something you can read: a script
line like SNDQUESTION:FILENAME=KRET_E511.WAV now shows how long the recording
is, who speaks it and at which event, and plays it straight from the disc image.

Four sources feed that:

- Audio headers. RIFF/WAVE and Ogg Vorbis parsed in-process from the image
  stream, no temp files. WAV needs only the first 4 kB, so 3.3 GB of samples
  never reaches the CPU. Not routed through :core — SoundLoader wants a
  FileHandle and a live Gdx.audio, and its contribution is a standard RIFF
  header. 13 884 recordings, 20.2 h.

- wavs/wav.snd. Reksio i Kapitan Nemo packs its whole voice cast into one
  79 MB container that :core does not read, so its speech was invisible here.
  Flat length-prefixed entries holding Ogg Vorbis (oggenc.exe ships on the
  disc). The parser walks the file to its exact last byte: 3274 entries.
  Payloads stay in place — offset and length are enough to serve them, and
  seeking 82 MB into the ISO costs 45 ms.

- dialogi.dta. Pipe-separated CP1250 giving every line a speaker and the event
  that triggers it, so "what is in this file" is answerable without any speech
  recognition. Scene-definition tables share the format, so a row only counts
  as dialogue when column 0 is an identifier rather than a path. 3819 lines,
  98% resolving to real audio.

- Script references. OBJECT:FIELD=VALUE is uniform even inside CODE={...},
  which the decoder folds onto one line, so one pass catches declarations and
  names woven into behaviour code alike. 19 735 references; 97% resolve.
  Names built by concatenation (+"_DEF.DTA") are excluded, but a real leading
  underscore (_WZIECIE_JABLKA.WAV) is kept.

Languages now come from install.ini's [Language] section, with LCIDs translated
through :core's LangCodeConverter rather than a second table here — that is what
settles wavs/slo/ as Slovak. Promote copies them into edition_language, only
ever adding, so curated entries survive.

Transcription is wired to whisper.cpp but never runs on its own: a button with a
progress bar, a stop that keeps what is already computed, and a transcribe
command. Results are generated, not read off the disc, so they live in their own
table with the model and tool named, and the UI labels them as machine guesses.
The pipeline was verified with a stub binary — ffmpeg hands whisper exactly
16 kHz mono, progress and cancellation work, and per-language selection follows
wavs/<code>/. No real transcripts were stored.

Also fixes a pre-existing bug: .cols set display:grid, which beat the browser's
[hidden] rule, so the Skrypty section never actually hid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 13:50:12 +02:00
Patryk GenschandClaude Opus 5 285b4ed316 Drop JAVA_TOOL_OPTIONS and --system from the image
JAVA_TOOL_OPTIONS made the JVM print "Picked up ..." on every invocation, and
the entrypoint already sets file.encoding through JAVA_OPTS. useradd --system
warned because the uid is above SYS_UID_MAX; the explicit uid is what matters.

Verified end to end on Docker 29.4: image builds, the full pipeline runs
inside the container against a read-only collection mount, the frontend and
MCP answer on the published port, and the database survives a restart in the
named volume. The read-only mount was confirmed to actually reject writes,
and the container runs as uid 10001, not root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:51:39 +02:00
Patryk GenschandClaude Opus 5 02988297ec Add web frontend and Docker packaging
Frontend is a single page served from classpath resources with a JSON API
alongside it. Kept separate from the MCP tools because the consumers differ:
a model reads formatted text, a browser needs structures. Only the database
is shared.

The page has three views - catalog with edition details, copies, detected
facts with evidence and a filterable file list; script search with FTS5
snippets and a viewer; and a format breakdown. No build step, no CDN, no
dependencies; light and dark follow the system.

New `serve` command puts the frontend and MCP on one port, one process and
one database connection, which is also what the container runs. `mcp` alone
still works for a headless setup.

Docker is a two-stage build: JDK plus the Rex-EMoolator submodule to compile,
JRE for runtime. installDist rather than build, so check - and with it
verifyCoreVersion - is skipped and git is not needed in the image. The pinned
:core tag is extracted from gradle.properties at build time and passed in by
the entrypoint, otherwise copy.core_version would record "nieznana" and
derived-artifact invalidation would stop working. Properties go through
JAVA_OPTS because Gradle's launcher treats everything after the script name
as application arguments.

Compose mounts the collection read-only and keeps the database and script
cache in a named volume. The port is bound to loopback.

Not verified: the image itself does not build here, the Docker daemon is not
running on this machine. The launcher, JAVA_OPTS handling and resource
packaging were tested against the installDist output directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:34:51 +02:00
Patryk GenschandClaude Opus 5 393bcde6a8 Add MCP server over Streamable HTTP
Seven read-only tools over the catalog database: list_titles, search_scripts,
get_script, get_edition, list_files, find_file and collection_stats. Writing
stays in the CLI, where the effect of a command is visible.

Transport is Streamable HTTP on the JDK's com.sun.net.httpserver, so Gson is
the only new dependency. No sessions are kept: every tool is stateless, so
Mcp-Session-Id is omitted and GET returns 405 rather than opening an SSE
stream we would never write to. Requests carrying a non-loopback Origin are
rejected, since a page in a browser can POST to localhost.

Tool results are formatted text rather than JSON. The consumer is a model
reading the answer, and prose costs less context than the same data wrapped
in objects.

Required arguments are validated against each tool's inputSchema before
dispatch, so a missing parameter is an explicit tool error instead of a
result computed from a default. find_file groups by the canonical lowercase
path, otherwise an extracted directory and its ISO look like two files.

Database access is serialized on one lock because sqlite-jdbc shares a single
connection; the workload is read-only and single-client.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:25:36 +02:00
Patryk GenschandClaude Opus 5 2a051c561b Add script decoding, full-text search and catalog entities
Script pipeline:
- decode: ScriptDecypher from :core into a content-addressed cache keyed by
  blob SHA-1, so a script shared by several images is decoded once. Cache
  invalidation is driven by artifact.tool_version, i.e. the pinned :core tag.
- find/cat: FTS5 index over decoded bodies. tokenchars '_' keeps identifiers
  whole; a query that fails to parse as FTS5 is retried as a quoted phrase.

Catalog entities:
- analyze: MetadataDetector reads dane/application.def for build date, game
  version, engine version and episodes. The APPLICATION object is located by
  type, not by name, since it is GAME, UFO or PIRACI depending on the title.
  CREATIONTIME is recorded separately from release_date because it is the
  project creation date, shared across a whole series.
- promote: builds titles and editions. KnownHashes entries conflate levels
  ("Reksio i UFO (pierwsza wersja)" is title plus edition label), so the
  parenthetical is split off and both UFO releases land under one title.
  Editions are merged on a fingerprint of engine DLL plus application.def
  hash; the DLL alone cannot separate the Herkules/Odyseusz two-in-one disc.
- set/lang: manual metadata a detector cannot infer - provenance, language
  lists with roles, engine/compiler/date overrides. Re-running promote only
  touches mechanical fields and leaves curated ones intact.

Schema:
- edition is rebuilt: dll_sha1 loses UNIQUE, since the two-in-one disc shares
  one engine library across two games. fingerprint becomes the merge key and
  *_override columns hold curated values. The rebuild only runs on an empty
  table; otherwise it fails loudly rather than dropping curated data.
- script_fts, an FTS5 virtual table keyed by blob rather than by file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:17:52 +02:00
29 changed files with 8113 additions and 18 deletions
+12
View File
@@ -0,0 +1,12 @@
# submoduł vendor/ MUSI zostać — composite build kompiluje z niego :core
.git
.gitmodules
.gradle
.idea
build
out
data
collection
vendor/Rex-EMoolator/.git
vendor/Rex-EMoolator/build
vendor/Rex-EMoolator/.gradle
+3
View File
@@ -7,3 +7,6 @@ out/
# baza i cache pochodnych — generowane, nie wersjonowane # baza i cache pochodnych — generowane, nie wersjonowane
data/ data/
# modele whisper.cpp montowane do kontenera — kilkaset MB, nie wersjonujemy
models/
+73
View File
@@ -0,0 +1,73 @@
# Build wymaga submodułu vendor/Rex-EMoolator — composite build kompiluje :core
# ze źródeł. Jeśli katalog jest pusty, settings.gradle przerwie z komunikatem.
FROM eclipse-temurin:21-jdk AS build
WORKDIR /src
# najpierw sam opis buildu: zmiana w kodzie nie unieważnia pobranych zależności
COPY gradlew settings.gradle build.gradle gradle.properties ./
COPY gradle ./gradle
COPY vendor ./vendor
RUN ./gradlew --no-daemon dependencies --configuration runtimeClasspath > /dev/null 2>&1 || true
COPY src ./src
# installDist zamiast build: pomija check, a więc i verifyCoreVersion, które
# potrzebowałoby gita w obrazie. Wersję i tak przypina submoduł.
RUN ./gradlew --no-daemon installDist \
&& grep '^coreVersion=' gradle.properties | cut -d= -f2 > /src/core-version
# whisper.cpp budowany ze źródeł: w repozytoriach Debiana go nie ma, a binarka
# z hosta jest nie do użycia — tam macOS, tu Linux.
#
# GGML_NATIVE=OFF, bo domyślnie ggml stroi kod pod procesor maszyny budującej
# (na arm64 przekazuje -mcpu=native+nodotprod..., czego GCC 12 nie rozumie),
# a obraz i tak ma chodzić także na innym sprzęcie niż ten, na którym powstał
FROM debian:bookworm-slim AS whisper
RUN apt-get update \
&& apt-get install -y --no-install-recommends git cmake build-essential ca-certificates \
&& git clone --depth 1 --branch v1.9.2 https://github.com/ggerganov/whisper.cpp /src \
&& cmake -S /src -B /src/build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
-DGGML_NATIVE=OFF -DWHISPER_BUILD_TESTS=OFF -DWHISPER_BUILD_EXAMPLES=ON \
&& cmake --build /src/build --target whisper-cli -j "$(nproc)" \
&& mkdir -p /out/bin /out/lib \
&& cp /src/build/bin/whisper-cli /out/bin/ \
&& find /src/build \( -name 'libwhisper.so*' -o -name 'libggml*.so*' \) \
-exec cp -P {} /out/lib/ \; \
&& rm -rf /var/lib/apt/lists/* /src
FROM eclipse-temurin:21-jre
WORKDIR /app
# obrazy płyt montowane z zewnątrz, tylko do odczytu; katalog danych to wolumen
VOLUME ["/data"]
# whisper.cpp i ffmpeg wchodzą do obrazu, bo to kilkadziesiąt megabajtów.
# Model NIE wchodzi: waży kilkaset megabajtów do kilku gigabajtów, każdy trzyma
# inny i wybór należy do użytkownika — montuje się go pod /models i wskazuje
# przez WHISPER_MODEL. Bez modelu katalog działa normalnie, a front mówi wprost,
# czego brakuje: mówcę i kontekst kwestii bierzemy z dialogi.dta, nie z modelu.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# kopiujemy wprost z drzewa budowania: `cmake --install` chciałby zainstalować
# wszystkie przykłady, a budujemy tylko whisper-cli
COPY --from=whisper /out/bin/whisper-cli /usr/local/bin/whisper-cli
COPY --from=whisper /out/lib/ /usr/local/lib/
RUN ldconfig
ENV WHISPER_BIN=/usr/local/bin/whisper-cli
# file.encoding ustawia entrypoint przez JAVA_OPTS; JAVA_TOOL_OPTIONS robiłoby
# to samo, ale JVM wypisuje wtedy "Picked up..." przy każdym uruchomieniu
ENV CATALOG_DATA=/data
COPY --from=build /src/build/install/rex-catalog /app
COPY --from=build /src/core-version /app/core-version
COPY docker-entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh \
&& useradd --uid 10001 --create-home katalog \
&& mkdir -p /data /media \
&& chown katalog:katalog /data
USER katalog
EXPOSE 8765
ENTRYPOINT ["/app/entrypoint.sh"]
# domyślnie wstajemy jako serwer; front i MCP dzielą port
CMD ["serve", "--host", "0.0.0.0"]
+76
View File
@@ -0,0 +1,76 @@
# Wariant pod NVIDIĘ — do transkrypcji na maszynie z kartą.
#
# Osobny plik, a nie przełącznik w Dockerfile, bo różnią się obrazy bazowe obu
# etapów: whisper.cpp trzeba zbudować przy toolkicie CUDA, a obraz uruchomieniowy
# musi nieść biblioteki CUDA i dopiero do nich dołożyć JRE. Wciśnięcie tego w
# jeden plik dałoby więcej warunków niż treści.
#
# Budowanie i uruchomienie (wymaga NVIDIA Container Toolkit po stronie hosta;
# na Windowsie: Docker Desktop z WSL2):
# docker build -f Dockerfile.cuda -t rex-catalog:cuda .
# docker run --rm --gpus all -v D:\Reksio:/media:ro -v katalog:/data \
# -v D:\modele:/models:ro -e WHISPER_MODEL=/models/ggml-medium.bin \
# rex-catalog:cuda transcribe
#
# Po przeniesieniu bazy z innej maszyny najpierw przypnij ją do kolekcji:
# docker run --rm -v D:\Reksio:/media:ro -v katalog:/data rex-catalog:cuda relocate /media
FROM eclipse-temurin:21-jdk AS build
WORKDIR /src
COPY gradlew settings.gradle build.gradle gradle.properties ./
COPY gradle ./gradle
COPY vendor ./vendor
RUN ./gradlew --no-daemon dependencies --configuration runtimeClasspath > /dev/null 2>&1 || true
COPY src ./src
RUN ./gradlew --no-daemon installDist \
&& grep '^coreVersion=' gradle.properties | cut -d= -f2 > /src/core-version
# CMAKE_CUDA_ARCHITECTURES=86 to Ampere z GeForce RTX 30 (3060 ma 8.6).
# Budowanie pod wszystkie architektury trwa wielokrotnie dłużej i puchnie,
# a i tak nie przyda się na innej karcie bez przebudowy.
FROM nvidia/cuda:12.6.2-devel-ubuntu24.04 AS whisper
ARG CUDA_ARCH=86
RUN apt-get update \
&& apt-get install -y --no-install-recommends git cmake build-essential ca-certificates \
&& git clone --depth 1 --branch v1.9.2 https://github.com/ggerganov/whisper.cpp /src \
&& cmake -S /src -B /src/build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
-DGGML_NATIVE=OFF -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH} \
-DWHISPER_BUILD_TESTS=OFF -DWHISPER_BUILD_EXAMPLES=ON \
&& cmake --build /src/build --target whisper-cli -j "$(nproc)" \
&& mkdir -p /out/bin /out/lib \
&& cp /src/build/bin/whisper-cli /out/bin/ \
&& find /src/build \( -name 'libwhisper.so*' -o -name 'libggml*.so*' \) \
-exec cp -P {} /out/lib/ \; \
&& rm -rf /var/lib/apt/lists/* /src
FROM nvidia/cuda:12.6.2-runtime-ubuntu24.04
WORKDIR /app
VOLUME ["/data"]
ENV CATALOG_DATA=/data
ENV WHISPER_BIN=/usr/local/bin/whisper-cli
# JRE z repozytoriów, bo obraz bazowy niesie CUDA, a nie Javę; ffmpeg do
# przepróbkowania nagrań na 16 kHz mono, których whisper.cpp wymaga
RUN apt-get update \
&& apt-get install -y --no-install-recommends openjdk-21-jre-headless ffmpeg \
&& rm -rf /var/lib/apt/lists/*
COPY --from=whisper /out/bin/whisper-cli /usr/local/bin/whisper-cli
COPY --from=whisper /out/lib/ /usr/local/lib/
RUN ldconfig
COPY --from=build /src/build/install/rex-catalog /app
COPY --from=build /src/core-version /app/core-version
COPY docker-entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh \
&& useradd --uid 10001 --create-home katalog \
&& mkdir -p /data /media \
&& chown katalog:katalog /data
USER katalog
EXPOSE 8765
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["serve", "--host", "0.0.0.0"]
+4
View File
@@ -29,6 +29,10 @@ dependencies {
runtimeOnly "com.badlogicgames.gdx:gdx-platform:$gdxVersion:natives-desktop" runtimeOnly "com.badlogicgames.gdx:gdx-platform:$gdxVersion:natives-desktop"
implementation 'org.xerial:sqlite-jdbc:3.46.1.3' implementation 'org.xerial:sqlite-jdbc:3.46.1.3'
// JSON-RPC serwera MCP. Transport stoi na com.sun.net.httpserver z JDK,
// więc poza parserem nie dokładamy nic.
implementation 'com.google.code.gson:gson:2.11.0'
} }
application { application {
+23
View File
@@ -0,0 +1,23 @@
services:
katalog:
build: .
image: rex-catalog:latest
ports:
# front pod /, MCP pod /mcp — jeden port, jeden proces
- "127.0.0.1:8765:8765"
volumes:
# kolekcja tylko do odczytu: katalog niczego w niej nie zmienia,
# a obrazy płyt są nieodtwarzalne
- ${COLLECTION:-./collection}:/media:ro
# baza oraz cache skryptów i miniatur — jedyny stan wart trzymania
- catalog-data:/data
# modele whisper.cpp: nie wchodzą do obrazu, bo ważą więcej niż on sam
# i każdy trzyma inny. Bez nich katalog działa, tylko bez transkrypcji.
- ${WHISPER_MODELS:-./models}:/models:ro
environment:
# wskaż konkretny plik modelu, np. ggml-large-v3.bin z podmontowanego katalogu
WHISPER_MODEL: ${WHISPER_MODEL:-/models/ggml-large-v3.bin}
restart: unless-stopped
volumes:
catalog-data:
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
# Launcher generowany przez Gradle traktuje wszystko po nazwie skryptu jako
# argumenty aplikacji, więc właściwości JVM muszą iść przez JAVA_OPTS.
set -e
# wersja :core jest przypięta submodułem w czasie budowania obrazu; bez niej
# copy.core_version zapisałoby "nieznana" i przestałoby działać unieważnianie
# cache'u pochodnych po bumpie emulatora
CORE_VERSION="${CORE_VERSION:-$(cat /app/core-version 2>/dev/null || echo nieznana)}"
JAVA_OPTS="-Dfile.encoding=UTF-8 -Dcatalog.data=${CATALOG_DATA:-/data} -Dcatalog.coreVersion=${CORE_VERSION} ${JAVA_OPTS}"
export JAVA_OPTS
exec /app/bin/rex-catalog "$@"
@@ -1,14 +1,34 @@
package pl.genschu.rexcatalog; package pl.genschu.rexcatalog;
import pl.genschu.rexcatalog.catalog.Editor;
import pl.genschu.rexcatalog.catalog.Promoter;
import pl.genschu.rexcatalog.catalog.Relocator;
import pl.genschu.rexcatalog.db.Database; import pl.genschu.rexcatalog.db.Database;
import pl.genschu.rexcatalog.ingest.Ingestor; import pl.genschu.rexcatalog.ingest.Ingestor;
import pl.genschu.rexcatalog.ingest.LanguageDetector;
import pl.genschu.rexcatalog.ingest.MetadataDetector;
import pl.genschu.rexcatalog.media.AudioIndexer;
import pl.genschu.rexcatalog.media.DialogueIndexer;
import pl.genschu.rexcatalog.media.GraphicsIndexer;
import pl.genschu.rexcatalog.script.RefIndexer;
import pl.genschu.rexcatalog.transcribe.Transcriber;
import pl.genschu.rexcatalog.mcp.McpServer;
import pl.genschu.rexcatalog.web.WebServer;
import pl.genschu.rexcatalog.script.ScriptDecoder;
import pl.genschu.rexcatalog.script.ScriptSearch;
import com.sun.net.httpserver.HttpServer;
import java.io.File; import java.io.File;
import java.net.InetSocketAddress;
import java.nio.file.Path; import java.nio.file.Path;
import java.sql.ResultSet; import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement; import java.sql.Statement;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.concurrent.Executors;
/** /**
* CLI katalogu. Krok 1: indeksowanie obrazów do SQLite. * CLI katalogu. Krok 1: indeksowanie obrazów do SQLite.
@@ -20,6 +40,8 @@ public final class Main {
System.getProperty("catalog.coreVersion", "nieznana"); System.getProperty("catalog.coreVersion", "nieznana");
public static void main(String[] args) throws Exception { public static void main(String[] args) throws Exception {
// dekodowanie grafiki idzie przez AWT; bez tego JVM próbowałby otworzyć ekran
System.setProperty("java.awt.headless", "true");
if (args.length == 0) { if (args.length == 0) {
usage(); usage();
return; return;
@@ -31,6 +53,19 @@ public final class Main {
case "ingest" -> ingest(dataDir, args); case "ingest" -> ingest(dataDir, args);
case "list" -> list(dataDir); case "list" -> list(dataDir);
case "stats" -> stats(dataDir); case "stats" -> stats(dataDir);
case "decode" -> decode(dataDir, args);
case "probe" -> probe(dataDir, args);
case "transcribe" -> transcribe(dataDir, args);
case "analyze" -> analyze(dataDir);
case "promote" -> promote(dataDir);
case "relocate" -> relocate(dataDir, args);
case "titles" -> titles(dataDir);
case "set" -> set(dataDir, args);
case "lang" -> lang(dataDir, args);
case "mcp" -> mcp(dataDir, args);
case "serve" -> serve(dataDir, args);
case "find" -> find(dataDir, args);
case "cat" -> cat(dataDir, args);
default -> usage(); default -> usage();
} }
} finally { } finally {
@@ -97,6 +132,509 @@ public final class Main {
} }
} }
private static void decode(Path dataDir, String[] args) throws Exception {
boolean force = args.length > 1 && args[1].equals("--force");
Headless.boot();
try (Database db = Database.open(dataDir)) {
ScriptDecoder decoder = new ScriptDecoder(db, dataDir, CORE_VERSION);
int pending = force ? -1 : decoder.pendingCount();
if (pending == 0) {
System.out.println("Cache skryptów aktualny (core " + CORE_VERSION + ").");
return;
}
System.out.printf("Odszyfrowuję skrypty (core %s)%s%n", CORE_VERSION,
force ? ", wymuszone od nowa" : "");
long started = System.currentTimeMillis();
ScriptDecoder.Stats stats = decoder.run(force);
long seconds = (System.currentTimeMillis() - started) / 1000;
System.out.printf("%nOdszyfrowanych: %d powtórzeń pominiętych: %d błędów: %d (%ds)%n",
stats.decoded(), stats.skipped(), stats.failed(), seconds);
System.out.printf("W indeksie: %d skryptów%n",
new ScriptSearch(db, dataDir).indexedCount());
}
}
/**
* Czyta nagłówki zasobów i buduje powiązania między nimi: fakty o dźwięku,
* zawartość archiwów, wiersze dialogów i odwołania ze skryptów.
*/
private static void probe(Path dataDir, String[] args) throws Exception {
boolean force = args.length > 1 && args[1].equals("--force");
Headless.boot();
try (Database db = Database.open(dataDir)) {
AudioIndexer audio = new AudioIndexer(db);
GraphicsIndexer graphics = new GraphicsIndexer(db, dataDir);
DialogueIndexer dialogue = new DialogueIndexer(db);
RefIndexer refs = new RefIndexer(db, dataDir);
if (!force && audio.pendingCount() == 0 && graphics.pendingCount() == 0
&& dialogue.pendingCount() == 0 && refs.pendingCount() == 0) {
System.out.println("Nic do przerobienia — wszystko już zbadane.");
return;
}
long started = System.currentTimeMillis();
System.out.println("Czytam nagłówki dźwięku i archiwa...");
AudioIndexer.Stats audioStats = audio.run(force);
System.out.printf(" %d plików dźwiękowych, %d archiwów (%d wpisów), błędów: %d%n",
audioStats.audio(), audioStats.archives(), audioStats.entries(),
audioStats.failed());
System.out.println("Dekoduję grafikę i buduję miniatury...");
GraphicsIndexer.Stats graphicsStats = graphics.run(force);
System.out.printf(" %d obrazów, %d animacji (%d klatek), nieczytelnych: %d%n",
graphicsStats.images(), graphicsStats.animations(), graphicsStats.frames(),
graphicsStats.failed());
System.out.println("Czytam tabele dialogów...");
DialogueIndexer.Stats dialogueStats = dialogue.run(force);
System.out.printf(" %d plików, %d wierszy, błędów: %d%n",
dialogueStats.files(), dialogueStats.lines(), dialogueStats.failed());
System.out.println("Zbieram odwołania ze skryptów...");
RefIndexer.Stats refStats = refs.run(force);
System.out.printf(" %d skryptów, %d odwołań, błędów: %d%n",
refStats.scripts(), refStats.refs(), refStats.failed());
System.out.printf("%nGotowe (%ds).%n", (System.currentTimeMillis() - started) / 1000);
}
}
/**
* Transkrypcja mowy. Osobna komenda, nie część {@code probe} — trwa godzinami
* i wymaga narzędzi spoza tego repozytorium, więc uruchamia się ją świadomie.
*/
private static void transcribe(Path dataDir, String[] args) throws Exception {
Transcriber.Tool tool = Transcriber.detect();
if (!tool.ready()) {
System.err.println(tool.note());
System.err.println("""
Ścieżki można wskazać wprost:
-Dcatalog.whisper.bin=/opt/homebrew/bin/whisper-cli
-Dcatalog.whisper.model=~/modele/ggml-large-v3.bin""");
return;
}
Headless.boot();
Integer edition = null;
String editionArg = option(args, "--edition", null);
if (editionArg != null) {
edition = Integer.parseInt(editionArg);
}
int limit = Integer.parseInt(option(args, "--limit", "0"));
boolean redo = Arrays.asList(args).contains("--redo");
try (Database db = Database.open(dataDir)) {
Object lock = new Object();
Transcriber transcriber = new Transcriber(db, lock);
System.out.printf("Model: %s%n", Path.of(tool.model()).getFileName());
transcriber.start(new Transcriber.Scope(edition, limit, redo));
Runtime.getRuntime().addShutdownHook(new Thread(transcriber::stop));
Transcriber.Progress last = null;
while (transcriber.running()) {
Transcriber.Progress now = transcriber.progress();
if (last == null || now.done() != last.done() || now.failed() != last.failed()) {
System.out.printf("\r %d/%d błędów: %d %-40s", now.done() + now.failed(),
now.total(), now.failed(),
now.current() == null ? "" : truncate(now.current(), 40));
System.out.flush();
last = now;
}
Thread.sleep(500);
}
Transcriber.Progress end = transcriber.progress();
System.out.printf("%n%s: %d przetranskrybowanych, %d nieudanych%n",
end.state(), end.done(), end.failed());
if (end.note() != null && end.total() == 0) {
System.out.println(end.note());
}
}
}
private static void analyze(Path dataDir) throws Exception {
// LanguageDetector czyta install.ini wprost z obrazu, więc potrzebuje :core
Headless.boot();
try (Database db = Database.open(dataDir)) {
List<MetadataDetector.RootResult> results =
new MetadataDetector(db, dataDir, CORE_VERSION).run();
if (results.isEmpty()) {
System.out.println("Brak plików application.def — uruchom najpierw: ingest, decode");
return;
}
System.out.printf("Analizuję application.def w %d korzeniach%n%n", results.size());
for (MetadataDetector.RootResult result : results) {
String where = result.rootPath().isEmpty()
? result.copyName()
: result.copyName() + " / " + result.rootPath();
System.out.println(" " + truncate(where, 60));
if (result.note() != null) {
System.out.println(" ! " + result.note());
continue;
}
for (MetadataDetector.Fact fact : result.facts()) {
if (fact.field().equals("app_def_sha1")) {
continue;
}
System.out.printf(" %-16s %-28s (pewność %.2f)%n",
fact.field(), truncate(fact.value(), 28), fact.confidence());
}
System.out.println();
}
languages(db);
}
}
/** Wersje językowe z install.ini — osobny detektor, bo czyta obraz, nie cache. */
private static void languages(Database db) throws Exception {
List<String> drift = LanguageDetector.verifyAgainstCore();
if (!drift.isEmpty()) {
System.err.println("! tablica języków rozjechała się z :core: "
+ String.join("; ", drift));
}
List<LanguageDetector.RootResult> results =
new LanguageDetector(db, CORE_VERSION).run();
if (results.isEmpty()) {
return;
}
System.out.println("Wersje językowe z install.ini\n");
for (LanguageDetector.RootResult result : results) {
String where = result.rootPath().isEmpty()
? result.copyName()
: result.copyName() + " / " + result.rootPath();
System.out.printf(" %-46s %-18s %s%n", truncate(where, 46),
String.join(" ", result.languages()),
result.note() != null ? "! " + result.note()
: "dubbing: " + String.join(" ", result.audioLanguages()));
}
System.out.println();
}
private static void promote(Path dataDir) throws Exception {
try (Database db = Database.open(dataDir)) {
List<Promoter.Assignment> assignments = new Promoter(db).run();
if (assignments.isEmpty()) {
System.out.println("Brak korzeni gier — uruchom najpierw: ingest");
return;
}
System.out.printf("Podpinam %d korzeni do tytułów i wydań%n%n", assignments.size());
int review = 0;
for (Promoter.Assignment a : assignments) {
String where = a.rootPath().isEmpty() ? a.copyName()
: a.copyName() + " / " + a.rootPath();
System.out.printf(" %-46s → %-32s %s%s%n", truncate(where, 46),
truncate(a.title(), 32),
a.label() == null ? "" : "[" + truncate(a.label(), 30) + "]",
a.ambiguous() ? " ⚠ do przejrzenia" : "");
if (a.ambiguous()) {
review++;
}
}
if (review > 0) {
System.out.printf("%n%d tytułów wymaga potwierdzenia nazwy — popraw przez:%n"
+ " set title <id|fragment> name=\"...\"%n", review);
}
}
}
/**
* Przestawia zapisane ścieżki do obrazów na nowe miejsce. Potrzebne po
* przeniesieniu bazy na inną maszynę albo po przełożeniu kolekcji — bez tego
* dźwięk, klatki animacji i transkrypcja nie mają skąd czytać.
*/
private static void relocate(Path dataDir, String[] args) throws Exception {
if (args.length < 2) {
System.err.println("""
relocate <katalog-kolekcji> [--verify] [--dry-run]
relocate /Volumes/Dysk/Reksio
relocate D:\\Reksio --verify
Dopasowuje po nazwie pliku i sprawdza rozmiar; --verify dokłada
porównanie pełnego odcisku SHA-256 (wolniej, ale bez wątpliwości).
Nazwy porównuje po normalizacji Unicode, więc kolekcja z macOS
dogaduje się z tą samą kolekcją na Windowsie.
PRZENIESIENIE KATALOGU NA INNĄ MASZYNĘ
1. skopiuj data/catalog.sqlite (a jeśli chcesz mieć podglądy
i skrypty — cały katalog data/)
2. na nowej maszynie: relocate <gdzie-leży-kolekcja>
3. sprawdź: list — kolumna STATUS nie może być pusta
Baza zapisuje bezwzględne ścieżki do obrazów płyt, bo katalog
czyta z nich dźwięk, klatki animacji i nagrania do transkrypcji.
Bez tego kroku wszystko widać, ale niczego nie da się odtworzyć.
Wracając z powrotem, uruchom relocate jeszcze raz. Uwaga: baza
wędruje w całości, więc jeśli w międzyczasie zmienisz coś po
obu stronach, jedna z wersji przepadnie — najbezpieczniej
trzymać się zasady, że w danej chwili pracuje jedna maszyna.
""");
return;
}
boolean verify = Arrays.asList(args).contains("--verify");
boolean dryRun = Arrays.asList(args).contains("--dry-run");
try (Database db = Database.open(dataDir)) {
List<Relocator.Result> results;
try {
results = new Relocator(db).run(Path.of(args[1]), verify, dryRun);
} catch (IllegalArgumentException e) {
System.err.println(e.getMessage());
return;
}
int przypiete = 0;
int problemy = 0;
for (Relocator.Result result : results) {
switch (result.status()) {
case "przypięta" -> {
System.out.printf(" %-44s → %s%n",
truncate(result.displayName(), 44), result.newPath());
przypiete++;
}
case "bez zmian" -> System.out.printf(" %-44s bez zmian%n",
truncate(result.displayName(), 44));
default -> {
System.out.printf(" %-44s ! %s%n",
truncate(result.displayName(), 44), result.note());
problemy++;
}
}
}
System.out.printf("%nPrzypiętych: %d nierozwiązanych: %d%s%n",
przypiete, problemy, dryRun ? " (próba, nic nie zapisano)" : "");
if (problemy > 0) {
System.out.println("Kopie bez dopasowania zostają ze starą ścieżką — "
+ "katalog nadal je pokazuje, ale nie odczyta z nich zawartości.");
}
}
}
private static void titles(Path dataDir) throws Exception {
try (Database db = Database.open(dataDir);
Statement st = db.connection().createStatement();
ResultSet rs = st.executeQuery("""
SELECT t.id AS title_id, t.name AS title, t.series, t.publisher,
e.id AS edition_id, e.label,
COALESCE(e.release_date_override, (
SELECT d.value FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id
AND d.root_path = g.root_path
WHERE g.edition_id = e.id AND d.field = 'release_date' LIMIT 1
)) AS release_date,
COALESCE(e.engine_override, (
SELECT g.engine_detected FROM game_root g
WHERE g.edition_id = e.id LIMIT 1
)) AS engine,
COALESCE(e.engine_version_override, (
SELECT d.value FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id
AND d.root_path = g.root_path
WHERE g.edition_id = e.id AND d.field = 'engine_version' LIMIT 1
)) AS engine_version,
(SELECT GROUP_CONCAT(DISTINCT lang) FROM edition_language
WHERE edition_id = e.id) AS langs,
(SELECT COUNT(*) FROM game_root WHERE edition_id = e.id) AS roots
FROM edition e
LEFT JOIN title t ON t.id = e.title_id
ORDER BY t.series IS NULL, t.series, t.name, release_date
""")) {
System.out.printf("%-4s %-30s %-26s %-11s %-13s %-9s %s%n",
"ID", "TYTUŁ", "WYDANIE", "DATA", "SILNIK", "JĘZYKI", "KOPIE");
while (rs.next()) {
System.out.printf("%-4d %-30s %-26s %-11s %-13s %-9s %d%n",
rs.getInt("edition_id"),
truncate(rs.getString("title"), 30),
truncate(nvl(rs.getString("label")), 26),
nvl(rs.getString("release_date")),
truncate(nvl(rs.getString("engine")), 13),
truncate(nvl(rs.getString("langs")), 9),
rs.getInt("roots"));
}
System.out.println("\nID to identyfikator wydania — używaj go w set edition i lang.");
}
}
private static void set(Path dataDir, String[] args) throws Exception {
if (args.length < 4) {
System.err.println("""
set <encja> <id|fragment> pole=wartość [pole=wartość...]
set copy "Wojna Troj" source_kind=wlasny_zgraj rip_tool=dd
set edition 3 release_date_override=2004-11-15 distributor="Aidem Media"
set title 2 name="Poznaj Mity: Herkules"
""");
System.err.println("Encje: " + String.join(", ", Editor.entities()));
return;
}
try (Database db = Database.open(dataDir)) {
try {
int changed = new Editor(db).set(args[1], args[2],
Arrays.asList(args).subList(3, args.length));
System.out.printf("Zmienionych wierszy: %d%n", changed);
} catch (IllegalArgumentException e) {
System.err.println(e.getMessage());
}
}
}
private static void lang(Path dataDir, String[] args) throws Exception {
if (args.length < 5 || !(args[1].equals("add") || args[1].equals("rm"))) {
System.err.println("""
lang add|rm <wydanie> <kod> <rola>
lang add 3 pl audio
lang add 3 cs text
lang rm 3 hu ui
Rola: audio, text, ui. Wydanie: id albo fragment nazwy tytułu.
""");
return;
}
try (Database db = Database.open(dataDir)) {
Editor editor = new Editor(db);
try {
if (args[1].equals("add")) {
editor.addLanguage(args[2], args[3], args[4]);
System.out.println("Dodano.");
} else {
int removed = editor.removeLanguage(args[2], args[3], args[4]);
System.out.println(removed > 0 ? "Usunięto." : "Nie było takiego wpisu.");
}
} catch (IllegalArgumentException e) {
System.err.println(e.getMessage());
}
}
}
private static void mcp(Path dataDir, String[] args) throws Exception {
String host = option(args, "--host", "127.0.0.1");
int port = Integer.parseInt(option(args, "--port", "8765"));
try (Database db = Database.open(dataDir)) {
McpServer server = new McpServer(db, dataDir);
server.start(host, port);
System.out.printf("Serwer MCP nasłuchuje na http://%s:%d/mcp%n", host, port);
System.out.println("Transport: Streamable HTTP, bez sesji, tylko do odczytu.");
System.out.println("Zatrzymanie: Ctrl+C");
Runtime.getRuntime().addShutdownHook(new Thread(server::stop));
// HttpServer pracuje na własnych wątkach, więc główny musi zaczekać
Thread.currentThread().join();
}
}
/** Front i MCP na jednym porcie — jeden proces, jeden kontener, jedno połączenie z bazą. */
private static void serve(Path dataDir, String[] args) throws Exception {
String host = option(args, "--host", "127.0.0.1");
int port = Integer.parseInt(option(args, "--port", "8765"));
try (Database db = Database.open(dataDir)) {
McpServer mcp = new McpServer(db, dataDir);
HttpServer http = HttpServer.create(new InetSocketAddress(host, port), 0);
mcp.register(http);
new WebServer(db, dataDir, mcp.lock()).register(http);
http.setExecutor(Executors.newFixedThreadPool(4));
http.start();
System.out.printf("Front: http://%s:%d/%n", host, port);
System.out.printf("MCP: http://%s:%d/mcp%n", host, port);
System.out.println("Zatrzymanie: Ctrl+C");
Runtime.getRuntime().addShutdownHook(new Thread(() -> http.stop(0)));
Thread.currentThread().join();
}
}
private static String option(String[] args, String name, String fallback) {
for (int i = 1; i < args.length - 1; i++) {
if (args[i].equals(name)) {
return args[i + 1];
}
}
return fallback;
}
private static void find(Path dataDir, String[] args) throws Exception {
if (args.length < 2) {
System.err.println("find wymaga zapytania, np. find SHOWCURSOR");
return;
}
String query = String.join(" ", Arrays.copyOfRange(args, 1, args.length));
try (Database db = Database.open(dataDir)) {
ScriptSearch search = new ScriptSearch(db, dataDir);
ScriptSearch.Result result;
try {
result = search.find(query, 25);
} catch (SQLException e) {
System.err.println("Nieprawidłowe zapytanie FTS5: " + e.getMessage());
return;
}
List<ScriptSearch.Hit> hits = result.hits();
if (hits.isEmpty()) {
System.out.println("Brak trafień dla: " + result.effectiveQuery());
return;
}
System.out.printf("Trafienia dla %s (%d)%s:%n%n",
result.effectiveQuery(), hits.size(),
result.quoted() ? " — potraktowane jako fraza" : "");
for (ScriptSearch.Hit hit : hits) {
System.out.println(" " + String.join("\n ", hit.paths()));
System.out.println(" w: " + String.join(", ", hit.games()));
System.out.println(" " + hit.snippet().replaceAll("\\s+", " ").trim());
System.out.println(" sha1: " + hit.sha1());
System.out.println();
}
}
}
private static void cat(Path dataDir, String[] args) throws Exception {
if (args.length < 2) {
System.err.println("cat wymaga ścieżki albo SHA-1, np. cat arcade.cnv");
return;
}
try (Database db = Database.open(dataDir)) {
ScriptSearch search = new ScriptSearch(db, dataDir);
List<String> candidates = search.resolve(args[1]);
if (candidates.isEmpty()) {
System.err.println("Nic nie pasuje do: " + args[1]);
return;
}
if (candidates.size() > 1) {
System.err.printf("Niejednoznaczne — %d różnych wersji tego pliku:%n",
candidates.size());
for (String sha1 : candidates) {
System.err.printf(" %s %s%n", sha1,
String.join(", ", search.gamesFor(sha1)));
}
System.err.println("Podaj SHA-1, żeby wskazać konkretną.");
return;
}
String sha1 = candidates.get(0);
String body = search.body(sha1);
if (body == null) {
System.err.println("Ten blob nie jest w cache'u — uruchom najpierw: decode");
return;
}
System.out.println(body);
}
}
private static void list(Path dataDir) throws Exception { private static void list(Path dataDir) throws Exception {
try (Database db = Database.open(dataDir); try (Database db = Database.open(dataDir);
Statement st = db.connection().createStatement(); Statement st = db.connection().createStatement();
@@ -180,6 +718,22 @@ public final class Main {
list wypisz zaindeksowane kopie list wypisz zaindeksowane kopie
stats statystyki bazy stats statystyki bazy
decode [--force] odszyfruj skrypty do cache'u i indeksu
probe [--force] zbadaj dźwięk, grafikę, dialogi i odwołania
transcribe [--edition N] [--limit N] [--redo]
rozpoznaj mowę przez whisper.cpp (wynik maszynowy)
analyze wyciągnij metadane z application.def
promote utwórz tytuły i wydania z wykrytych faktów
relocate <katalog> przypnij bazę do kolekcji w nowym miejscu
titles lista tytułów z metadanymi
set <encja> <id> ... edytuj metadane ręczne
lang add|rm ... wersje językowe wydania
find <zapytanie> przeszukaj skrypty (składnia FTS5)
cat <ścieżka|sha1> wypisz odszyfrowany skrypt
mcp [--host H] [--port P] serwer MCP (Streamable HTTP, domyślnie 127.0.0.1:8765)
serve [--host H] [--port P] front i MCP na jednym porcie
Baza: -Dcatalog.data=<katalog> (domyślnie ./data) Baza: -Dcatalog.data=<katalog> (domyślnie ./data)
"""); """);
} }
@@ -0,0 +1,232 @@
package pl.genschu.rexcatalog.catalog;
import pl.genschu.rexcatalog.db.Database;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
/**
* Ręczna edycja metadanych, których żaden detektor nie wymyśli: skąd pochodzi plik,
* jakie ma wersje językowe, czym to skompilowano.
*
* <p>Pola są na białej liście per encja — literówka w nazwie kolumny ma się skończyć
* komunikatem, a nie zapytaniem SQL sklejonym z wejścia.
*/
public final class Editor {
private static final Map<String, Set<String>> EDITABLE = Map.of(
"copy", new LinkedHashSet<>(List.of(
"source_kind", "source_url", "source_ref", "acquired_at",
"rip_tool", "media_note", "verified", "notes", "display_name")),
"edition", new LinkedHashSet<>(List.of(
"label", "engine_override", "engine_version_override",
"compiler_override", "release_date_override", "distributor", "notes")),
"title", new LinkedHashSet<>(List.of("name", "series", "publisher"))
);
private static final Set<String> SOURCE_KINDS =
Set.of("wlasny_zgraj", "archive_org", "redump", "inne", "nieznane");
private static final Set<String> LANGUAGE_ROLES = Set.of("audio", "text", "ui");
private final Database db;
public Editor(Database db) {
this.db = db;
}
public static Set<String> editableFields(String entity) {
return EDITABLE.getOrDefault(entity, Set.of());
}
public static Set<String> entities() {
return EDITABLE.keySet();
}
/** Zwraca liczbę zmienionych wierszy albo rzuca z opisem, co jest nie tak. */
public int set(String entity, String id, List<String> assignments) throws Exception {
Set<String> allowed = EDITABLE.get(entity);
if (allowed == null) {
throw new IllegalArgumentException("Nieznana encja: " + entity
+ " (dostępne: " + String.join(", ", EDITABLE.keySet()) + ")");
}
if (assignments.isEmpty()) {
throw new IllegalArgumentException("Podaj co najmniej jedno przypisanie pole=wartość");
}
List<String> columns = new ArrayList<>();
List<String> values = new ArrayList<>();
for (String assignment : merge(assignments)) {
int eq = assignment.indexOf('=');
if (eq <= 0) {
throw new IllegalArgumentException("Oczekiwano pole=wartość, dostałem: " + assignment);
}
String column = assignment.substring(0, eq).trim().toLowerCase(Locale.ROOT);
String value = unquote(assignment.substring(eq + 1).trim());
if (!allowed.contains(column)) {
throw new IllegalArgumentException("Pole '" + column + "' nie jest edytowalne w "
+ entity + " (dostępne: " + String.join(", ", allowed) + ")");
}
if (column.equals("source_kind") && !SOURCE_KINDS.contains(value)) {
throw new IllegalArgumentException("source_kind musi być jednym z: "
+ String.join(", ", SOURCE_KINDS));
}
columns.add(column);
values.add(value.isEmpty() ? null : value);
}
long rowId = resolveId(entity, id);
String sql = "UPDATE " + entity + " SET "
+ String.join(", ", columns.stream().map(c -> c + " = ?").toList())
+ " WHERE id = ?";
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
int index = 1;
for (String value : values) {
st.setString(index++, value);
}
st.setLong(index, rowId);
return st.executeUpdate();
}
}
/**
* Skleja wartości wielowyrazowe. Token bez znaku równości nie może być osobnym
* przypisaniem, więc dopisujemy go do poprzedniej wartości — dzięki temu
* {@code distributor=Komputer Świat Gry} działa bez walki z cudzysłowami,
* które i tak gubi {@code gradle run --args}.
*/
private static List<String> merge(List<String> tokens) {
List<String> merged = new ArrayList<>();
for (String token : tokens) {
if (token.contains("=") || merged.isEmpty()) {
merged.add(token);
} else {
merged.set(merged.size() - 1, merged.get(merged.size() - 1) + " " + token);
}
}
return merged;
}
private static String unquote(String value) {
if (value.length() >= 2 && value.startsWith("\"") && value.endsWith("\"")) {
return value.substring(1, value.length() - 1);
}
return value;
}
/**
* Przyjmuje albo liczbowy identyfikator, albo fragment nazwy — przy pracy z CLI
* nikt nie pamięta, że Wojna Trojańska ma copy_id 13.
*/
private long resolveId(String entity, String id) throws Exception {
if (id.matches("\\d+")) {
return Long.parseLong(id);
}
String column = switch (entity) {
case "copy" -> "display_name";
case "title" -> "name";
case "edition" -> "label";
default -> throw new IllegalArgumentException("Encja " + entity
+ " wymaga liczbowego id");
};
List<Long> ids = new ArrayList<>();
List<String> names = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT id, " + column + " FROM " + entity
+ " WHERE lower(" + column + ") LIKE ? ORDER BY id")) {
st.setString(1, "%" + id.toLowerCase(Locale.ROOT) + "%");
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
ids.add(rs.getLong(1));
names.add(rs.getString(2));
}
}
}
if (ids.isEmpty()) {
throw new IllegalArgumentException("Nic nie pasuje do: " + id);
}
if (ids.size() > 1) {
StringBuilder sb = new StringBuilder("Niejednoznaczne — pasuje " + ids.size() + ":");
for (int i = 0; i < ids.size(); i++) {
sb.append("\n ").append(ids.get(i)).append(" ").append(names.get(i));
}
throw new IllegalArgumentException(sb.toString());
}
return ids.get(0);
}
/**
* Wersje językowe są listą, nie polem — wydanie bywa miksem (polski + czeski +
* węgierski), a rola rozdziela dubbing od napisów i interfejsu.
*/
public void addLanguage(String editionId, String lang, String role) throws Exception {
if (!LANGUAGE_ROLES.contains(role)) {
throw new IllegalArgumentException("Rola musi być jedną z: "
+ String.join(", ", LANGUAGE_ROLES));
}
long id = resolveEdition(editionId);
try (PreparedStatement st = db.connection().prepareStatement("""
INSERT INTO edition_language (edition_id, lang, role) VALUES (?, ?, ?)
ON CONFLICT(edition_id, lang, role) DO NOTHING
""")) {
st.setLong(1, id);
st.setString(2, lang.toLowerCase(Locale.ROOT));
st.setString(3, role);
st.executeUpdate();
}
}
public int removeLanguage(String editionId, String lang, String role) throws Exception {
long id = resolveEdition(editionId);
try (PreparedStatement st = db.connection().prepareStatement(
"DELETE FROM edition_language WHERE edition_id = ? AND lang = ? AND role = ?")) {
st.setLong(1, id);
st.setString(2, lang.toLowerCase(Locale.ROOT));
st.setString(3, role);
return st.executeUpdate();
}
}
/** Wydanie wskazujemy po id albo po nazwie tytułu, do którego należy. */
private long resolveEdition(String identifier) throws Exception {
if (identifier.matches("\\d+")) {
return Long.parseLong(identifier);
}
List<Long> ids = new ArrayList<>();
List<String> labels = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT e.id, t.name || COALESCE(' — ' || e.label, '')
FROM edition e LEFT JOIN title t ON t.id = e.title_id
WHERE lower(t.name) LIKE ? OR lower(e.label) LIKE ?
ORDER BY e.id
""")) {
String pattern = "%" + identifier.toLowerCase(Locale.ROOT) + "%";
st.setString(1, pattern);
st.setString(2, pattern);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
ids.add(rs.getLong(1));
labels.add(rs.getString(2));
}
}
}
if (ids.isEmpty()) {
throw new IllegalArgumentException("Żadne wydanie nie pasuje do: " + identifier);
}
if (ids.size() > 1) {
StringBuilder sb = new StringBuilder("Niejednoznaczne — pasuje " + ids.size() + ":");
for (int i = 0; i < ids.size(); i++) {
sb.append("\n ").append(ids.get(i)).append(" ").append(labels.get(i));
}
throw new IllegalArgumentException(sb.toString());
}
return ids.get(0);
}
}
@@ -0,0 +1,307 @@
package pl.genschu.rexcatalog.catalog;
import pl.genschu.rexcatalog.db.Database;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.text.Normalizer;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
/**
* Materializuje encje katalogu z faktów zebranych przez detektory: tworzy tytuły
* i wydania, po czym podpina do nich korzenie gier.
*
* <p>Operacja jest powtarzalna i nie nadpisuje pracy człowieka — pola {@code *_override},
* {@code label}, {@code distributor} i {@code notes} zostają nietknięte, a tytuł raz
* nazwany zachowuje nazwę, bo kluczem jest slug, nie nazwa.
*/
public final class Promoter {
private final Database db;
public Promoter(Database db) {
this.db = db;
}
public record Assignment(String copyName, String rootPath, String title, String label,
String fingerprint, boolean ambiguous) {
}
private record RootFacts(long copyId, String copyName, String rootPath, String copySha256,
String dllSha1, String appDefSha1, String edition, String family,
int rootsOnCopy) {
}
public List<Assignment> run() throws Exception {
List<RootFacts> roots = collect();
List<Assignment> assignments = new ArrayList<>();
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
try {
for (RootFacts root : roots) {
String fingerprint = fingerprint(root);
Naming naming = naming(root);
long titleId = upsertTitle(conn, naming);
long editionId = upsertEdition(conn, fingerprint, titleId, naming, root);
linkRoot(conn, root.copyId(), root.rootPath(), editionId);
applyLanguages(conn, root.copyId(), root.rootPath(), editionId);
assignments.add(new Assignment(root.copyName(), root.rootPath(),
naming.title(), naming.label(), fingerprint, naming.needsReview()));
}
// po zmianie reguły nazewniczej stare tytuły zostają bez wydań — sprzątamy je,
// żeby `titles` nie pokazywał duchów po poprzednim przebiegu
try (PreparedStatement st = conn.prepareStatement("""
DELETE FROM title WHERE id NOT IN
(SELECT title_id FROM edition WHERE title_id IS NOT NULL)
""")) {
st.executeUpdate();
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return assignments;
}
/**
* Klucz scalania. Hash biblioteki plus odcisk application.def — dwie kopie tego
* samego wydania (obraz i katalog rozpakowany) dają ten sam odcisk i jedno wydanie,
* a dwie gry z jednej płyty rozchodzą się na application.def.
*
* <p>Gdy nie ma ani biblioteki, ani opisu projektu (np. silnik KI), zostaje
* tożsamość nośnika — świadomie ostrożna, bo nie mamy czym scalać.
*/
private static String fingerprint(RootFacts root) {
if (root.dllSha1() != null || root.appDefSha1() != null) {
return "engine:" + nvl(root.dllSha1()) + "|app:" + nvl(root.appDefSha1());
}
return "copy:" + nvl(root.copySha256()) + "|root:" + root.rootPath();
}
/** Rozbiór nazwy z tablicy hashy na trzy poziomy katalogu. */
private record Naming(String title, String label, String series, boolean needsReview) {
}
/**
* Wpis w {@code KnownHashes} miesza dzieło z wydaniem: „Reksio i UFO (pierwsza wersja)"
* to tytuł plus etykieta w nawiasie. Rozdzielenie ich sprawia, że dwa wydania tej samej
* gry trafiają pod jeden tytuł, zamiast udawać dwie różne gry.
*
* <p>Dwupack wymaga dodatkowo rozbicia po ukośniku: jeden wpis opisuje obie gry
* („Poznaj Mity: Herkules/Odyseusz"), bo obie stoją na tej samej bibliotece.
* Wybór członu opieramy na nazwie katalogu korzenia.
*/
private static Naming naming(RootFacts root) {
String edition = root.edition();
if (edition == null || edition.isBlank()) {
// brak wpisu w tablicy hashy — z nazwy pliku wyjdzie co najwyżej przybliżenie
return new Naming(root.copyName().replaceAll("(?i)\\.(iso|zip)$", "").trim(),
null, null, true);
}
String base = edition;
String label = null;
int paren = edition.indexOf(" (");
if (paren > 0) {
base = edition.substring(0, paren).trim();
// w danych trafia się nawias bez domknięcia, stąd replaceAll zamiast substring
label = edition.substring(paren + 2).replaceAll("\\)\\s*$", "").trim();
}
boolean needsReview = false;
if (root.rootsOnCopy() > 1 && !root.rootPath().isEmpty()) {
int colon = base.lastIndexOf(':');
String prefix = colon >= 0 ? base.substring(0, colon + 1).trim() + " " : "";
String rest = colon >= 0 ? base.substring(colon + 1).trim() : base;
String matched = null;
for (String part : rest.split("/")) {
if (part.trim().equalsIgnoreCase(root.rootPath().trim())) {
matched = prefix + part.trim();
}
}
if (matched != null) {
base = matched;
} else {
base = base + " (" + root.rootPath() + ")";
needsReview = true;
}
}
return new Naming(base, label, series(base), needsReview);
}
/** „Poznaj Mity: Herkules" → seria „Poznaj Mity". Bez dwukropka nie zgadujemy. */
private static String series(String title) {
int colon = title.indexOf(": ");
return colon > 0 ? title.substring(0, colon).trim() : null;
}
/**
* Przenosi wykryte wersje językowe do wydania.
*
* <p>Tylko dopisuje: {@code DO NOTHING} przy konflikcie i żadnego kasowania,
* bo lista języków jest kuratorska. Detektor dokłada to, co widać na płycie
* ({@code install.ini}, podkatalogi {@code wavs/}), a ręczne wpisy zostają
* nietknięte.
*/
private void applyLanguages(Connection conn, long copyId, String rootPath, long editionId)
throws Exception {
try (PreparedStatement read = conn.prepareStatement("""
SELECT field, value FROM detection
WHERE copy_id = ? AND root_path = ? AND detector = 'LanguageDetector'
AND field IN ('languages', 'audio_languages')
""");
PreparedStatement write = conn.prepareStatement("""
INSERT INTO edition_language (edition_id, lang, role)
VALUES (?, ?, ?)
ON CONFLICT(edition_id, lang, role) DO NOTHING
""")) {
read.setLong(1, copyId);
read.setString(2, rootPath);
try (ResultSet rs = read.executeQuery()) {
while (rs.next()) {
String role = rs.getString("field").equals("audio_languages") ? "audio" : "ui";
String value = rs.getString("value");
if (value == null || value.isBlank()) {
continue;
}
for (String lang : value.split(",")) {
if (lang.isBlank()) {
continue;
}
write.setLong(1, editionId);
write.setString(2, lang.trim());
write.setString(3, role);
write.executeUpdate();
}
}
}
}
}
private long upsertTitle(Connection conn, Naming naming) throws Exception {
String slug = slug(naming.title());
// nazwa nie jest nadpisywana: klucz to slug, więc ręczna zmiana nazwy przetrwa
try (PreparedStatement st = conn.prepareStatement("""
INSERT INTO title (slug, name, series) VALUES (?, ?, ?)
ON CONFLICT(slug) DO UPDATE SET series = COALESCE(title.series, excluded.series)
""")) {
st.setString(1, slug);
st.setString(2, naming.title());
st.setString(3, naming.series());
st.executeUpdate();
}
try (PreparedStatement st = conn.prepareStatement("SELECT id FROM title WHERE slug = ?")) {
st.setString(1, slug);
try (ResultSet rs = st.executeQuery()) {
rs.next();
return rs.getLong(1);
}
}
}
private long upsertEdition(Connection conn, String fingerprint, long titleId,
Naming naming, RootFacts root) throws Exception {
// aktualizujemy wyłącznie pola mechaniczne — poprawki człowieka zostają,
// dlatego label tylko uzupełniamy, gdy jest pusty
try (PreparedStatement st = conn.prepareStatement("""
INSERT INTO edition (fingerprint, title_id, label, dll_sha1, app_def_sha1)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(fingerprint) DO UPDATE SET
title_id = excluded.title_id,
label = COALESCE(edition.label, excluded.label),
dll_sha1 = excluded.dll_sha1,
app_def_sha1 = excluded.app_def_sha1
""")) {
st.setString(1, fingerprint);
st.setLong(2, titleId);
st.setString(3, naming.label());
st.setString(4, root.dllSha1());
st.setString(5, root.appDefSha1());
st.executeUpdate();
}
try (PreparedStatement st = conn.prepareStatement(
"SELECT id FROM edition WHERE fingerprint = ?")) {
st.setString(1, fingerprint);
try (ResultSet rs = st.executeQuery()) {
rs.next();
return rs.getLong(1);
}
}
}
private void linkRoot(Connection conn, long copyId, String rootPath, long editionId)
throws Exception {
try (PreparedStatement st = conn.prepareStatement(
"UPDATE game_root SET edition_id = ? WHERE copy_id = ? AND root_path = ?")) {
st.setLong(1, editionId);
st.setLong(2, copyId);
st.setString(3, rootPath);
st.executeUpdate();
}
}
private List<RootFacts> collect() throws Exception {
Map<Long, Integer> rootsPerCopy = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT copy_id, COUNT(*) n FROM game_root GROUP BY copy_id");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
rootsPerCopy.put(rs.getLong("copy_id"), rs.getInt("n"));
}
}
List<RootFacts> out = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT g.copy_id, g.root_path, c.display_name, c.sha256,
MAX(CASE WHEN d.field = 'engine_dll_sha1' THEN d.value END) AS dll_sha1,
MAX(CASE WHEN d.field = 'app_def_sha1' THEN d.value END) AS app_def_sha1,
MAX(CASE WHEN d.field = 'edition' THEN d.value END) AS edition,
MAX(CASE WHEN d.field = 'family' THEN d.value END) AS family
FROM game_root g
JOIN copy c ON c.id = g.copy_id
LEFT JOIN detection d ON d.copy_id = g.copy_id AND d.root_path = g.root_path
GROUP BY g.copy_id, g.root_path
ORDER BY c.display_name, g.root_path
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
long copyId = rs.getLong("copy_id");
out.add(new RootFacts(copyId, rs.getString("display_name"),
rs.getString("root_path"), rs.getString("sha256"),
rs.getString("dll_sha1"), rs.getString("app_def_sha1"),
rs.getString("edition"), rs.getString("family"),
rootsPerCopy.getOrDefault(copyId, 1)));
}
}
return out;
}
/** Slug bez ogonków i znaków spoza [a-z0-9-], żeby był stabilnym kluczem tytułu. */
static String slug(String name) {
String ascii = Normalizer.normalize(name.replace('ł', 'l').replace('Ł', 'L'),
Normalizer.Form.NFD)
.replaceAll("\\p{M}", "");
return ascii.toLowerCase(Locale.ROOT)
.replaceAll("[^a-z0-9]+", "-")
.replaceAll("^-+|-+$", "");
}
private static String nvl(String value) {
return value == null ? "" : value;
}
}
@@ -0,0 +1,178 @@
package pl.genschu.rexcatalog.catalog;
import pl.genschu.rexcatalog.db.Database;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.text.Normalizer;
import java.util.Locale;
import java.util.Map;
/**
* Przypina katalog do kolekcji leżącej w innym miejscu.
*
* <p>{@code copy.path} zapisuje, gdzie obraz płyty był w chwili indeksowania —
* u autora {@code /Users/.../Reksio/...}, w kontenerze {@code /media/...},
* a na innej maszynie jeszcze inaczej. Baza jest przenośna, ale bez przestawienia
* tych ścieżek nic z niej nie odczytamy: podgląd dźwięku, renderowanie klatek
* i transkrypcja czytają wprost z obrazów.
*
* <p>Dopasowanie idzie po nazwie pliku i jest sprawdzane rozmiarem — a na żądanie
* także pełnym odciskiem. Nazwa sama w sobie niczego nie dowodzi, więc kopia,
* której rozmiar się nie zgadza, zostaje nietknięta i jest zgłaszana.
*
* <p>Nazwy porównujemy po normalizacji Unicode. macOS trzyma je rozłożone (NFD:
* {@code n} plus znak łączący), Windows i Linux złożone (NFC), więc
* {@code Wojna Trojańska.iso} z jednego systemu nie jest równe temu samemu
* napisowi z drugiego, choć wygląda identycznie. To właśnie przy przenoszeniu
* między maszynami boli najbardziej.
*/
public final class Relocator {
private final Database db;
public Relocator(Database db) {
this.db = db;
}
/**
* @param status {@code przypięta}, {@code bez zmian}, {@code nie znaleziono}
* albo {@code rozjazd}
*/
public record Result(String displayName, String oldPath, String newPath, String status,
String note) {
}
private record Copy(long id, String displayName, String path, long size, String sha256) {
}
public List<Result> run(Path root, boolean verify, boolean dryRun) throws Exception {
if (!Files.isDirectory(root)) {
throw new IllegalArgumentException("to nie jest katalog: " + root);
}
Map<String, List<Path>> byName = index(root);
List<Result> results = new ArrayList<>();
try (PreparedStatement update = db.connection().prepareStatement(
"UPDATE copy SET path = ? WHERE id = ?")) {
for (Copy copy : copies()) {
Path current = Path.of(copy.path());
String name = current.getFileName().toString();
if (Files.exists(current)) {
results.add(new Result(copy.displayName(), copy.path(), copy.path(),
"bez zmian", "jest tam, gdzie było"));
continue;
}
List<Path> candidates = byName.getOrDefault(key(name), List.of());
Path match = pick(candidates, copy, verify);
if (match == null) {
results.add(new Result(copy.displayName(), copy.path(), null,
candidates.isEmpty() ? "nie znaleziono" : "rozjazd",
candidates.isEmpty()
? "nie ma pliku o tej nazwie w podanym katalogu"
: "znalazłem nazwę, ale nie zgadza się zawartość"));
continue;
}
String target = match.toAbsolutePath().toString();
if (!dryRun) {
update.setString(1, target);
update.setLong(2, copy.id());
update.executeUpdate();
}
results.add(new Result(copy.displayName(), copy.path(), target,
"przypięta", null));
}
}
return results;
}
/**
* Wybiera kandydata, którego zawartość zgadza się z zapisaną. Rozmiar odsiewa
* pomyłki od ręki; {@code verify} dokłada odcisk, co przy sześciu gigabajtach
* trwa, ale daje pewność.
*/
private Path pick(List<Path> candidates, Copy copy, boolean verify) throws Exception {
for (Path candidate : candidates) {
long size = Files.isDirectory(candidate) ? directorySize(candidate)
: Files.size(candidate);
if (copy.size() > 0 && size != copy.size()) {
continue;
}
if (verify && copy.sha256() != null && !Files.isDirectory(candidate)
&& !copy.sha256().equalsIgnoreCase(sha256(candidate))) {
continue;
}
return candidate;
}
return null;
}
/** Zbiera nazwy plików i katalogów w nowym miejscu — jedno przejście, potem tylko odpytywanie. */
private static Map<String, List<Path>> index(Path root) throws IOException {
Map<String, List<Path>> out = new HashMap<>();
try (var stream = Files.walk(root, 3)) {
stream.forEach(path -> out.computeIfAbsent(key(path.getFileName().toString()),
k -> new ArrayList<>()).add(path));
}
return out;
}
/** Wspólna postać nazwy: złożona i bez wielkości liter. */
private static String key(String name) {
return Normalizer.normalize(name, Normalizer.Form.NFC).toLowerCase(Locale.ROOT);
}
private static long directorySize(Path directory) throws IOException {
try (var stream = Files.walk(directory)) {
return stream.filter(Files::isRegularFile).mapToLong(path -> {
try {
return Files.size(path);
} catch (IOException e) {
return 0;
}
}).sum();
}
}
private static String sha256(Path file) throws Exception {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] buffer = new byte[1 << 20];
try (InputStream in = Files.newInputStream(file)) {
int read;
while ((read = in.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
}
StringBuilder out = new StringBuilder();
for (byte b : digest.digest()) {
out.append(Character.forDigit((b >> 4) & 0xF, 16));
out.append(Character.forDigit(b & 0xF, 16));
}
return out.toString();
}
private List<Copy> copies() throws Exception {
List<Copy> out = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT id, display_name, path, size, sha256 FROM copy ORDER BY id");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
out.add(new Copy(rs.getLong("id"), rs.getString("display_name"),
rs.getString("path"), rs.getLong("size"), rs.getString("sha256")));
}
}
return out;
}
}
@@ -4,6 +4,8 @@ import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.sql.Connection; import java.sql.Connection;
import java.sql.DriverManager; import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException; import java.sql.SQLException;
import java.sql.Statement; import java.sql.Statement;
@@ -41,6 +43,7 @@ public final class Database implements AutoCloseable {
} }
private void applySchema() throws SQLException { private void applySchema() throws SQLException {
dropLegacyEdition();
try (Statement st = connection.createStatement()) { try (Statement st = connection.createStatement()) {
for (String ddl : SCHEMA) { for (String ddl : SCHEMA) {
st.executeUpdate(ddl); st.executeUpdate(ddl);
@@ -49,6 +52,52 @@ public final class Database implements AutoCloseable {
migrate(); migrate();
} }
/**
* Stara wersja {@code edition} miała {@code dll_sha1 UNIQUE}, co jest nie do
* pogodzenia z dwupackiem — dwie gry na jednej płycie dzielą bibliotekę silnika.
* Tabela jest przebudowywana tylko wtedy, gdy jest pusta; jeśli zdążyły w niej
* osiąść dane kuratorskie, wołamy o ręczną migrację zamiast je skasować.
*/
private void dropLegacyEdition() throws SQLException {
if (!tableExists("edition") || hasColumn("edition", "fingerprint")) {
return;
}
try (Statement st = connection.createStatement();
ResultSet rs = st.executeQuery("SELECT COUNT(*) FROM edition")) {
if (rs.next() && rs.getLong(1) > 0) {
throw new SQLException("Tabela edition ma stary układ i zawiera dane. "
+ "Przenieś je ręcznie albo usuń data/catalog.sqlite i przebuduj "
+ "(ingest, decode, analyze, promote).");
}
}
try (Statement st = connection.createStatement()) {
st.executeUpdate("DROP TABLE edition");
}
}
private boolean tableExists(String table) throws SQLException {
try (PreparedStatement st = connection.prepareStatement(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?")) {
st.setString(1, table);
try (ResultSet rs = st.executeQuery()) {
return rs.next();
}
}
}
/** {@code table_xinfo}, nie {@code table_info}: to drugie pomija kolumny wyliczane. */
private boolean hasColumn(String table, String column) throws SQLException {
try (Statement st = connection.createStatement();
ResultSet rs = st.executeQuery("PRAGMA table_xinfo(" + table + ")")) {
while (rs.next()) {
if (column.equalsIgnoreCase(rs.getString("name"))) {
return true;
}
}
return false;
}
}
/** /**
* Dokłada kolumny, których {@code CREATE TABLE IF NOT EXISTS} nie doda do już * Dokłada kolumny, których {@code CREATE TABLE IF NOT EXISTS} nie doda do już
* istniejącej tabeli. Tylko {@code ADD COLUMN} — bez przebudowy tabel, bo na * istniejącej tabeli. Tylko {@code ADD COLUMN} — bez przebudowy tabel, bo na
@@ -56,17 +105,43 @@ public final class Database implements AutoCloseable {
*/ */
private void migrate() throws SQLException { private void migrate() throws SQLException {
addColumnIfMissing("copy", "fs_type", "TEXT"); addColumnIfMissing("copy", "fs_type", "TEXT");
// media zaczęło od dźwięku, ale grafika opisuje się tymi samymi kategoriami:
// fakt o blobie, kluczowany odciskiem treści
addColumnIfMissing("media", "width", "INTEGER");
addColumnIfMissing("media", "height", "INTEGER");
addColumnIfMissing("media", "frames", "INTEGER");
addColumnIfMissing("media", "fps", "INTEGER");
addColumnIfMissing("media", "author", "TEXT");
addBasenameIndex();
}
/**
* Nazwa pliku bez katalogu, jako kolumna wyliczana i zaindeksowana.
*
* <p>Rozwiązywanie odwołań ze skryptów pyta o nią raz na odwołanie, a jeden
* skrypt potrafi ich mieć siedemset — bez indeksu każde z nich przeglądało
* wszystkie 38 tysięcy ścieżek. Kolumna jest wirtualna, więc nie zajmuje
* miejsca i nie wymaga uzupełniania przy zapisie.
*/
private void addBasenameIndex() throws SQLException {
if (!hasColumn("file", "basename")) {
try (Statement st = connection.createStatement()) {
st.executeUpdate("ALTER TABLE file ADD COLUMN basename TEXT "
+ "GENERATED ALWAYS AS "
+ "(replace(path, rtrim(path, replace(path, '/', '')), '')) VIRTUAL");
}
}
try (Statement st = connection.createStatement()) {
st.executeUpdate("CREATE INDEX IF NOT EXISTS idx_file_basename "
+ "ON file(copy_id, basename)");
}
} }
private void addColumnIfMissing(String table, String column, String type) throws SQLException { private void addColumnIfMissing(String table, String column, String type) throws SQLException {
if (hasColumn(table, column)) {
return;
}
try (Statement st = connection.createStatement()) { try (Statement st = connection.createStatement()) {
try (java.sql.ResultSet rs = st.executeQuery("PRAGMA table_info(" + table + ")")) {
while (rs.next()) {
if (column.equalsIgnoreCase(rs.getString("name"))) {
return;
}
}
}
st.executeUpdate("ALTER TABLE " + table + " ADD COLUMN " + column + " " + type); st.executeUpdate("ALTER TABLE " + table + " ADD COLUMN " + column + " " + type);
} }
} }
@@ -95,19 +170,27 @@ public final class Database implements AutoCloseable {
publisher TEXT DEFAULT 'Aidem Media' publisher TEXT DEFAULT 'Aidem Media'
) )
""", """,
// Wydanie jest tożsamością kuratorską: co wykryte, siedzi w `detection`
// z dowodem, a tutaj trzymamy tylko poprawki człowieka (*_override)
// i pola, których nie da się wykryć.
//
// fingerprint to klucz scalania kopii w jedno wydanie. Sam hash biblioteki
// nie wystarcza — Herkules i Odyseusz mają identyczny Piklib 8, a różnią się
// dopiero odciskiem application.def.
""" """
CREATE TABLE IF NOT EXISTS edition ( CREATE TABLE IF NOT EXISTS edition (
id INTEGER PRIMARY KEY, id INTEGER PRIMARY KEY,
title_id INTEGER REFERENCES title(id), title_id INTEGER REFERENCES title(id),
label TEXT, fingerprint TEXT NOT NULL UNIQUE,
dll_sha1 TEXT UNIQUE, label TEXT,
engine_detected TEXT, dll_sha1 TEXT,
engine_override TEXT, app_def_sha1 TEXT,
compiler_detected TEXT, engine_override TEXT,
compiler_override TEXT, engine_version_override TEXT,
release_date TEXT, compiler_override TEXT,
distributor TEXT, release_date_override TEXT,
notes TEXT distributor TEXT,
notes TEXT
) )
""", """,
""" """
@@ -216,6 +299,121 @@ public final class Database implements AutoCloseable {
created_at TEXT, created_at TEXT,
PRIMARY KEY (blob_sha1, kind) PRIMARY KEY (blob_sha1, kind)
) )
""",
// Indeks pełnotekstowy odszyfrowanych skryptów. Kluczem jest blob, nie plik —
// ten sam skrypt bywa w kilku obrazach, a wtedy szukamy raz i pokazujemy,
// w których kopiach występuje.
//
// tokenchars '_' trzyma identyfikatory w całości (SHOW_CURSOR to jeden token,
// nie dwa), a remove_diacritics pozwala znaleźć polski tekst bez ogonków.
"""
CREATE VIRTUAL TABLE IF NOT EXISTS script_fts USING fts5(
body,
blob_sha1 UNINDEXED,
tokenize = "unicode61 remove_diacritics 2 tokenchars '_'"
)
""",
// Zawartość archiwów, które trzymają wiele zasobów w jednym pliku —
// wavs/wav.snd z Kapitana Nemo to 3274 kwestie w Ogg Vorbisie. Bez tego
// cała warstwa głosowa jednej gry jest w katalogu niewidoczna.
//
// Payloadów nie wypakowujemy: offset i length wystarczą, żeby je odczytać
// z obrazu płyty na żądanie. sha1 liczymy przy indeksowaniu, bo daje
// wspólny identyfikator dla faktów i transkryptów.
"""
CREATE TABLE IF NOT EXISTS archive_entry (
container_sha1 TEXT NOT NULL REFERENCES blob(sha1),
name TEXT NOT NULL,
name_raw TEXT,
offset INTEGER NOT NULL,
size INTEGER NOT NULL,
sha1 TEXT,
format TEXT,
tool_version TEXT NOT NULL,
PRIMARY KEY (container_sha1, name)
)
""",
"CREATE INDEX IF NOT EXISTS idx_archive_sha1 ON archive_entry(sha1)",
// Fakty odczytane z nagłówka zasobu. Bez klucza obcego do blob, bo sha1
// bywa też odciskiem wpisu w archiwum, którego w blob nie ma — obie
// przestrzenie są adresowane treścią, więc się nie pomieszają.
"""
CREATE TABLE IF NOT EXISTS media (
sha1 TEXT PRIMARY KEY,
kind TEXT NOT NULL,
codec TEXT,
duration_ms INTEGER,
sample_rate INTEGER,
channels INTEGER,
bits INTEGER,
note TEXT,
tool_version TEXT NOT NULL,
probed_at TEXT
)
""",
// Odwołania skryptu do zasobów: OBIEKT:FILENAME=PLIK.WAV. Zapisujemy je
// niezależnie od tego, czy plik da się znaleźć — nierozwiązane odwołanie
// jest samo w sobie informacją o wydaniu.
"""
CREATE TABLE IF NOT EXISTS script_ref (
script_sha1 TEXT NOT NULL REFERENCES blob(sha1),
object TEXT NOT NULL,
field TEXT NOT NULL,
value TEXT NOT NULL,
name TEXT NOT NULL,
format TEXT,
line INTEGER,
PRIMARY KEY (script_sha1, object, field, value)
)
""",
"CREATE INDEX IF NOT EXISTS idx_script_ref_name ON script_ref(name)",
// dialogi.dta: trigger | identyfikator kwestii | rozmówca | flagi.
// Daje każdej kwestii mówcę i kontekst zdarzenia bez transkrypcji.
"""
CREATE TABLE IF NOT EXISTS dialogue_line (
dta_sha1 TEXT NOT NULL REFERENCES blob(sha1),
line INTEGER NOT NULL,
section TEXT,
trigger TEXT,
audio_name TEXT,
speaker TEXT,
extra TEXT,
PRIMARY KEY (dta_sha1, line)
)
""",
"CREATE INDEX IF NOT EXISTS idx_dialogue_audio ON dialogue_line(audio_name)",
// Transkrypty są WYGENEROWANE, nie odczytane z płyty — dlatego osobna
// tabela z nazwą modelu i narzędzia. Nigdy nie mieszają się z metadanymi
// wydania ani z faktami detektorów.
"""
CREATE TABLE IF NOT EXISTS transcript (
sha1 TEXT PRIMARY KEY,
text TEXT NOT NULL,
lang TEXT,
model TEXT NOT NULL,
tool TEXT NOT NULL,
duration_ms INTEGER,
created_at TEXT
)
""",
"""
CREATE VIRTUAL TABLE IF NOT EXISTS transcript_fts USING fts5(
body,
sha1 UNINDEXED,
tokenize = "unicode61 remove_diacritics 2 tokenchars '_'"
)
""",
// Nazwane sekwencje w animacji: BEZRUCH, GADA_START, GADA_1...
// Mówią, co postać potrafi zrobić, więc są treścią, a nie tylko liczbą.
"""
CREATE TABLE IF NOT EXISTS anim_event (
ann_sha1 TEXT NOT NULL,
name TEXT NOT NULL,
frames_count INTEGER,
loop_start INTEGER,
loop_end INTEGER,
PRIMARY KEY (ann_sha1, name)
)
""" """
}; };
} }
@@ -0,0 +1,392 @@
package pl.genschu.rexcatalog.ingest;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.bloomooemulator.utils.LangCodeConverter;
import pl.genschu.rexcatalog.db.Database;
import java.io.File;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Ustala wersje językowe wydania na podstawie {@code install.ini} i układu katalogu
* {@code wavs/}.
*
* <p>Sekcja {@code [Language]} instalatora wymienia identyfikatory LCID Windows:
* <pre>
* 0415 = Polish
* 040e = Hungarian
* 0405 = Czech
* 041b = Slovak
* </pre>
* Tłumaczenie LCID na kod silnika bierzemy z {@link LangCodeConverter} w {@code :core},
* żeby katalog i emulator nie rozjechały się we własnych tabelach. To ta klasa
* rozstrzyga, że {@code slo} oznacza słowacki, a nie słoweński.
*
* <p>Dubbing widać dodatkowo w drzewie plików: <i>Wojna Trojańska</i> ma
* {@code wavs/cze/}, {@code wavs/hun/} i {@code wavs/slo/} po 543 pliki każdy,
* a polski leży wprost w {@code wavs/}. Język bez własnego podkatalogu, a wymieniony
* w instalatorze, jest więc językiem podstawowym płyty.
*/
public final class LanguageDetector {
private static final String DETECTOR = "LanguageDetector";
private static final Charset CP1250 = Charset.forName("windows-1250");
/** Wiersze {@code [Language]}: klucz to czterocyfrowy LCID, reszta to numery porządkowe. */
private static final Pattern LCID_LINE =
Pattern.compile("^\\s*([0-9A-Fa-f]{4})\\s*=\\s*(.+?)\\s*$");
private static final Pattern SECTION = Pattern.compile("^\\s*\\[(.+?)]\\s*$");
/**
* LCID → ISO 639-1, dla języków, które silnik faktycznie obsługuje.
*
* <p>Kody dwuliterowe, bo takie przyjmuje {@code lang add}. Kluczem jest LCID,
* a nie kod silnika, bo ten drugi bywa spolszczony ({@code niem} to niemiecki).
* Zgodność z {@code :core} pilnuje {@link #verifyAgainstCore()}.
*/
private static final Map<String, String> LCID_TO_ISO = Map.of(
"0402", "bg",
"0405", "cs",
"0407", "de",
"040E", "hu",
"0415", "pl",
"0418", "ro",
"0419", "ru",
"041B", "sk");
private final Database db;
private final String coreVersion;
public LanguageDetector(Database db, String coreVersion) {
this.db = db;
this.coreVersion = coreVersion;
}
public record RootResult(String copyName, String rootPath, String note,
List<String> languages, List<String> audioLanguages) {
}
private record Target(long copyId, String copyName, String container, String rootPath,
String filePath) {
}
/**
* Sprawdza, czy nasza tabela LCID → ISO opisuje te same języki co {@code :core}.
* Rozjazd oznacza, że emulator zmienił mapowanie i katalog też musi.
*/
public static List<String> verifyAgainstCore() {
List<String> problems = new ArrayList<>();
for (String lcid : LCID_TO_ISO.keySet()) {
String engine = LangCodeConverter.lcidToIsoCode(lcid);
if ("unknown".equals(engine)) {
problems.add(lcid + ": :core nie zna już tego LCID");
}
}
return problems;
}
public List<RootResult> run() throws Exception {
List<Target> targets = targets();
List<RootResult> results = new ArrayList<>();
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
try (PreparedStatement delete = conn.prepareStatement(
"DELETE FROM detection WHERE copy_id = ? AND root_path = ? AND detector = ?");
PreparedStatement insert = conn.prepareStatement("""
INSERT INTO detection
(copy_id, root_path, field, value, confidence, evidence,
detector, core_version, detected_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""")) {
String now = Instant.now().toString();
Map<String, IFileSystem> open = new LinkedHashMap<>();
for (Target target : targets) {
delete.setLong(1, target.copyId());
delete.setString(2, target.rootPath());
delete.setString(3, DETECTOR);
delete.executeUpdate();
IFileSystem fs;
try {
fs = open.computeIfAbsent(target.container(), path -> {
try {
return AssetSourceDispatcher.openAssets(new File(path));
} catch (Exception e) {
throw new IllegalStateException(e.getMessage(), e);
}
});
} catch (RuntimeException e) {
results.add(new RootResult(target.copyName(), target.rootPath(),
"nie otwieram obrazu: " + e.getMessage(), List.of(), List.of()));
continue;
}
byte[] raw;
try (InputStream in = fs.open(target.filePath())) {
raw = in.readAllBytes();
} catch (Exception e) {
results.add(new RootResult(target.copyName(), target.rootPath(),
"nie czytam install.ini: " + e.getMessage(), List.of(), List.of()));
continue;
}
Map<String, String> declared = parseLanguages(new String(raw, CP1250));
if (declared.isEmpty()) {
results.add(new RootResult(target.copyName(), target.rootPath(),
"install.ini bez sekcji [Language]", List.of(), List.of()));
continue;
}
Set<String> audioDirs = audioDirectories(target.copyId(), target.rootPath());
List<String> languages = new ArrayList<>();
List<String> audio = new ArrayList<>();
List<String> withoutDirectory = new ArrayList<>();
StringBuilder evidence = new StringBuilder("install.ini [Language]:");
for (Map.Entry<String, String> entry : declared.entrySet()) {
String lcid = entry.getKey().toUpperCase(Locale.ROOT);
String iso = LCID_TO_ISO.get(lcid);
String engine = LangCodeConverter.lcidToIsoCode(lcid).toLowerCase(Locale.ROOT);
String code = iso != null ? iso : lcid;
languages.add(code);
evidence.append(' ').append(entry.getValue())
.append(" (").append(lcid).append(')');
if (audioDirs.contains(engine)) {
audio.add(code);
} else {
withoutDirectory.add(code);
}
}
// język bez własnego podkatalogu ma dubbing wprost w wavs/ — ale tylko
// wtedy, gdy jest jeden; przy dwóch nie wiadomo, który to
if (withoutDirectory.size() == 1) {
audio.add(0, withoutDirectory.get(0));
}
write(insert, target, "languages", String.join(",", languages), 0.95,
evidence.toString(), now);
if (!audio.isEmpty()) {
write(insert, target, "audio_languages", String.join(",", audio), 0.9,
audioEvidence(audioDirs, withoutDirectory), now);
}
results.add(new RootResult(target.copyName(), target.rootPath(), null,
languages, audio));
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return results;
}
private static String audioEvidence(Set<String> dirs, List<String> withoutDirectory) {
StringBuilder out = new StringBuilder();
if (!dirs.isEmpty()) {
out.append("podkatalogi wavs/: ").append(String.join(", ", dirs));
}
if (withoutDirectory.size() == 1) {
if (!out.isEmpty()) {
out.append("; ");
}
out.append("bez podkatalogu, dźwięk w wavs/: ").append(withoutDirectory.get(0));
}
return out.toString();
}
private void write(PreparedStatement insert, Target target, String field, String value,
double confidence, String evidence, String now) throws Exception {
insert.setLong(1, target.copyId());
insert.setString(2, target.rootPath());
insert.setString(3, field);
insert.setString(4, value);
insert.setDouble(5, confidence);
insert.setString(6, evidence);
insert.setString(7, DETECTOR);
insert.setString(8, coreVersion);
insert.setString(9, now);
insert.executeUpdate();
}
/** Czyta sekcję {@code [Language]}, pomijając numery porządkowe i pole {@code Nr}. */
static Map<String, String> parseLanguages(String text) {
Map<String, String> out = new LinkedHashMap<>();
boolean inside = false;
for (String line : text.split("\r?\n")) {
Matcher section = SECTION.matcher(line);
if (section.matches()) {
inside = section.group(1).equalsIgnoreCase("Language");
continue;
}
if (!inside) {
continue;
}
Matcher entry = LCID_LINE.matcher(line);
if (entry.matches()) {
out.put(entry.group(1), entry.group(2));
}
}
return out;
}
/** Podkatalogi {@code wavs/} w obrębie korzenia gry — nazwane kodem silnika. */
private Set<String> audioDirectories(long copyId, String rootPath) throws Exception {
String prefix = (rootPath.isEmpty() ? "" : rootPath + "/") + "wavs/";
Set<String> out = new LinkedHashSet<>();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT DISTINCT path FROM file WHERE copy_id = ? AND path LIKE ?")) {
st.setLong(1, copyId);
st.setString(2, prefix + "%/%");
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String rest = rs.getString("path").substring(prefix.length());
int slash = rest.indexOf('/');
if (slash > 0) {
out.add(rest.substring(0, slash));
}
}
}
}
// sfx to efekty dźwiękowe, nie wersja językowa
out.remove("sfx");
return out;
}
/** {@code install.ini} pasujący do korzenia gry — najpłytszy w obrębie prefiksu. */
/**
* Dopasowuje {@code install.ini} do korzenia gry.
*
* <p>Zwykle plik leży w obrębie korzenia, ale na dwupacku instalator gry
* głównej stoi w korzeniu płyty, a druga gra ma własny w swoim podkatalogu
* ({@code odyseusz/install.ini}). Dlatego korzeń, który nie ma własnego pliku,
* sięga po ten z płyty — o ile nie należy on do żadnego innego korzenia.
*/
private List<Target> targets() throws Exception {
Map<Long, Copy> copies = copies();
List<Candidate> candidates = candidates(copies.keySet());
List<Target> out = new ArrayList<>();
for (Copy copy : copies.values()) {
for (String rootPath : copy.roots()) {
String prefix = rootPath.isEmpty() ? "" : rootPath + "/";
Candidate best = shallowest(candidates, copy.id(),
canonical -> canonical.startsWith(prefix));
if (best == null) {
// instalator gry głównej dwupacka: w korzeniu płyty, nieprzypisany
best = shallowest(candidates, copy.id(), canonical -> copy.roots().stream()
.noneMatch(root -> !root.isEmpty()
&& canonical.startsWith(root + "/")));
}
if (best != null) {
out.add(new Target(copy.id(), copy.displayName(), copy.container(),
rootPath, best.filePath()));
}
}
}
return out;
}
private record Copy(long id, String displayName, String container, List<String> roots) {
}
private record Candidate(long copyId, String canonical, String filePath) {
}
private static Candidate shallowest(List<Candidate> candidates, long copyId,
java.util.function.Predicate<String> accepts) {
Candidate best = null;
for (Candidate candidate : candidates) {
if (candidate.copyId() != copyId || !accepts.test(candidate.canonical())) {
continue;
}
if (best == null || depth(candidate.canonical()) < depth(best.canonical())) {
best = candidate;
}
}
return best;
}
private Map<Long, Copy> copies() throws Exception {
Map<Long, Copy> out = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT DISTINCT g.copy_id, g.root_path, c.display_name, c.path AS container
FROM game_root g
JOIN copy c ON c.id = g.copy_id
WHERE c.status <> 'unreadable'
ORDER BY g.copy_id, g.root_path
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
long id = rs.getLong("copy_id");
out.computeIfAbsent(id, key -> new Copy(key, rsString(rs, "display_name"),
rsString(rs, "container"), new ArrayList<>()))
.roots().add(rs.getString("root_path"));
}
}
return out;
}
private List<Candidate> candidates(Set<Long> knownCopies) throws Exception {
List<Candidate> out = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT f.copy_id, f.path AS canonical, COALESCE(f.path_raw, f.path) AS file_path
FROM file f
WHERE f.path LIKE '%install.ini'
ORDER BY f.copy_id, f.path
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
long copyId = rs.getLong("copy_id");
if (knownCopies.contains(copyId)) {
out.add(new Candidate(copyId, rs.getString("canonical"),
rs.getString("file_path")));
}
}
}
return out;
}
/** {@code computeIfAbsent} nie przepuszcza wyjątku SQL, a kolumny są tu zawsze obecne. */
private static String rsString(ResultSet rs, String column) {
try {
return rs.getString(column);
} catch (java.sql.SQLException e) {
throw new IllegalStateException(e);
}
}
private static int depth(String path) {
return (int) path.chars().filter(c -> c == '/').count();
}
}
@@ -0,0 +1,232 @@
package pl.genschu.rexcatalog.ingest;
import pl.genschu.rexcatalog.db.Database;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Wyciąga metadane wydania z {@code dane/application.def} — pliku opisu projektu,
* który Piklib i BlooMoo trzymają razem z grą.
*
* <p>Źródło jest lepsze niż tablice hashy, bo pochodzi z samych danych gry, a nie
* z listy rozpoznanych bibliotek. Działa na odszyfrowanym cache'u, więc wymaga
* wcześniejszego {@code decode}.
*/
public final class MetadataDetector {
private static final String DETECTOR = "MetadataDetector";
/** Obiekt typu APPLICATION nie zawsze nazywa się GAME — bywa UFO, PIRACI. */
private static final Pattern APPLICATION =
Pattern.compile("^(\\w+):TYPE=APPLICATION\\s*$", Pattern.MULTILINE);
private static final Pattern ISO_DATE = Pattern.compile("^(\\d{4}-\\d{2}-\\d{2})");
private final Database db;
private final Path dataDir;
private final String coreVersion;
public MetadataDetector(Database db, Path dataDir, String coreVersion) {
this.db = db;
this.dataDir = dataDir;
this.coreVersion = coreVersion;
}
public record Fact(String field, String value, double confidence, String evidence) {
}
public record RootResult(String copyName, String rootPath, String note, List<Fact> facts) {
}
private record Target(long copyId, String copyName, String rootPath,
String filePath, String blobSha1, String artifactPath) {
}
public List<RootResult> run() throws Exception {
List<Target> targets = targets();
List<RootResult> results = new ArrayList<>();
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
try (PreparedStatement delete = conn.prepareStatement(
"DELETE FROM detection WHERE copy_id = ? AND root_path = ? AND detector = ?");
PreparedStatement insert = conn.prepareStatement("""
INSERT INTO detection
(copy_id, root_path, field, value, confidence, evidence,
detector, core_version, detected_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""")) {
String now = Instant.now().toString();
for (Target target : targets) {
delete.setLong(1, target.copyId());
delete.setString(2, target.rootPath());
delete.setString(3, DETECTOR);
delete.executeUpdate();
if (target.artifactPath() == null) {
results.add(new RootResult(target.copyName(), target.rootPath(),
"brak application.def w cache'u — uruchom decode", List.of()));
continue;
}
String body = Files.readString(dataDir.resolve(target.artifactPath()),
StandardCharsets.UTF_8);
List<Fact> facts = parse(body, target.filePath(), target.blobSha1());
if (facts.isEmpty()) {
results.add(new RootResult(target.copyName(), target.rootPath(),
"nie znaleziono obiektu TYPE=APPLICATION", List.of()));
continue;
}
for (Fact fact : facts) {
insert.setLong(1, target.copyId());
insert.setString(2, target.rootPath());
insert.setString(3, fact.field());
insert.setString(4, fact.value());
insert.setDouble(5, fact.confidence());
insert.setString(6, fact.evidence());
insert.setString(7, DETECTOR);
insert.setString(8, coreVersion);
insert.setString(9, now);
insert.executeUpdate();
}
results.add(new RootResult(target.copyName(), target.rootPath(), null, facts));
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return results;
}
private List<Fact> parse(String body, String filePath, String blobSha1) {
Matcher application = APPLICATION.matcher(body);
if (!application.find()) {
return List.of();
}
String object = application.group(1);
List<Fact> facts = new ArrayList<>();
// odcisk pliku opisu — w dwupacku to jedyne, co odróżnia dwa korzenie,
// bo bibliotekę Piklib obie gry mają bit w bit tę samą
facts.add(new Fact("app_def_sha1", blobSha1, 1.0, filePath));
facts.add(new Fact("app_object", object, 1.0, filePath + ": OBJECT=" + object));
field(body, object, "LASTMODIFYTIME").ifPresent(raw -> {
Matcher iso = ISO_DATE.matcher(raw);
// data ostatniej modyfikacji projektu przybliża datę builda, nie wydania —
// stąd pewność poniżej jedynki nawet przy czytelnym formacie
facts.add(iso.find()
? new Fact("release_date", iso.group(1), 0.7,
evidence(filePath, object, "LASTMODIFYTIME", raw))
: new Fact("release_date", raw, 0.2,
evidence(filePath, object, "LASTMODIFYTIME", raw)
+ " (nierozpoznany format daty)"));
});
field(body, object, "CREATIONTIME").ifPresent(raw ->
// wspólna dla całej serii — to data założenia projektu, nie wydania
facts.add(new Fact("project_created", raw, 1.0,
evidence(filePath, object, "CREATIONTIME", raw))));
field(body, object, "VERSION").ifPresent(raw ->
facts.add(new Fact("game_version", raw, 1.0,
evidence(filePath, object, "VERSION", raw))));
field(body, object, "BLOOMOO_VERSION").ifPresent(raw ->
facts.add(new Fact("engine_version", raw, 0.95,
evidence(filePath, object, "BLOOMOO_VERSION", raw))));
field(body, object, "EPISODES").ifPresent(raw ->
facts.add(new Fact("episodes", raw, 1.0,
evidence(filePath, object, "EPISODES", raw))));
field(body, object, "AUTHOR")
.map(MetadataDetector::unquote)
.filter(value -> !value.isBlank())
.ifPresent(value -> facts.add(new Fact("author", value, 1.0,
evidence(filePath, object, "AUTHOR", value))));
return facts;
}
private static java.util.Optional<String> field(String body, String object, String key) {
Matcher m = Pattern.compile("^" + Pattern.quote(object) + ":" + key + "=(.*)$",
Pattern.MULTILINE).matcher(body);
if (!m.find()) {
return java.util.Optional.empty();
}
String value = m.group(1).trim();
return value.isEmpty() ? java.util.Optional.empty() : java.util.Optional.of(value);
}
/** Skrypty bywają cytowane zarówno ASCII, jak i typograficznymi cudzysłowami. */
private static String unquote(String value) {
return value.replaceAll("^[\"”„“]+|[\"”„“]+$", "").trim();
}
private static String evidence(String filePath, String object, String key, String value) {
return filePath + ": " + object + ":" + key + "=" + value;
}
/**
* Dopasowuje application.def do korzenia gry. W dwupacku każdy korzeń ma własny,
* więc wybieramy plik leżący najpłycej pod danym prefiksem.
*/
private List<Target> targets() throws Exception {
Map<String, Target> best = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT g.copy_id, c.display_name, g.root_path,
COALESCE(f.path_raw, f.path) AS file_path, f.path AS canonical,
f.blob_sha1, a.path AS artifact_path
FROM game_root g
JOIN copy c ON c.id = g.copy_id
JOIN file f ON f.copy_id = g.copy_id
LEFT JOIN artifact a ON a.blob_sha1 = f.blob_sha1 AND a.kind = 'script'
WHERE f.path LIKE '%application.def'
ORDER BY g.copy_id, g.root_path
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String rootPath = rs.getString("root_path");
String canonical = rs.getString("canonical");
String prefix = rootPath.isEmpty() ? "" : rootPath + "/";
if (!canonical.startsWith(prefix)) {
continue;
}
String key = rs.getLong("copy_id") + "" + rootPath;
Target candidate = new Target(rs.getLong("copy_id"), rs.getString("display_name"),
rootPath, rs.getString("file_path"), rs.getString("blob_sha1"),
rs.getString("artifact_path"));
Target current = best.get(key);
if (current == null || depth(canonical) < depth(current.filePath())) {
best.put(key, candidate);
}
}
}
return new ArrayList<>(best.values());
}
private static int depth(String path) {
return (int) path.chars().filter(c -> c == '/').count();
}
}
@@ -0,0 +1,319 @@
package pl.genschu.rexcatalog.mcp;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import pl.genschu.rexcatalog.db.Database;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.Executors;
/**
* Serwer MCP w transporcie Streamable HTTP, na {@code com.sun.net.httpserver} z JDK.
*
* <p>Sesji nie prowadzimy: wszystkie narzędzia są bezstanowe i tylko do odczytu,
* a specyfikacja pozwala pominąć {@code Mcp-Session-Id}. Z tego samego powodu
* {@code GET} na endpoint zwraca 405 — nie mamy komunikatów inicjowanych przez serwer,
* więc nie ma po co otwierać strumienia SSE.
*/
public final class McpServer {
/** Wersje protokołu, które umiemy obsłużyć; pierwsza jest domyślna. */
private static final List<String> SUPPORTED_PROTOCOLS =
List.of("2025-06-18", "2025-03-26", "2024-11-05");
private static final String SERVER_NAME = "rex-catalog";
private static final String SERVER_VERSION = "0.1.0";
private final Tools tools;
private final Map<String, Tools.Tool> byName = new LinkedHashMap<>();
private final Object lock = new Object();
private HttpServer http;
public McpServer(Database db, Path dataDir) {
this.tools = new Tools(db, dataDir);
for (Tools.Tool tool : tools.tools()) {
byName.put(tool.name(), tool);
}
}
public void start(String host, int port) throws IOException {
http = HttpServer.create(new InetSocketAddress(host, port), 0);
register(http);
// pula wątków, ale dostęp do bazy i tak serializujemy — patrz toolsCall()
http.setExecutor(Executors.newFixedThreadPool(4));
http.start();
}
/** Podpina endpoint MCP do cudzego serwera — front i MCP dzielą jeden port. */
public void register(HttpServer server) {
server.createContext("/mcp", this::handle);
}
/**
* Blokada serializująca dostęp do bazy. Front musi używać tej samej, bo obie
* warstwy siedzą na jednym połączeniu sqlite-jdbc.
*/
public Object lock() {
return lock;
}
public void stop() {
if (http != null) {
http.stop(0);
}
}
private void handle(HttpExchange exchange) throws IOException {
try {
if (!isOriginAllowed(exchange)) {
// ochrona przed DNS rebinding — wymagana przez specyfikację transportu
respond(exchange, 403, "text/plain", "Niedozwolony nagłówek Origin");
return;
}
switch (exchange.getRequestMethod()) {
case "POST" -> handlePost(exchange);
case "GET" -> respond(exchange, 405, "text/plain",
"Ten serwer nie wysyła komunikatów z własnej inicjatywy");
case "DELETE" -> respond(exchange, 405, "text/plain", "Brak sesji do zamknięcia");
default -> respond(exchange, 405, "text/plain", "Nieobsługiwana metoda");
}
} catch (Exception e) {
respond(exchange, 500, "text/plain", "Błąd serwera: " + e);
} finally {
exchange.close();
}
}
private void handlePost(HttpExchange exchange) throws IOException {
String body;
try (InputStream in = exchange.getRequestBody()) {
body = new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
JsonElement parsed;
try {
parsed = JsonParser.parseString(body);
} catch (Exception e) {
respond(exchange, 400, "application/json",
error(null, -32700, "Nieparsowalny JSON").toString());
return;
}
// wsadu nie rozdzielamy na osobne odpowiedzi w strumieniu — zwracamy tablicę
if (parsed.isJsonArray()) {
JsonArray responses = new JsonArray();
for (JsonElement element : parsed.getAsJsonArray()) {
JsonObject response = dispatch(element.getAsJsonObject());
if (response != null) {
responses.add(response);
}
}
if (responses.isEmpty()) {
respond(exchange, 202, "text/plain", "");
} else {
respond(exchange, 200, "application/json", responses.toString());
}
return;
}
JsonObject response = dispatch(parsed.getAsJsonObject());
if (response == null) {
// powiadomienie: brak treści odpowiedzi
respond(exchange, 202, "text/plain", "");
} else {
respond(exchange, 200, "application/json", response.toString());
}
}
/** Zwraca odpowiedź albo {@code null}, gdy przyszło powiadomienie. */
private JsonObject dispatch(JsonObject request) {
JsonElement id = request.get("id");
String method = request.has("method") ? request.get("method").getAsString() : "";
boolean notification = id == null || id.isJsonNull();
try {
JsonObject result = switch (method) {
case "initialize" -> initialize(request);
case "tools/list" -> toolsList();
case "tools/call" -> toolsCall(request);
case "ping" -> new JsonObject();
default -> null;
};
if (notification) {
return null;
}
if (result == null) {
return error(id, -32601, "Nieznana metoda: " + method);
}
JsonObject response = new JsonObject();
response.addProperty("jsonrpc", "2.0");
response.add("id", id);
response.add("result", result);
return response;
} catch (Exception e) {
if (notification) {
return null;
}
String message = e.getMessage() == null ? e.toString() : e.getMessage();
return error(id, -32603, message);
}
}
private JsonObject initialize(JsonObject request) {
String requested = null;
if (request.has("params") && request.getAsJsonObject("params").has("protocolVersion")) {
requested = request.getAsJsonObject("params").get("protocolVersion").getAsString();
}
// odbijamy wersję klienta, jeśli ją znamy — inaczej proponujemy własną najnowszą
String agreed = SUPPORTED_PROTOCOLS.contains(requested)
? requested : SUPPORTED_PROTOCOLS.get(0);
JsonObject result = new JsonObject();
result.addProperty("protocolVersion", agreed);
JsonObject capabilities = new JsonObject();
capabilities.add("tools", new JsonObject());
result.add("capabilities", capabilities);
JsonObject info = new JsonObject();
info.addProperty("name", SERVER_NAME);
info.addProperty("version", SERVER_VERSION);
result.add("serverInfo", info);
result.addProperty("instructions",
"Katalog kolekcji gier Aidem Media (seria Reksio, Poznaj Mity): obrazy płyt, "
+ "sumy kontrolne, metadane wydań i odszyfrowane skrypty. "
+ "Zacznij od list_titles, żeby poznać zawartość; "
+ "search_scripts przeszukuje treść skryptów gier.");
return result;
}
private JsonObject toolsList() {
JsonArray array = new JsonArray();
for (Tools.Tool tool : byName.values()) {
JsonObject entry = new JsonObject();
entry.addProperty("name", tool.name());
entry.addProperty("description", tool.description());
entry.add("inputSchema", tool.inputSchema());
array.add(entry);
}
JsonObject result = new JsonObject();
result.add("tools", array);
return result;
}
private JsonObject toolsCall(JsonObject request) {
JsonObject params = request.has("params")
? request.getAsJsonObject("params") : new JsonObject();
String name = params.has("name") ? params.get("name").getAsString() : "";
JsonObject arguments = params.has("arguments") && params.get("arguments").isJsonObject()
? params.getAsJsonObject("arguments") : new JsonObject();
Tools.Tool tool = byName.get(name);
if (tool == null) {
return toolResult("Nie ma narzędzia o nazwie " + name
+ ". Dostępne: " + String.join(", ", byName.keySet()), true);
}
// wymagane argumenty sprawdzamy ze schematu, żeby brak parametru był błędem
// narzędzia, a nie cichym wynikiem policzonym z wartości domyślnej
List<String> missing = new ArrayList<>();
for (JsonElement required : tool.inputSchema().getAsJsonArray("required")) {
String key = required.getAsString();
if (!arguments.has(key) || arguments.get(key).isJsonNull()) {
missing.add(key);
}
}
if (!missing.isEmpty()) {
return toolResult("Brakuje wymaganych argumentów narzędzia " + name + ": "
+ String.join(", ", missing), true);
}
try {
String text;
// sqlite-jdbc dzieli jedno połączenie, więc wywołania serializujemy;
// obciążenie jest czysto odczytowe i pojedynczy klient go nie odczuje
synchronized (lock) {
text = tool.handler().apply(arguments);
}
return toolResult(text, false);
} catch (Exception e) {
String message = e.getMessage() == null ? e.toString() : e.getMessage();
return toolResult("Narzędzie " + name + " zawiodło: " + message, true);
}
}
private static JsonObject toolResult(String text, boolean isError) {
JsonObject content = new JsonObject();
content.addProperty("type", "text");
content.addProperty("text", text);
JsonArray array = new JsonArray();
array.add(content);
JsonObject result = new JsonObject();
result.add("content", array);
result.addProperty("isError", isError);
return result;
}
private static JsonObject error(JsonElement id, int code, String message) {
JsonObject error = new JsonObject();
error.addProperty("code", code);
error.addProperty("message", message);
JsonObject response = new JsonObject();
response.addProperty("jsonrpc", "2.0");
if (id == null) {
response.add("id", com.google.gson.JsonNull.INSTANCE);
} else {
response.add("id", id);
}
response.add("error", error);
return response;
}
/**
* Przeglądarka na dowolnej stronie może wysłać POST na localhost, więc żądanie
* z nagłówkiem Origin spoza pętli zwrotnej odrzucamy. Klienci MCP (procesy,
* nie strony) Origin zwykle nie wysyłają i takie żądania przechodzą.
*/
private static boolean isOriginAllowed(HttpExchange exchange) {
String origin = exchange.getRequestHeaders().getFirst("Origin");
if (origin == null || origin.isBlank()) {
return true;
}
try {
String host = URI.create(origin).getHost();
return host != null && Set.of("localhost", "127.0.0.1", "::1", "[::1]").contains(host);
} catch (Exception e) {
return false;
}
}
private static void respond(HttpExchange exchange, int status, String contentType, String body)
throws IOException {
byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
if (!body.isEmpty()) {
exchange.getResponseHeaders().set("Content-Type", contentType + "; charset=utf-8");
}
exchange.sendResponseHeaders(status, bytes.length == 0 ? -1 : bytes.length);
if (bytes.length > 0) {
exchange.getResponseBody().write(bytes);
}
}
}
@@ -0,0 +1,659 @@
package pl.genschu.rexcatalog.mcp;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import pl.genschu.rexcatalog.db.Database;
import pl.genschu.rexcatalog.script.ScriptSearch;
import java.nio.file.Path;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* Narzędzia MCP nad bazą katalogu. Wszystkie są tylko do odczytu — serwer ma
* odpowiadać na pytania o kolekcję, a nie ją modyfikować; indeksowanie i edycja
* zostają w CLI, gdzie widać, co się dzieje.
*
* <p>Odpowiedzi są tekstem sformatowanym dla modelu, nie JSON-em: agent ma je
* czytać, a nie parsować, a tekst zużywa mniej kontekstu niż to samo w obiektach.
*/
public final class Tools {
private final Database db;
private final ScriptSearch search;
public Tools(Database db, Path dataDir) {
this.db = db;
this.search = new ScriptSearch(db, dataDir);
}
public interface Handler {
String apply(JsonObject arguments) throws Exception;
}
public record Tool(String name, String description, JsonObject inputSchema, Handler handler) {
}
public List<Tool> tools() {
List<Tool> tools = new ArrayList<>();
tools.add(new Tool("list_titles",
"Lista tytułów w kolekcji wraz z wydaniami: silnik, data builda, "
+ "wersje językowe i liczba posiadanych kopii. Zacznij od tego, "
+ "żeby poznać zawartość kolekcji.",
schema(Map.of(), List.of()),
args -> listTitles()));
tools.add(new Tool("search_scripts",
"Przeszukuje pełnotekstowo odszyfrowane skrypty gier (CNV, DEF, CLASS). "
+ "Składnia FTS5; identyfikatory z podkreśleniem są jednym tokenem. "
+ "Zwraca ścieżkę, gry, w których plik występuje, fragment i SHA-1.",
schema(Map.of(
"query", property("string", "Zapytanie, np. CANVAS_OBSERVER albo TYPE=EPISODE"),
"limit", property("integer", "Maksymalna liczba trafień (domyślnie 20)")
), List.of("query")),
args -> searchScripts(string(args, "query"), integer(args, "limit", 20))));
tools.add(new Tool("get_script",
"Zwraca pełną odszyfrowaną treść skryptu. Wskaż go pełnym SHA-1 "
+ "albo fragmentem ścieżki; przy niejednoznaczności dostaniesz listę kandydatów.",
schema(Map.of("ref", property("string", "SHA-1 bloba albo fragment ścieżki")),
List.of("ref")),
args -> getScript(string(args, "ref"))));
tools.add(new Tool("get_edition",
"Pełne metadane wydania: tytuł, silnik, daty, języki, kopie na dysku "
+ "oraz wszystkie wykryte fakty wraz z dowodem i pewnością.",
schema(Map.of("edition_id", property("integer", "Identyfikator wydania z list_titles")),
List.of("edition_id")),
args -> getEdition(integer(args, "edition_id", -1))));
tools.add(new Tool("list_files",
"Wypisuje pliki należące do wydania. Można zawęzić po formacie "
+ "(CNV, IMG, ANN, WAV...) i po fragmencie ścieżki.",
schema(Map.of(
"edition_id", property("integer", "Identyfikator wydania"),
"format", property("string", "Filtr formatu, np. CNV"),
"path_contains", property("string", "Fragment ścieżki"),
"limit", property("integer", "Maksymalna liczba plików (domyślnie 100)")
), List.of("edition_id")),
args -> listFiles(integer(args, "edition_id", -1), string(args, "format"),
string(args, "path_contains"), integer(args, "limit", 100))));
tools.add(new Tool("find_file",
"Szuka pliku po fragmencie ścieżki w całej kolekcji i pokazuje, w których "
+ "grach występuje oraz czy jest bit w bit ten sam. Przydatne do "
+ "porównywania wydań.",
schema(Map.of(
"path_contains", property("string", "Fragment ścieżki, np. arcade.cnv"),
"limit", property("integer", "Maksymalna liczba plików (domyślnie 50)")
), List.of("path_contains")),
args -> findFile(string(args, "path_contains"), integer(args, "limit", 50))));
tools.add(new Tool("get_script_assets",
"Zasoby, do których odwołuje się skrypt: pliki dźwiękowe, animacje, obrazy "
+ "i tablice. Przy kwestiach mówionych podaje długość nagrania, "
+ "postać, która je wypowiada, i zdarzenie, przy którym padają — "
+ "a jeśli transkrypcja została uruchomiona, także rozpoznaną treść. "
+ "Przy grafice: wymiary, liczbę klatek animacji i autora.",
schema(Map.of(
"ref", property("string", "SHA-1 skryptu albo fragment ścieżki"),
"format", property("string", "Zawęź do formatu, np. WAV")
), List.of("ref")),
args -> scriptAssets(string(args, "ref"), string(args, "format"))));
tools.add(new Tool("find_audio",
"Szuka nagrań po nazwie pliku, po postaci, która mówi, albo po treści "
+ "transkryptu. Zwraca długość, format, grę i — o ile jest — "
+ "rozpoznany tekst. Obejmuje też kwestie schowane w archiwum wav.snd.",
schema(Map.of(
"query", property("string", "Fragment nazwy, imię postaci albo słowa z kwestii"),
"limit", property("integer", "Maksymalna liczba trafień (domyślnie 30)")
), List.of("query")),
args -> findAudio(string(args, "query"), integer(args, "limit", 30))));
tools.add(new Tool("collection_stats",
"Statystyki kolekcji: liczba kopii, plików, unikalnych blobów, "
+ "rozkład formatów i stopień współdzielenia zasobów między grami.",
schema(Map.of(), List.of()),
args -> stats()));
return tools;
}
// ---------- implementacje ----------
/**
* Odwołania skryptu rozwiązane do konkretnych zasobów. Nierozwiązane też
* pokazujemy — nazwa, której na płycie nie ma, mówi coś o wydaniu.
*/
private String scriptAssets(String ref, String format) throws Exception {
List<String> candidates = search.resolve(ref);
if (candidates.isEmpty()) {
return "Nic nie pasuje do: " + ref;
}
if (candidates.size() > 1) {
StringBuilder sb = new StringBuilder("Niejednoznaczne — podaj SHA-1:\n");
for (String sha1 : candidates) {
sb.append(" ").append(sha1).append(" ")
.append(String.join(", ", search.gamesFor(sha1))).append('\n');
}
return sb.toString();
}
String sha1 = candidates.get(0);
StringBuilder sb = new StringBuilder();
sb.append(String.join(", ", search.pathsFor(sha1))).append('\n')
.append(String.join(", ", search.gamesFor(sha1))).append("\n\n");
String sql = pl.genschu.rexcatalog.media.MediaIndex.RESOLVE_SQL
+ (format == null || format.isBlank() ? "" : " AND r.format = ?5\n")
+ "ORDER BY r.format, r.line";
int missing = 0;
int shown = 0;
var place = new pl.genschu.rexcatalog.media.MediaIndex(db).place(sha1);
if (place == null) {
return sb + "Tego skryptu nie ma w żadnej kopii.";
}
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
pl.genschu.rexcatalog.media.MediaIndex.bindResolve(st, place, sha1);
if (format != null && !format.isBlank()) {
st.setString(5, format.toUpperCase(java.util.Locale.ROOT));
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String target = rs.getString("target");
if (target == null) {
missing++;
continue;
}
sb.append(String.format("%-26s %-9s %s%n", rs.getString("value"),
rs.getString("format"),
rs.getString("object") + ":" + rs.getString("field")));
String detail = audioDetail(target, rs.getString("speaker"),
rs.getString("trigger"));
if (!detail.isEmpty()) {
sb.append(" ").append(detail).append('\n');
}
shown++;
}
}
}
if (shown == 0 && missing == 0) {
return sb + "Ten skrypt nie odwołuje się do żadnych zasobów.";
}
if (missing > 0) {
sb.append(String.format("%n%d odwołań bez pliku na płycie.%n", missing));
}
return sb.toString();
}
/** Jedna linijka faktów o zasobie: ile trwa albo jak duży jest, kto mówi, co widać. */
private String audioDetail(String sha1, String speaker, String trigger) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT m.kind, m.duration_ms, m.codec, m.width, m.height, m.frames, m.fps,
m.author, t.text, t.model
FROM media m LEFT JOIN transcript t ON t.sha1 = m.sha1
WHERE m.sha1 = ?
""")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
if (!rs.next()) {
return "";
}
String kind = rs.getString("kind");
if ("image".equals(kind) || "anim".equals(kind)) {
StringBuilder graphic = new StringBuilder();
graphic.append(rs.getInt("width")).append('×').append(rs.getInt("height"));
if ("anim".equals(kind)) {
graphic.append(" · ").append(rs.getInt("frames")).append(" klatek");
if (rs.getInt("fps") > 0) {
graphic.append(" · ").append(rs.getInt("fps")).append(" kl./s");
}
}
if (rs.getString("author") != null) {
graphic.append(" · ").append(rs.getString("author"));
}
return graphic.toString();
}
if (!"audio".equals(kind)) {
return "";
}
StringBuilder sb = new StringBuilder();
sb.append(seconds(rs.getInt("duration_ms"))).append(' ')
.append(rs.getString("codec"));
if (speaker != null) {
sb.append(" · mówi ").append(speaker);
}
if (trigger != null) {
sb.append(" · przy ").append(trigger);
}
String text = rs.getString("text");
if (text != null && !text.isBlank()) {
sb.append("\n „").append(text.replace('\n', ' ').trim())
.append("” (rozpoznane maszynowo, ").append(rs.getString("model"))
.append(')');
}
return sb.toString();
}
}
}
private String findAudio(String query, int limit) throws Exception {
StringBuilder sb = new StringBuilder();
int found = 0;
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT klip.name, klip.copy, klip.sha1, m.duration_ms, m.codec,
(SELECT d.speaker FROM dialogue_line d
WHERE d.audio_name = klip.short LIMIT 1) AS speaker,
(SELECT d.trigger FROM dialogue_line d
WHERE d.audio_name = klip.short LIMIT 1) AS trigger,
t.text AS transcript
FROM (
SELECT f.blob_sha1 AS sha1, COALESCE(f.path_raw, f.path) AS name,
CASE WHEN instr(f.path, '/') > 0
THEN replace(f.path, rtrim(f.path, replace(f.path, '/', '')), '')
ELSE f.path END AS short,
c.display_name AS copy
FROM file f JOIN blob b ON b.sha1 = f.blob_sha1
JOIN copy c ON c.id = f.copy_id WHERE b.format = 'WAV'
UNION
SELECT ae.sha1, ae.name_raw, ae.name, c.display_name
FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
JOIN copy c ON c.id = f.copy_id
) AS klip
JOIN media m ON m.sha1 = klip.sha1
LEFT JOIN transcript t ON t.sha1 = klip.sha1
WHERE instr(lower(klip.name), lower(?)) > 0
OR (t.text IS NOT NULL AND instr(lower(t.text), lower(?)) > 0)
OR EXISTS (SELECT 1 FROM dialogue_line d
WHERE d.audio_name = klip.short
AND lower(d.speaker) = lower(?))
GROUP BY klip.sha1 ORDER BY klip.name LIMIT ?
""")) {
st.setString(1, query);
st.setString(2, query);
st.setString(3, query);
st.setInt(4, Math.min(limit, 200));
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
sb.append(String.format("%-26s %7s %s%n", rs.getString("name"),
seconds(rs.getInt("duration_ms")), rs.getString("copy")));
String speaker = rs.getString("speaker");
if (speaker != null) {
sb.append(" mówi ").append(speaker);
if (rs.getString("trigger") != null) {
sb.append(" · przy ").append(rs.getString("trigger"));
}
sb.append('\n');
}
String text = rs.getString("transcript");
if (text != null && !text.isBlank()) {
sb.append("").append(text.replace('\n', ' ').trim()).append("\n");
}
sb.append(" sha1: ").append(rs.getString("sha1")).append('\n');
found++;
}
}
}
return found == 0 ? "Nic nie pasuje do: " + query
: found + " nagrań:\n\n" + sb;
}
private static String seconds(int ms) {
return ms <= 0 ? "" : String.format(java.util.Locale.ROOT, "%.1f s", ms / 1000.0);
}
private String listTitles() throws Exception {
StringBuilder sb = new StringBuilder();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT e.id AS edition_id, t.name AS title, t.series, e.label,
COALESCE(e.release_date_override, (
SELECT d.value FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id AND d.root_path = g.root_path
WHERE g.edition_id = e.id AND d.field = 'release_date' LIMIT 1)) AS release_date,
COALESCE(e.engine_override, (
SELECT g.engine_detected FROM game_root g
WHERE g.edition_id = e.id LIMIT 1)) AS engine,
(SELECT GROUP_CONCAT(lang || '/' || role, ', ') FROM edition_language
WHERE edition_id = e.id) AS langs,
(SELECT COUNT(*) FROM game_root WHERE edition_id = e.id) AS roots
FROM edition e LEFT JOIN title t ON t.id = e.title_id
ORDER BY t.series IS NULL, t.series, t.name, release_date
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
sb.append("#").append(rs.getInt("edition_id")).append(" ")
.append(rs.getString("title"));
if (rs.getString("label") != null) {
sb.append(" [").append(rs.getString("label")).append("]");
}
sb.append("\n silnik: ").append(nvl(rs.getString("engine")))
.append(" data: ").append(nvl(rs.getString("release_date")))
.append(" kopie: ").append(rs.getInt("roots"))
.append(" języki: ").append(nvl(rs.getString("langs")))
.append("\n");
}
}
if (sb.isEmpty()) {
return "Katalog jest pusty — uruchom ingest, decode, analyze, promote.";
}
return sb + "\nDaty pochodzą z LASTMODIFYTIME w application.def i przybliżają "
+ "datę builda, nie datę wydania.";
}
private String searchScripts(String query, int limit) throws Exception {
ScriptSearch.Result result = search.find(query, Math.min(limit, 100));
if (result.hits().isEmpty()) {
return "Brak trafień dla: " + result.effectiveQuery();
}
StringBuilder sb = new StringBuilder("Trafienia dla ")
.append(result.effectiveQuery())
.append(result.quoted() ? " (potraktowane jako fraza)" : "")
.append(": ").append(result.hits().size()).append("\n\n");
for (ScriptSearch.Hit hit : result.hits()) {
sb.append(String.join("\n", hit.paths())).append("\n")
.append(" gry: ").append(String.join(", ", hit.games())).append("\n")
.append(" ").append(hit.snippet().replaceAll("\\s+", " ").trim()).append("\n")
.append(" sha1: ").append(hit.sha1()).append("\n\n");
}
return sb.toString();
}
private String getScript(String ref) throws Exception {
List<String> candidates = search.resolve(ref);
if (candidates.isEmpty()) {
return "Nic nie pasuje do: " + ref;
}
if (candidates.size() > 1) {
StringBuilder sb = new StringBuilder("Niejednoznaczne — "
+ candidates.size() + " różnych wersji tego pliku:\n");
for (String sha1 : candidates) {
sb.append(" ").append(sha1).append(" ")
.append(String.join(", ", search.gamesFor(sha1))).append("\n");
}
return sb + "\nPowtórz z wybranym SHA-1.";
}
String sha1 = candidates.get(0);
String body = search.body(sha1);
if (body == null) {
return "Ten plik nie jest odszyfrowany (nie jest skryptem albo brak go w cache'u).";
}
return "# " + String.join(", ", search.pathsFor(sha1))
+ "\n# gry: " + String.join(", ", search.gamesFor(sha1))
+ "\n# sha1: " + sha1 + "\n\n" + body;
}
private String getEdition(int editionId) throws Exception {
StringBuilder sb = new StringBuilder();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT e.id, t.name AS title, t.series, t.publisher, e.label, e.fingerprint,
e.dll_sha1, e.distributor, e.notes,
e.engine_override, e.engine_version_override,
e.compiler_override, e.release_date_override
FROM edition e LEFT JOIN title t ON t.id = e.title_id WHERE e.id = ?
""")) {
st.setInt(1, editionId);
try (ResultSet rs = st.executeQuery()) {
if (!rs.next()) {
return "Nie ma wydania o id " + editionId + ".";
}
sb.append("Tytuł: ").append(rs.getString("title")).append("\n");
append(sb, "Seria", rs.getString("series"));
append(sb, "Wydawca", rs.getString("publisher"));
append(sb, "Etykieta wydania", rs.getString("label"));
append(sb, "Dystrybutor", rs.getString("distributor"));
append(sb, "Notatki", rs.getString("notes"));
append(sb, "Hash biblioteki silnika", rs.getString("dll_sha1"));
append(sb, "Override silnika", rs.getString("engine_override"));
append(sb, "Override wersji silnika", rs.getString("engine_version_override"));
append(sb, "Override kompilera", rs.getString("compiler_override"));
append(sb, "Override daty", rs.getString("release_date_override"));
}
}
sb.append("\nWersje językowe:\n");
boolean any = false;
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT lang, role FROM edition_language WHERE edition_id = ? ORDER BY lang, role")) {
st.setInt(1, editionId);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
any = true;
sb.append(" ").append(rs.getString("lang"))
.append(" (").append(rs.getString("role")).append(")\n");
}
}
}
if (!any) {
sb.append(" (nie uzupełniono)\n");
}
sb.append("\nKopie na dysku:\n");
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT c.display_name, c.path, c.container, c.fs_type, c.status, c.file_count,
c.size, c.sha256, c.source_kind, c.source_url, g.root_path
FROM game_root g JOIN copy c ON c.id = g.copy_id
WHERE g.edition_id = ? ORDER BY c.display_name
""")) {
st.setInt(1, editionId);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
sb.append(" ").append(rs.getString("display_name"));
String root = rs.getString("root_path");
if (root != null && !root.isEmpty()) {
sb.append(" (korzeń ").append(root).append("/)");
}
sb.append("\n ").append(rs.getString("path"))
.append("\n ").append(rs.getString("container"))
.append(" / ").append(nvl(rs.getString("fs_type")))
.append(", ").append(rs.getInt("file_count")).append(" plików, ")
.append(rs.getLong("size") / 1024 / 1024).append(" MB")
.append(", status ").append(rs.getString("status"))
.append("\n źródło: ").append(nvl(rs.getString("source_kind")));
if (rs.getString("source_url") != null) {
sb.append("").append(rs.getString("source_url"));
}
sb.append("\n sha256: ").append(nvl(rs.getString("sha256"))).append("\n");
}
}
}
sb.append("\nWykryte fakty (z dowodem):\n");
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT DISTINCT d.field, d.value, d.confidence, d.evidence, d.detector
FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id AND d.root_path = g.root_path
WHERE g.edition_id = ? ORDER BY d.detector, d.field
""")) {
st.setInt(1, editionId);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
sb.append(String.format(" %-16s %-30s pewność %.2f%n",
rs.getString("field"), truncate(rs.getString("value"), 30),
rs.getDouble("confidence")));
sb.append(" dowód: ").append(truncate(rs.getString("evidence"), 100))
.append("\n");
}
}
}
return sb.toString();
}
private String listFiles(int editionId, String format, String pathContains, int limit)
throws Exception {
StringBuilder sql = new StringBuilder("""
SELECT DISTINCT COALESCE(f.path_raw, f.path) AS path, b.format, b.size,
b.encrypted, f.blob_sha1
FROM game_root g
JOIN file f ON f.copy_id = g.copy_id
JOIN blob b ON b.sha1 = f.blob_sha1
WHERE g.edition_id = ?
""");
List<Object> params = new ArrayList<>();
params.add(editionId);
if (format != null && !format.isBlank()) {
sql.append(" AND b.format = ?\n");
params.add(format.toUpperCase(java.util.Locale.ROOT));
}
if (pathContains != null && !pathContains.isBlank()) {
sql.append(" AND f.path LIKE ?\n");
params.add("%" + pathContains.toLowerCase(java.util.Locale.ROOT) + "%");
}
sql.append("ORDER BY path LIMIT ?");
params.add(Math.min(limit, 1000));
StringBuilder sb = new StringBuilder();
int count = 0;
try (PreparedStatement st = db.connection().prepareStatement(sql.toString())) {
for (int i = 0; i < params.size(); i++) {
st.setObject(i + 1, params.get(i));
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
count++;
sb.append(String.format("%-60s %-8s %8d B%s %s%n",
truncate(rs.getString("path"), 60), nvl(rs.getString("format")),
rs.getLong("size"), rs.getInt("encrypted") == 1 ? " [szyfr]" : "",
rs.getString("blob_sha1")));
}
}
}
if (count == 0) {
return "Brak plików spełniających kryteria.";
}
return "Plików: " + count + (count >= Math.min(limit, 1000)
? " (obcięte do limitu — zawęź filtr)" : "") + "\n\n" + sb;
}
private String findFile(String pathContains, int limit) throws Exception {
Map<String, List<String>> byPath = new LinkedHashMap<>();
Map<String, List<String>> hashes = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT f.path AS path, f.blob_sha1, c.display_name
FROM file f JOIN copy c ON c.id = f.copy_id
WHERE f.path LIKE ? ORDER BY f.path, c.display_name LIMIT ?
""")) {
st.setString(1, "%" + pathContains.toLowerCase(java.util.Locale.ROOT) + "%");
st.setInt(2, Math.min(limit, 500) * 8);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String key = rs.getString("path");
byPath.computeIfAbsent(key, k -> new ArrayList<>())
.add(rs.getString("display_name") + " ["
+ rs.getString("blob_sha1").substring(0, 8) + "]");
hashes.computeIfAbsent(key, k -> new ArrayList<>())
.add(rs.getString("blob_sha1"));
}
}
}
if (byPath.isEmpty()) {
return "Nic nie pasuje do: " + pathContains;
}
StringBuilder sb = new StringBuilder();
int shown = 0;
for (Map.Entry<String, List<String>> entry : byPath.entrySet()) {
if (shown++ >= limit) {
sb.append("\n(obcięte — zawęź fragment ścieżki)\n");
break;
}
long distinct = hashes.get(entry.getKey()).stream().distinct().count();
sb.append(entry.getKey()).append("\n ")
.append(String.join("\n ", entry.getValue())).append("\n")
.append(distinct == 1
? " → identyczna treść we wszystkich\n"
: "" + distinct + " różne wersje treści\n")
.append("\n");
}
return sb.toString();
}
private String stats() throws Exception {
StringBuilder sb = new StringBuilder();
sb.append("kopie: ").append(scalar("SELECT COUNT(*) FROM copy")).append("\n")
.append("tytuły: ").append(scalar("SELECT COUNT(*) FROM title")).append("\n")
.append("wydania: ").append(scalar("SELECT COUNT(*) FROM edition")).append("\n")
.append("pliki (wystąpienia): ").append(scalar("SELECT COUNT(*) FROM file")).append("\n")
.append("bloby (unikalne): ").append(scalar("SELECT COUNT(*) FROM blob")).append("\n")
.append("bloby w >1 kopii: ").append(scalar(
"SELECT COUNT(*) FROM (SELECT blob_sha1 FROM file GROUP BY blob_sha1 "
+ "HAVING COUNT(DISTINCT copy_id) > 1)")).append("\n")
.append("skrypty w indeksie: ").append(scalar("SELECT COUNT(*) FROM script_fts"))
.append("\n\nFormaty:\n");
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT format, COUNT(*) n, SUM(size) bytes FROM blob GROUP BY format "
+ "ORDER BY n DESC LIMIT 20");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
sb.append(String.format(" %-10s %6d %8.1f MB%n", nvl(rs.getString("format")),
rs.getInt("n"), rs.getLong("bytes") / 1024.0 / 1024.0));
}
}
return sb.toString();
}
// ---------- pomocnicze ----------
private long scalar(String sql) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(sql);
ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getLong(1) : 0;
}
}
private static void append(StringBuilder sb, String label, String value) {
if (value != null && !value.isBlank()) {
sb.append(label).append(": ").append(value).append("\n");
}
}
private static JsonObject property(String type, String description) {
JsonObject property = new JsonObject();
property.addProperty("type", type);
property.addProperty("description", description);
return property;
}
private static JsonObject schema(Map<String, JsonObject> properties, List<String> required) {
JsonObject schema = new JsonObject();
schema.addProperty("type", "object");
JsonObject props = new JsonObject();
properties.forEach(props::add);
schema.add("properties", props);
JsonArray requiredArray = new JsonArray();
required.forEach(requiredArray::add);
schema.add("required", requiredArray);
return schema;
}
private static String string(JsonObject args, String name) {
return args != null && args.has(name) && !args.get(name).isJsonNull()
? args.get(name).getAsString() : null;
}
private static int integer(JsonObject args, String name, int fallback) {
return args != null && args.has(name) && !args.get(name).isJsonNull()
? args.get(name).getAsInt() : fallback;
}
private static String nvl(String value) {
return value == null ? "" : value;
}
private static String truncate(String value, int max) {
if (value == null) {
return "";
}
return value.length() <= max ? value : value.substring(0, max - 1) + "";
}
}
@@ -0,0 +1,273 @@
package pl.genschu.rexcatalog.media;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.rexcatalog.db.Database;
import java.io.File;
import java.io.InputStream;
import java.security.MessageDigest;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
/**
* Wypełnia tabele {@code media} i {@code archive_entry} faktami z nagłówków dźwięku.
*
* <p>Jednostką pracy jest blob, tak jak przy skryptach — ten sam plik w kilku
* obrazach czytamy raz. Dla luźnych WAV-ów wystarczy początek pliku, więc 3,3 GB
* próbek nie przechodzi przez procesor; archiwa {@code .snd} czytamy w całości,
* bo liczymy odcisk każdego wpisu.
*/
public final class AudioIndexer {
/**
* Wersja własnych parserów, nie {@code :core} — te nagłówki czytamy sami,
* więc unieważnia je zmiana tego kodu, a nie bump emulatora.
*/
public static final String VERSION = "media-1";
private static final String KIND = "audio";
private final Database db;
public AudioIndexer(Database db) {
this.db = db;
}
public record Stats(int audio, int archives, int entries, int failed) {
}
private record Pending(String pathInContainer, String sha1, String format) {
}
public Stats run(boolean force) throws Exception {
Map<String, List<Pending>> byContainer = pendingWork(force);
if (byContainer.isEmpty()) {
return new Stats(0, 0, 0, 0);
}
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
int audio = 0;
int archives = 0;
int entries = 0;
int failed = 0;
Set<String> done = new HashSet<>();
try (PreparedStatement mediaStmt = conn.prepareStatement("""
INSERT INTO media (sha1, kind, codec, duration_ms, sample_rate, channels,
bits, note, tool_version, probed_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(sha1) DO UPDATE SET
kind = excluded.kind, codec = excluded.codec,
duration_ms = excluded.duration_ms, sample_rate = excluded.sample_rate,
channels = excluded.channels, bits = excluded.bits, note = excluded.note,
tool_version = excluded.tool_version, probed_at = excluded.probed_at
""");
PreparedStatement entryStmt = conn.prepareStatement("""
INSERT INTO archive_entry (container_sha1, name, name_raw, offset, size,
sha1, format, tool_version)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(container_sha1, name) DO UPDATE SET
name_raw = excluded.name_raw, offset = excluded.offset,
size = excluded.size, sha1 = excluded.sha1,
format = excluded.format, tool_version = excluded.tool_version
""")) {
for (Map.Entry<String, List<Pending>> group : byContainer.entrySet()) {
File container = new File(group.getKey());
IFileSystem fs;
try {
fs = AssetSourceDispatcher.openAssets(container);
} catch (Exception e) {
System.err.printf(" ! nie otwieram %s: %s%n",
container.getName(), e.getMessage());
failed += group.getValue().size();
continue;
}
for (Pending pending : group.getValue()) {
if (!done.add(pending.sha1())) {
continue;
}
try {
if ("SND".equals(pending.format())) {
entries += indexArchive(fs, pending, entryStmt, mediaStmt);
archives++;
} else {
probeLooseFile(fs, pending, mediaStmt);
audio++;
}
} catch (Exception e) {
System.err.printf(" ! %s (%s): %s%n", pending.pathInContainer(),
container.getName(), e.getMessage());
failed++;
}
}
conn.commit();
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return new Stats(audio, archives, entries, failed);
}
/** Luźny plik dźwiękowy: czytamy tyle, ile trzeba na nagłówek, i nic więcej. */
private void probeLooseFile(IFileSystem fs, Pending pending, PreparedStatement mediaStmt)
throws Exception {
byte[] head;
try (InputStream in = fs.open(pending.pathInContainer())) {
head = in.readNBytes(AudioProbe.WAV_HEAD);
}
writeMedia(mediaStmt, pending.sha1(), AudioProbe.probe(head));
}
/** Archiwum: każdy wpis dostaje własny odcisk, pozycję w kontenerze i fakty. */
private int indexArchive(IFileSystem fs, Pending pending, PreparedStatement entryStmt,
PreparedStatement mediaStmt) throws Exception {
int[] count = {0};
MessageDigest sha1 = MessageDigest.getInstance("SHA-1");
try (InputStream in = fs.open(pending.pathInContainer())) {
SndArchive.walk(in, (entry, payload) -> {
try {
sha1.reset();
String digest = hex(sha1.digest(payload));
AudioProbe.Info info = AudioProbe.probe(payload);
entryStmt.setString(1, pending.sha1());
entryStmt.setString(2, entry.name());
entryStmt.setString(3, entry.nameRaw());
entryStmt.setLong(4, entry.offset());
entryStmt.setInt(5, entry.size());
entryStmt.setString(6, digest);
entryStmt.setString(7, formatOf(entry.name(), info));
entryStmt.setString(8, VERSION);
entryStmt.executeUpdate();
writeMedia(mediaStmt, digest, info);
count[0]++;
} catch (Exception e) {
throw new java.io.IOException(
"wpis " + entry.nameRaw() + ": " + e.getMessage(), e);
}
});
}
return count[0];
}
/**
* Rozszerzenie nazwy bywa mylące — wpisy w {@code wav.snd} nazywają się
* {@code *.wav}, a w środku mają Vorbisa. Wierzymy zawartości.
*/
private static String formatOf(String name, AudioProbe.Info info) {
if ("vorbis".equals(info.codec())) {
return "OGG";
}
int dot = name.lastIndexOf('.');
return dot < 0 ? null : name.substring(dot + 1).toUpperCase(Locale.ROOT);
}
private void writeMedia(PreparedStatement stmt, String sha1, AudioProbe.Info info)
throws Exception {
stmt.setString(1, sha1);
stmt.setString(2, KIND);
stmt.setString(3, info.codec());
if (info.durationMs() < 0) {
stmt.setNull(4, java.sql.Types.INTEGER);
} else {
stmt.setInt(4, info.durationMs());
}
stmt.setInt(5, info.sampleRate());
stmt.setInt(6, info.channels());
stmt.setInt(7, info.bits());
stmt.setString(8, info.note());
stmt.setString(9, VERSION);
stmt.setString(10, Instant.now().toString());
stmt.executeUpdate();
}
/**
* Zbiera pliki do przerobienia, pogrupowane po kontenerze — żeby każdy obraz
* otwierać raz. Luźne WAV-y sprawdzamy po tabeli {@code media}, archiwa po tym,
* czy mają już wpisy z aktualną wersją parsera.
*/
private Map<String, List<Pending>> pendingWork(boolean force) throws Exception {
String condition = force ? "1 = 1" : """
(
(b.format = 'WAV' AND (m.sha1 IS NULL OR m.tool_version <> ?))
OR
(b.format = 'SND' AND NOT EXISTS (
SELECT 1 FROM archive_entry ae
WHERE ae.container_sha1 = b.sha1 AND ae.tool_version = ?))
)
""";
String sql = """
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
f.blob_sha1, b.format
FROM copy c
JOIN file f ON f.copy_id = c.id
JOIN blob b ON b.sha1 = f.blob_sha1
LEFT JOIN media m ON m.sha1 = b.sha1
WHERE c.status <> 'unreadable'
AND b.format IN ('WAV', 'SND')
AND %s
ORDER BY c.id
""".formatted(condition);
Map<String, List<Pending>> byContainer = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
if (!force) {
st.setString(1, VERSION);
st.setString(2, VERSION);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
byContainer.computeIfAbsent(rs.getString("container"), k -> new ArrayList<>())
.add(new Pending(rs.getString("in_container"),
rs.getString("blob_sha1"), rs.getString("format")));
}
}
}
return byContainer;
}
/** Ile plików czeka na przerobienie — do komunikatu przed startem. */
public int pendingCount() throws Exception {
int total = 0;
Set<String> unique = new HashSet<>();
for (List<Pending> list : pendingWork(false).values()) {
for (Pending pending : list) {
if (unique.add(pending.sha1())) {
total++;
}
}
}
return total;
}
private static String hex(byte[] bytes) {
StringBuilder out = new StringBuilder(bytes.length * 2);
for (byte b : bytes) {
out.append(Character.forDigit((b >> 4) & 0xF, 16));
out.append(Character.forDigit(b & 0xF, 16));
}
return out.toString();
}
}
@@ -0,0 +1,195 @@
package pl.genschu.rexcatalog.media;
import java.nio.charset.StandardCharsets;
/**
* Odczyt nagłówków dźwięku: RIFF/WAVE oraz Ogg Vorbis.
*
* <p>Bez {@code :core} i bez libGDX — {@code SoundLoader} z emulatora wymaga
* {@code FileHandle} i działającego {@code Gdx.audio}, a jego wkładem jest
* parsowanie standardowego nagłówka RIFF. Tutaj czytamy prosto ze strumienia
* z obrazu płyty, bez plików tymczasowych.
*
* <p>Dla WAV wystarczy początek pliku: długość dźwięku stoi w nagłówku porcji
* {@code data}, więc 3,3 GB próbek nie musi przejść przez procesor.
*/
public final class AudioProbe {
/** Ile bajtów początku pliku wystarcza, by znaleźć porcje {@code fmt } i {@code data}. */
public static final int WAV_HEAD = 4096;
private AudioProbe() {
}
/**
* @param codec nazwa kodeka albo jego numer, gdy nieznany
* @param durationMs długość w milisekundach, {@code -1} gdy nie do wyliczenia
* @param note co poszło nie tak, gdy nagłówek był nietypowy
*/
public record Info(String codec, int durationMs, int sampleRate, int channels,
int bits, String note) {
public static Info unreadable(String note) {
return new Info(null, -1, 0, 0, 0, note);
}
}
/** Rozpoznaje kontener po magicznych bajtach i wybiera właściwy parser. */
public static Info probe(byte[] data) {
if (data.length >= 4 && matches(data, 0, "OggS")) {
return probeOgg(data);
}
if (data.length >= 12 && matches(data, 0, "RIFF") && matches(data, 8, "WAVE")) {
return probeWav(data);
}
return Info.unreadable("nierozpoznany kontener dźwięku");
}
// ---------- RIFF / WAVE ----------
private static Info probeWav(byte[] data) {
int position = 12;
int format = 0;
int channels = 0;
int sampleRate = 0;
int bits = 0;
long byteRate = 0;
long dataSize = -1;
while (position + 8 <= data.length) {
String chunkId = new String(data, position, 4, StandardCharsets.US_ASCII);
long chunkSize = u32(data, position + 4);
int body = position + 8;
if (chunkId.equals("fmt ") && body + 16 <= data.length) {
format = u16(data, body);
channels = u16(data, body + 2);
sampleRate = (int) u32(data, body + 4);
byteRate = u32(data, body + 8);
bits = u16(data, body + 14);
} else if (chunkId.equals("data")) {
dataSize = chunkSize;
break;
}
// porcje są wyrównane do parzystej granicy
position = body + (int) chunkSize + (int) (chunkSize & 1);
if (chunkSize <= 0) {
break;
}
}
if (sampleRate <= 0 || channels <= 0) {
return Info.unreadable("brak czytelnej porcji fmt");
}
int durationMs = -1;
if (dataSize >= 0) {
// dla kompresji (ADPCM, MP3 w kontenerze WAV) liczba próbek nie wynika
// z liczby bajtów, ale byteRate jest w nagłówku deklarowany zawsze
long bytesPerSecond = byteRate > 0
? byteRate
: (long) sampleRate * channels * Math.max(bits, 8) / 8;
if (bytesPerSecond > 0) {
durationMs = (int) (dataSize * 1000 / bytesPerSecond);
}
}
return new Info(wavCodec(format), durationMs, sampleRate, channels, bits,
dataSize < 0 ? "nie znaleziono porcji data w nagłówku" : null);
}
private static String wavCodec(int format) {
return switch (format) {
case 0x0001 -> "pcm";
case 0x0002 -> "ms-adpcm";
case 0x0011 -> "ima-adpcm";
case 0x0055 -> "mp3";
case 0x0161 -> "wma";
case 0xFFFE -> "pcm-extensible";
default -> "wav-0x%04X".formatted(format);
};
}
// ---------- Ogg Vorbis ----------
/**
* Wymaga całego payloadu, bo długość dźwięku stoi w pozycji ziarna ostatniej
* strony. Wpisy w {@code wav.snd} mają średnio 24 kB, więc to nie jest problem.
*/
public static Info probeOgg(byte[] data) {
int channels = 0;
int sampleRate = 0;
long lastGranule = -1;
int position = 0;
boolean sawIdentification = false;
while (position + 27 <= data.length && matches(data, position, "OggS")) {
int segmentCount = data[position + 26] & 0xFF;
int segmentTable = position + 27;
if (segmentTable + segmentCount > data.length) {
break;
}
int payloadSize = 0;
for (int i = 0; i < segmentCount; i++) {
payloadSize += data[segmentTable + i] & 0xFF;
}
int payload = segmentTable + segmentCount;
long granule = u64(data, position + 6);
if (granule != -1L) {
lastGranule = granule;
}
if (!sawIdentification && payloadSize >= 16 && payload + 16 <= data.length
&& (data[payload] & 0xFF) == 0x01 && matches(data, payload + 1, "vorbis")) {
channels = data[payload + 11] & 0xFF;
sampleRate = (int) u32(data, payload + 12);
sawIdentification = true;
}
position = payload + payloadSize;
}
if (!sawIdentification || sampleRate <= 0) {
return Info.unreadable("brak nagłówka identyfikacyjnego Vorbisa");
}
// ziarno równe zeru to nie brak informacji, tylko pusty strumień —
// w wav.snd leży kilkanaście takich plików, nazwanych wprost _pusty0.wav
int durationMs = lastGranule >= 0
? (int) (lastGranule * 1000 / sampleRate)
: -1;
return new Info("vorbis", durationMs, sampleRate, channels, 0, null);
}
// ---------- pomocnicze ----------
private static boolean matches(byte[] data, int offset, String ascii) {
if (offset + ascii.length() > data.length) {
return false;
}
for (int i = 0; i < ascii.length(); i++) {
if ((data[offset + i] & 0xFF) != ascii.charAt(i)) {
return false;
}
}
return true;
}
private static int u16(byte[] data, int offset) {
return (data[offset] & 0xFF) | ((data[offset + 1] & 0xFF) << 8);
}
private static long u32(byte[] data, int offset) {
return (data[offset] & 0xFFL)
| ((data[offset + 1] & 0xFFL) << 8)
| ((data[offset + 2] & 0xFFL) << 16)
| ((data[offset + 3] & 0xFFL) << 24);
}
private static long u64(byte[] data, int offset) {
long value = 0;
for (int i = 7; i >= 0; i--) {
value = (value << 8) | (data[offset + i] & 0xFFL);
}
return value;
}
}
@@ -0,0 +1,279 @@
package pl.genschu.rexcatalog.media;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.rexcatalog.db.Database;
import java.io.File;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
/**
* Czyta tabele {@code .dta} i wyciąga z nich wiersze dialogów.
*
* <p>{@code dialogi.dta} to zwykły tekst CP1250 rozdzielony pionowymi kreskami:
* <pre>
* click_zawor0|Rex_Aw07|rex
* use_mlotek_on_tlok0|Kret_Aw14|rex|f__
* </pre>
* Kolumny to zdarzenie, identyfikator kwestii, rozmówca i flagi. Identyfikator
* odpowiada nazwie pliku dźwiękowego, a jego przedrostek — postaci
* ({@code Kret} to Kretes, {@code Nar} narrator).
*
* <p>Dzięki temu każda kwestia ma mówcę i kontekst zdarzenia bez transkrypcji.
* Wiersze zaczynające się od myślników są komentarzami autorów i niosą nazwy
* planszy — trzymamy je jako sekcję kolejnych wierszy.
*/
public final class DialogueIndexer {
public static final String VERSION = "dialogue-1";
private static final Charset CP1250 = Charset.forName("windows-1250");
/** Identyfikator kwestii: przedrostek postaci, podkreślenie, oznaczenie sceny. */
private static final Pattern AUDIO_ID = Pattern.compile("[A-Za-z0-9]{2,}_[A-Za-z0-9]+");
private static final Pattern SECTION = Pattern.compile("^-{3,}\\s*(.*)$");
private final Database db;
public DialogueIndexer(Database db) {
this.db = db;
}
public record Stats(int files, int lines, int failed) {
}
private record Pending(String pathInContainer, String sha1) {
}
public Stats run(boolean force) throws Exception {
Map<String, List<Pending>> byContainer = pendingWork(force);
if (byContainer.isEmpty()) {
return new Stats(0, 0, 0);
}
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
int files = 0;
int lines = 0;
int failed = 0;
Set<String> done = new HashSet<>();
try (PreparedStatement delete = conn.prepareStatement(
"DELETE FROM dialogue_line WHERE dta_sha1 = ?");
PreparedStatement insert = conn.prepareStatement("""
INSERT INTO dialogue_line (dta_sha1, line, section, trigger,
audio_name, speaker, extra)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(dta_sha1, line) DO UPDATE SET
section = excluded.section, trigger = excluded.trigger,
audio_name = excluded.audio_name, speaker = excluded.speaker,
extra = excluded.extra
""");
// ślad, że plik już przeszedł przez parser — inaczej tabele .dta bez
// dialogów (zapisy stanu gry) czytalibyśmy przy każdym uruchomieniu
PreparedStatement mark = conn.prepareStatement("""
INSERT INTO media (sha1, kind, note, tool_version, probed_at)
VALUES (?, 'dialogue', ?, ?, ?)
ON CONFLICT(sha1) DO UPDATE SET
kind = excluded.kind, note = excluded.note,
tool_version = excluded.tool_version, probed_at = excluded.probed_at
""")) {
for (Map.Entry<String, List<Pending>> group : byContainer.entrySet()) {
File container = new File(group.getKey());
IFileSystem fs;
try {
fs = AssetSourceDispatcher.openAssets(container);
} catch (Exception e) {
System.err.printf(" ! nie otwieram %s: %s%n",
container.getName(), e.getMessage());
failed += group.getValue().size();
continue;
}
for (Pending pending : group.getValue()) {
if (!done.add(pending.sha1())) {
continue;
}
try {
byte[] raw;
try (InputStream in = fs.open(pending.pathInContainer())) {
raw = in.readAllBytes();
}
delete.setString(1, pending.sha1());
delete.executeUpdate();
int written = parse(pending.sha1(), new String(raw, CP1250), insert);
if (written > 0) {
lines += written;
files++;
}
mark.setString(1, pending.sha1());
mark.setString(2, written + " wierszy dialogu");
mark.setString(3, VERSION);
mark.setString(4, java.time.Instant.now().toString());
mark.executeUpdate();
} catch (Exception e) {
System.err.printf(" ! %s (%s): %s%n", pending.pathInContainer(),
container.getName(), e.getMessage());
failed++;
}
}
conn.commit();
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return new Stats(files, lines, failed);
}
/**
* Zapisuje tylko wiersze, w których druga kolumna wygląda na identyfikator
* kwestii. To odsiewa inne tabele {@code .dta} — zapisy stanu gry mają ten sam
* format, ale w drugiej kolumnie zwykłe wartości.
*/
private int parse(String sha1, String text, PreparedStatement insert) throws Exception {
String[] rows = text.split("\r?\n", -1);
String section = null;
int written = 0;
for (int number = 0; number < rows.length; number++) {
String row = rows[number].trim();
if (row.isEmpty()) {
continue;
}
String[] columns = row.split("\\|", -1);
String marker = sectionLabel(row, columns);
if (marker != null) {
section = marker.isEmpty() ? null : marker;
continue;
}
if (columns.length < 2) {
continue;
}
String id = columns[1].trim();
if (!AUDIO_ID.matcher(id).matches() || !isTrigger(columns[0])) {
continue;
}
insert.setString(1, sha1);
insert.setInt(2, number + 1);
insert.setString(3, section);
insert.setString(4, blankToNull(columns[0]));
insert.setString(5, audioName(id));
insert.setString(6, speaker(id));
insert.setString(7, columns.length > 2
? blankToNull(String.join("|",
java.util.Arrays.copyOfRange(columns, 2, columns.length)))
: null);
insert.executeUpdate();
written++;
}
return written;
}
/**
* Pierwsza kolumna wiersza dialogu to nazwa zdarzenia, nie plik.
*
* <p>Ten sam format {@code .dta} niosą tabele definicji planszy
* ({@code MIASTO\\DO_ALCHOMIKA.ANN|DO_ALCHOMIKA|...}), gdzie druga kolumna
* wygląda jak identyfikator kwestii, choć jest nazwą przejścia. Ścieżka
* w kolumnie zerowej rozstrzyga, że to nie dialog.
*/
private static boolean isTrigger(String column) {
String value = column.trim();
return !value.isEmpty()
&& value.indexOf('\\') < 0
&& value.indexOf('/') < 0
&& value.indexOf('.') < 0;
}
/** Komentarz autorów: myślniki na początku wiersza albo w którejś kolumnie. */
private static String sectionLabel(String row, String[] columns) {
var direct = SECTION.matcher(row);
if (direct.matches()) {
return direct.group(1).trim();
}
for (String column : columns) {
var inside = SECTION.matcher(column.trim());
if (inside.matches()) {
return inside.group(1).trim();
}
}
return null;
}
/** Identyfikatory są bez rozszerzenia, a pliki i wpisy archiwum kończą się na .wav. */
static String audioName(String id) {
String name = id.toLowerCase(Locale.ROOT);
return name.contains(".") ? name : name + ".wav";
}
static String speaker(String id) {
int underscore = id.indexOf('_');
return underscore <= 0 ? null : id.substring(0, underscore);
}
private static String blankToNull(String value) {
String trimmed = value == null ? "" : value.trim();
return trimmed.isEmpty() ? null : trimmed;
}
private Map<String, List<Pending>> pendingWork(boolean force) throws Exception {
String sql = """
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
f.blob_sha1
FROM copy c
JOIN file f ON f.copy_id = c.id
JOIN blob b ON b.sha1 = f.blob_sha1
LEFT JOIN media m ON m.sha1 = b.sha1
WHERE c.status <> 'unreadable' AND b.format = 'DTA'
""" + (force ? "" : """
AND (m.sha1 IS NULL OR m.tool_version <> ?)
""") + "ORDER BY c.id";
Map<String, List<Pending>> byContainer = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
if (!force) {
st.setString(1, VERSION);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
byContainer.computeIfAbsent(rs.getString("container"), k -> new ArrayList<>())
.add(new Pending(rs.getString("in_container"),
rs.getString("blob_sha1")));
}
}
}
return byContainer;
}
public int pendingCount() throws Exception {
Set<String> unique = new HashSet<>();
for (List<Pending> list : pendingWork(false).values()) {
list.forEach(pending -> unique.add(pending.sha1()));
}
return unique.size();
}
}
@@ -0,0 +1,291 @@
package pl.genschu.rexcatalog.media;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.rexcatalog.db.Database;
import javax.imageio.ImageIO;
import java.awt.Graphics2D;
import java.awt.RenderingHints;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Buduje miniatury obrazów i animacji oraz zapisuje ich fakty.
*
* <p>W cache'u ląduje tylko miniatura — pełny obraz odtwarzamy z płyty na żądanie.
* Inaczej 10 459 plików rozpakowanych do PNG zajęłoby więcej niż sama kolekcja,
* a odczyt z obrazu i tak trwa milisekundy.
*
* <p>Animacje dostają dodatkowo listę nazwanych zdarzeń: {@code BEZRUCH},
* {@code GADA_START}, {@code GADA_1} — to opis tego, co postać potrafi zrobić,
* więc traktujemy je jak treść, a nie jak liczbę klatek.
*/
public final class GraphicsIndexer {
public static final String VERSION = "grafika-1";
private static final String KIND = "preview";
/** Dłuższy bok miniatury. Podgląd ma się otwierać od razu, nie ładować. */
private static final int THUMBNAIL = 320;
private final Database db;
private final Path dataDir;
public GraphicsIndexer(Database db, Path dataDir) {
this.db = db;
this.dataDir = dataDir;
}
public record Stats(int images, int animations, int frames, int failed) {
}
private record Pending(String pathInContainer, String sha1, String format) {
}
public Stats run(boolean force) throws Exception {
Map<String, List<Pending>> byContainer = pendingWork(force);
if (byContainer.isEmpty()) {
return new Stats(0, 0, 0, 0);
}
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
int images = 0;
int animations = 0;
int frames = 0;
int failed = 0;
Set<String> done = new HashSet<>();
try (PreparedStatement mediaStmt = conn.prepareStatement("""
INSERT INTO media (sha1, kind, codec, width, height, frames, fps, author,
note, tool_version, probed_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(sha1) DO UPDATE SET
kind = excluded.kind, codec = excluded.codec, width = excluded.width,
height = excluded.height, frames = excluded.frames, fps = excluded.fps,
author = excluded.author, note = excluded.note,
tool_version = excluded.tool_version, probed_at = excluded.probed_at
""");
PreparedStatement artifactStmt = conn.prepareStatement("""
INSERT INTO artifact (blob_sha1, kind, path, tool_version, created_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(blob_sha1, kind) DO UPDATE SET
path = excluded.path, tool_version = excluded.tool_version,
created_at = excluded.created_at
""");
PreparedStatement eventDelete = conn.prepareStatement(
"DELETE FROM anim_event WHERE ann_sha1 = ?");
PreparedStatement eventInsert = conn.prepareStatement("""
INSERT INTO anim_event (ann_sha1, name, frames_count, loop_start, loop_end)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(ann_sha1, name) DO UPDATE SET
frames_count = excluded.frames_count, loop_start = excluded.loop_start,
loop_end = excluded.loop_end
""")) {
for (Map.Entry<String, List<Pending>> group : byContainer.entrySet()) {
File container = new File(group.getKey());
IFileSystem fs;
try {
fs = AssetSourceDispatcher.openAssets(container);
} catch (Exception e) {
System.err.printf(" ! nie otwieram %s: %s%n",
container.getName(), e.getMessage());
failed += group.getValue().size();
continue;
}
for (Pending pending : group.getValue()) {
if (!done.add(pending.sha1())) {
continue;
}
try {
byte[] raw;
try (InputStream in = fs.open(pending.pathInContainer())) {
raw = in.readAllBytes();
}
if ("ANN".equals(pending.format())) {
frames += indexAnimation(pending.sha1(), raw, mediaStmt, artifactStmt,
eventDelete, eventInsert);
animations++;
} else {
indexImage(pending.sha1(), raw, mediaStmt, artifactStmt);
images++;
}
} catch (Throwable e) {
// uszkodzone i nietypowe pliki zdarzają się na tłoczonych płytach;
// zapisujemy powód, żeby dało się je potem odnaleźć
markUnreadable(mediaStmt, pending, e);
failed++;
}
}
conn.commit();
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return new Stats(images, animations, frames, failed);
}
private void indexImage(String sha1, byte[] raw, PreparedStatement mediaStmt,
PreparedStatement artifactStmt) throws Exception {
PikGraphics.Picture picture = PikGraphics.readImage(raw);
writePreview(sha1, picture.image(), artifactStmt);
write(mediaStmt, sha1, "image", "pik" + picture.colorDepth(), picture.width(),
picture.height(), 1, 0, null,
"kompresja " + picture.compression());
}
private int indexAnimation(String sha1, byte[] raw, PreparedStatement mediaStmt,
PreparedStatement artifactStmt, PreparedStatement eventDelete,
PreparedStatement eventInsert) throws Exception {
PikGraphics.Animation animation = PikGraphics.readAnimation(raw);
if (!animation.frames().isEmpty()) {
writePreview(sha1, animation.frames().get(0).image(), artifactStmt);
}
write(mediaStmt, sha1, "anim", "nvp" + animation.colorDepth(), animation.maxWidth(),
animation.maxHeight(), animation.frames().size(), animation.fps(),
blankToNull(animation.author()), blankToNull(animation.description()));
eventDelete.setString(1, sha1);
eventDelete.executeUpdate();
for (PikGraphics.AnimEvent event : animation.events()) {
eventInsert.setString(1, sha1);
eventInsert.setString(2, event.name());
eventInsert.setInt(3, event.framesCount());
eventInsert.setInt(4, event.loopStart());
eventInsert.setInt(5, event.loopEnd());
eventInsert.executeUpdate();
}
return animation.frames().size();
}
private void writePreview(String sha1, BufferedImage image, PreparedStatement artifactStmt)
throws Exception {
String relative = cachePath(sha1);
Path target = dataDir.resolve(relative);
Files.createDirectories(target.getParent());
ImageIO.write(thumbnail(image), "png", target.toFile());
artifactStmt.setString(1, sha1);
artifactStmt.setString(2, KIND);
artifactStmt.setString(3, relative);
artifactStmt.setString(4, VERSION);
artifactStmt.setString(5, Instant.now().toString());
artifactStmt.executeUpdate();
}
/** Zmniejsza z zachowaniem proporcji; mniejsze obrazy zostawia w spokoju. */
static BufferedImage thumbnail(BufferedImage source) {
int width = source.getWidth();
int height = source.getHeight();
if (width <= THUMBNAIL && height <= THUMBNAIL) {
return source;
}
double scale = (double) THUMBNAIL / Math.max(width, height);
int newWidth = Math.max(1, (int) Math.round(width * scale));
int newHeight = Math.max(1, (int) Math.round(height * scale));
BufferedImage out = new BufferedImage(newWidth, newHeight, BufferedImage.TYPE_INT_ARGB);
Graphics2D g = out.createGraphics();
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BILINEAR);
g.drawImage(source, 0, 0, newWidth, newHeight, null);
g.dispose();
return out;
}
private void markUnreadable(PreparedStatement mediaStmt, Pending pending, Throwable cause)
throws Exception {
String reason = cause.getMessage() == null ? cause.toString() : cause.getMessage();
write(mediaStmt, pending.sha1(), "ANN".equals(pending.format()) ? "anim" : "image",
null, 0, 0, 0, 0, null, "nieczytelny: " + reason);
}
private void write(PreparedStatement stmt, String sha1, String kind, String codec,
int width, int height, int frames, int fps, String author, String note)
throws Exception {
stmt.setString(1, sha1);
stmt.setString(2, kind);
stmt.setString(3, codec);
stmt.setInt(4, width);
stmt.setInt(5, height);
stmt.setInt(6, frames);
stmt.setInt(7, fps);
stmt.setString(8, author);
stmt.setString(9, note);
stmt.setString(10, VERSION);
stmt.setString(11, Instant.now().toString());
stmt.executeUpdate();
}
private static String blankToNull(String value) {
return value == null || value.isBlank() ? null : value.trim();
}
/** Rozgałęzienie po dwóch pierwszych znakach — inaczej jeden katalog na 10 tysięcy plików. */
private static String cachePath(String sha1) {
return "artifacts/" + KIND + "/" + sha1.substring(0, 2) + "/" + sha1 + ".png";
}
private Map<String, List<Pending>> pendingWork(boolean force) throws Exception {
String sql = """
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
f.blob_sha1, b.format
FROM copy c
JOIN file f ON f.copy_id = c.id
JOIN blob b ON b.sha1 = f.blob_sha1
LEFT JOIN media m ON m.sha1 = b.sha1
WHERE c.status <> 'unreadable' AND b.format IN ('IMG', 'ANN')
""" + (force ? "" : """
AND (m.sha1 IS NULL OR m.tool_version <> ?)
""") + "ORDER BY c.id";
Map<String, List<Pending>> byContainer = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
if (!force) {
st.setString(1, VERSION);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
byContainer.computeIfAbsent(rs.getString("container"), k -> new ArrayList<>())
.add(new Pending(rs.getString("in_container"),
rs.getString("blob_sha1"), rs.getString("format")));
}
}
}
return byContainer;
}
public int pendingCount() throws Exception {
Set<String> unique = new HashSet<>();
for (List<Pending> list : pendingWork(false).values()) {
list.forEach(pending -> unique.add(pending.sha1()));
}
return unique.size();
}
}
@@ -0,0 +1,295 @@
package pl.genschu.rexcatalog.media;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.rexcatalog.db.Database;
import java.io.File;
import java.io.InputStream;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Locale;
/**
* Wyszukuje zasoby po nazwie i podaje ich bajty prosto z obrazu płyty.
*
* <p>Niczego nie wypakowujemy na dysk: kopia zostaje jednym plikiem, a katalog
* odczytuje z niej fragment na żądanie. Zasób bywa osobnym plikiem albo wpisem
* w archiwum {@code .snd} — dla odbiorcy to bez różnicy, bo jedno i drugie
* adresujemy odciskiem treści.
*/
public final class MediaIndex {
private final Database db;
public MediaIndex(Database db) {
this.db = db;
}
/**
* @param name nazwa pliku do pokazania i pobrania
* @param contentType typ MIME wynikający z rzeczywistej zawartości
* @param inArchive czy zasób leży w archiwum, a nie osobno na płycie
*/
public record Clip(String sha1, String name, String contentType, byte[] bytes,
boolean inArchive) {
}
/** Miejsce zasobu w kontenerze: {@code offset < 0} oznacza cały plik. */
private record Location(String container, String pathInContainer, String name,
String format, long offset, int size) {
}
/**
* Gdzie silnik szuka pliku, do którego odwołuje się skrypt.
*
* <p>Reguły są takie same jak w Piklibie i nie są heurystyką:
* <ul>
* <li>{@code FILENAME=SHIP_MASK.IMG} — bez ścieżki i bez dolara oznacza plik
* <b>obok skryptu</b>, w katalogu sceny;</li>
* <li>{@code $} to <b>korzeń gry</b>, więc {@code $COMMON\DIALOGS.DTA} wskazuje
* {@code <korzeń>/common/dialogs.dta}, a {@code $WAVS\...} — katalog wavs;</li>
* <li>pliki WAV leżą w {@code wavs/}; efekty dźwiękowe nie są tu wołane wprost,
* bo podpina je animacja.</li>
* </ul>
*
* <p>Bez tego rozróżnienia nazwa {@code bkg.img} jest bezużyteczna: w samej
* <i>Wojnie Trojańskiej</i> jest ich siedemnaście, po jednym na planszę.
* Dopasowanie po samej nazwie zostaje jako ostatnia deska ratunku i jest
* oznaczane jako niepewne.
*
* <p>Parametry: katalog skryptu, korzeń gry, identyfikator kopii (kilka razy)
* i odcisk skryptu — podane wprost, a nie liczone we wspólnym wyrażeniu, bo
* SQLite przeliczałby je dla każdego z kilkuset odwołań osobno.
*/
public static final String RESOLVE_SQL = """
SELECT r.object, r.field, r.value, r.name, r.format, r.line,
COALESCE(
-- $ = korzeń gry
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4 AND r.value LIKE '$%'
AND f2.path = ?2 || ltrim(substr(lower(replace(r.value, '\', '/')), 2), '/')
LIMIT 1),
-- obok skryptu
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4 AND f2.path = ?1 || r.name LIMIT 1),
-- ścieżka względna wobec katalogu skryptu
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4
AND f2.path = ?1 || lower(replace(r.value, '\', '/')) LIMIT 1),
-- dźwięk mowy: katalog wavs w korzeniu gry
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4 AND r.format = 'WAV'
AND f2.path = ?2 || 'wavs/' || r.name LIMIT 1),
-- muzyka bywa wprost w korzeniu
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4 AND f2.path = ?2 || r.name LIMIT 1),
-- kwestie spakowane do archiwum wav.snd
(SELECT ae.sha1 FROM archive_entry ae
JOIN file f3 ON f3.blob_sha1 = ae.container_sha1
WHERE f3.copy_id = ?4 AND ae.name = r.name LIMIT 1),
-- ostatnia deska ratunku: ta sama nazwa gdziekolwiek w kopii
(SELECT f2.blob_sha1 FROM file f2
WHERE f2.copy_id = ?4 AND f2.basename = r.name LIMIT 1)
) AS target,
(SELECT COUNT(DISTINCT f2.blob_sha1) FROM file f2
WHERE f2.copy_id = ?4 AND f2.basename = r.name) AS wariantow,
(SELECT 1 FROM file f2
WHERE f2.copy_id = ?4
AND (f2.path = ?1 || r.name
OR f2.path = ?1 || lower(replace(r.value, '\', '/'))
OR (r.value LIKE '$%' AND f2.path = ?2
|| ltrim(substr(lower(replace(r.value, '\', '/')), 2), '/'))
OR (r.format = 'WAV' AND f2.path = ?2 || 'wavs/' || r.name)
OR f2.path = ?2 || r.name)
LIMIT 1) AS pewne,
(SELECT d.speaker FROM dialogue_line d
WHERE d.audio_name = r.name LIMIT 1) AS speaker,
(SELECT d.trigger FROM dialogue_line d
WHERE d.audio_name = r.name LIMIT 1) AS trigger
FROM script_ref r
WHERE r.script_sha1 = ?3
""";
/** Katalog skryptu, korzeń gry i kopia — kontekst potrzebny do {@link #RESOLVE_SQL}. */
public record ScriptPlace(long copyId, String dir, String root) {
}
/**
* Ustala, gdzie w kopii leży skrypt i który korzeń gry go obejmuje.
* Na dwupacku liczy się korzeń najgłębszy z pasujących.
*/
public ScriptPlace place(String scriptSha1) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT f.copy_id,
CASE WHEN instr(f.path, '/') > 0
THEN rtrim(f.path, replace(f.path, '/', ''))
ELSE '' END AS dir,
COALESCE((
SELECT CASE WHEN g.root_path = '' THEN ''
ELSE g.root_path || '/' END
FROM game_root g
WHERE g.copy_id = f.copy_id
AND (g.root_path = '' OR f.path LIKE g.root_path || '/%')
ORDER BY length(g.root_path) DESC LIMIT 1
), '') AS root
FROM file f WHERE f.blob_sha1 = ? LIMIT 1
""")) {
st.setString(1, scriptSha1);
try (ResultSet rs = st.executeQuery()) {
return rs.next()
? new ScriptPlace(rs.getLong("copy_id"), rs.getString("dir"),
rs.getString("root"))
: null;
}
}
}
/** Wiąże parametry {@link #RESOLVE_SQL} w ustalonej kolejności. */
public static void bindResolve(PreparedStatement st, ScriptPlace place, String scriptSha1)
throws java.sql.SQLException {
st.setString(1, place.dir());
st.setString(2, place.root());
st.setString(3, scriptSha1);
st.setLong(4, place.copyId());
}
/** Odczytuje zasób po odcisku treści; {@code null}, gdy nie ma go w katalogu. */
public Clip read(String sha1) throws Exception {
Location location = locate(sha1);
if (location == null) {
return null;
}
IFileSystem fs = AssetSourceDispatcher.openAssets(new File(location.container()));
byte[] bytes;
try (InputStream in = fs.open(location.pathInContainer())) {
if (location.offset() < 0) {
bytes = in.readAllBytes();
} else {
skipFully(in, location.offset());
bytes = in.readNBytes(location.size());
}
}
return new Clip(sha1, location.name(), contentType(location.format(), bytes), bytes,
location.offset() >= 0);
}
/**
* Znajduje odcisk zasobu o danej nazwie w obrębie kopii — najpierw wśród plików,
* potem w archiwach. Tak odwołanie ze skryptu ({@code FILENAME=BUREKTOR_M030.WAV})
* staje się czymś, co da się odsłuchać.
*/
public String resolve(long copyId, String name) throws Exception {
String needle = name.toLowerCase(Locale.ROOT);
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT blob_sha1 FROM file
WHERE copy_id = ? AND basename = ?
LIMIT 1
""")) {
st.setLong(1, copyId);
st.setString(2, needle);
try (ResultSet rs = st.executeQuery()) {
if (rs.next()) {
return rs.getString(1);
}
}
}
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT ae.sha1 FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
WHERE f.copy_id = ? AND ae.name = ?
LIMIT 1
""")) {
st.setLong(1, copyId);
st.setString(2, needle);
try (ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getString(1) : null;
}
}
}
private Location locate(String sha1) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
f.path AS canonical, b.format
FROM file f
JOIN copy c ON c.id = f.copy_id
JOIN blob b ON b.sha1 = f.blob_sha1
WHERE f.blob_sha1 = ? AND c.status <> 'unreadable'
LIMIT 1
""")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
if (rs.next()) {
String canonical = rs.getString("canonical");
int slash = canonical.lastIndexOf('/');
return new Location(rs.getString("container"), rs.getString("in_container"),
slash < 0 ? canonical : canonical.substring(slash + 1),
rs.getString("format"), -1, 0);
}
}
}
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
ae.name, ae.format, ae.offset, ae.size
FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
JOIN copy c ON c.id = f.copy_id
WHERE ae.sha1 = ? AND c.status <> 'unreadable'
LIMIT 1
""")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
if (rs.next()) {
return new Location(rs.getString("container"), rs.getString("in_container"),
rs.getString("name"), rs.getString("format"),
rs.getLong("offset"), rs.getInt("size"));
}
}
}
return null;
}
/**
* Rozszerzenie bywa mylące — wpisy w {@code wav.snd} nazywają się {@code *.wav},
* a niosą Vorbisa. Rozstrzygają magiczne bajty.
*/
private static String contentType(String format, byte[] bytes) {
if (bytes.length >= 4) {
if (bytes[0] == 'O' && bytes[1] == 'g' && bytes[2] == 'g' && bytes[3] == 'S') {
return "audio/ogg";
}
if (bytes[0] == 'R' && bytes[1] == 'I' && bytes[2] == 'F' && bytes[3] == 'F') {
return "audio/wav";
}
}
if (format == null) {
return "application/octet-stream";
}
return switch (format.toUpperCase(Locale.ROOT)) {
case "WAV" -> "audio/wav";
case "OGG" -> "audio/ogg";
case "MPG" -> "video/mpeg";
default -> "application/octet-stream";
};
}
private static void skipFully(InputStream in, long count) throws java.io.IOException {
long remaining = count;
while (remaining > 0) {
long skipped = in.skip(remaining);
if (skipped <= 0) {
if (in.read() < 0) {
throw new java.io.EOFException("kontener krótszy niż pozycja wpisu");
}
remaining--;
} else {
remaining -= skipped;
}
}
}
}
@@ -0,0 +1,361 @@
package pl.genschu.rexcatalog.media;
import pl.genschu.bloomooemulator.encoding.CLZW2Compression;
import pl.genschu.bloomooemulator.encoding.CRLECompression;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
/**
* Dekoduje grafikę Aidem — obrazy {@code PIK\0} i animacje {@code NVP\0} —
* do zwykłych obrazów Javy, bez OpenGL-a.
*
* <p>Rozpakowanie bierzemy z {@code :core} ({@link CLZW2Compression},
* {@link CRLECompression}), bo to najtrudniejsza część i nie ma sensu jej
* powielać. Nagłówki czytamy tutaj, i to nie z wyboru: {@code ImageLoader}
* trzyma parser w metodzie prywatnej, a {@code AnimoLoader} — mimo że jego
* sygnatura wygląda na niezależną od grafiki — buduje obiekty {@code Image},
* których konstruktor od razu tworzy {@code Texture}. W procesie bez kontekstu
* OpenGL-a jedno i drugie jest nie do użycia. Układ pól odwzorowuje te loadery
* jeden do jednego, żeby katalog i emulator czytały te same bajty tak samo.
*/
public final class PikGraphics {
private static final Charset LATIN1 = StandardCharsets.ISO_8859_1;
/** Rodzaje kompresji, tak jak numeruje je {@code Image.CompressionTypes}. */
private static final int NONE = 0;
private static final int CLZW = 2;
private static final int CRLE_IN_CLZW = 3;
private static final int CRLE = 4;
private static final int JPEG = 5;
private PikGraphics() {
}
public record Picture(int width, int height, int colorDepth, int compression,
int offsetX, int offsetY, BufferedImage image) {
}
public record Frame(String name, int width, int height, int offsetX, int offsetY,
int compression, BufferedImage image) {
}
/** Zdarzenie animacji: nazwana sekwencja odsyłająca do numerów klatek. */
public record AnimEvent(String name, int framesCount, int loopStart, int loopEnd,
List<Integer> frames) {
}
public record Animation(int colorDepth, int fps, int opacity, int maxWidth, int maxHeight,
String author, String description, List<Frame> frames,
List<AnimEvent> events) {
}
// ---------- IMG ----------
public static Picture readImage(byte[] data) throws IOException {
Reader reader = new Reader(data);
reader.expectMagic("PIK", "obraz");
int width = reader.i32();
int height = reader.i32();
int colorDepth = reader.i32();
int imageSize = reader.i32();
reader.skip(4);
int compression = reader.i32();
int alphaSize = reader.i32();
int offsetX = reader.i32();
int offsetY = reader.i32();
byte[] imageData = reader.bytes(imageSize);
byte[] alphaData = alphaSize > 0 ? reader.bytes(alphaSize) : null;
// ImageLoader z :core traktuje w plikach IMG kompresję 4 jak jej brak.
// To quirk samego IMG — w klatkach animacji ta sama czwórka oznacza
// prawdziwe CRLE i AnimoLoader przekazuje ją dalej bez zmiany.
BufferedImage image = compose(width, height, colorDepth,
compression == CRLE ? NONE : compression, imageData, alphaData);
return new Picture(width, height, colorDepth, compression, offsetX, offsetY, image);
}
// ---------- ANN ----------
public static Animation readAnimation(byte[] data) throws IOException {
Reader reader = new Reader(data);
reader.expectMagic("NVP", "animacja");
int imagesCount = reader.u16();
int colorDepth = reader.u16();
int eventsCount = reader.u16();
reader.skip(13);
int fps = reader.i32();
reader.skip(4);
int opacity = reader.u8();
reader.skip(12);
String author = reader.lengthPrefixed();
String description = reader.lengthPrefixed();
List<AnimEvent> events = new ArrayList<>();
for (int i = 0; i < eventsCount; i++) {
events.add(readEvent(reader));
}
// najpierw wszystkie nagłówki klatek, dopiero potem ich dane — tak samo
// jak w AnimoLoader, bo tak leżą w pliku
int[][] meta = new int[imagesCount][7];
String[] names = new String[imagesCount];
for (int i = 0; i < imagesCount; i++) {
meta[i][0] = reader.u16(); // szerokość
meta[i][1] = reader.u16(); // wysokość
meta[i][2] = reader.i16(); // przesunięcie X
meta[i][3] = reader.i16(); // przesunięcie Y
meta[i][4] = reader.u16(); // kompresja
meta[i][5] = reader.i32(); // rozmiar danych
reader.skip(14);
meta[i][6] = reader.i32(); // rozmiar warstwy alfa
names[i] = reader.fixedString(20);
}
List<Frame> frames = new ArrayList<>();
int maxWidth = 0;
int maxHeight = 0;
for (int i = 0; i < imagesCount; i++) {
byte[] imageData = reader.bytes(meta[i][5]);
byte[] alphaData = reader.bytes(meta[i][6]);
BufferedImage image = compose(meta[i][0], meta[i][1], colorDepth, meta[i][4],
imageData, alphaData.length == 0 ? null : alphaData);
frames.add(new Frame(names[i], meta[i][0], meta[i][1], meta[i][2], meta[i][3],
meta[i][4], image));
maxWidth = Math.max(maxWidth, meta[i][0]);
maxHeight = Math.max(maxHeight, meta[i][1]);
}
return new Animation(colorDepth, fps, opacity, maxWidth, maxHeight, author, description,
frames, events);
}
private static AnimEvent readEvent(Reader reader) throws IOException {
String name = reader.fixedString(32).toUpperCase(java.util.Locale.ROOT);
int framesCount = reader.u16();
reader.skip(4);
int loopStart = reader.u16();
int loopEnd = reader.u16();
reader.skip(4); // liczba powtórzeń i licznik
reader.skip(4);
reader.skip(4); // flagi
reader.u8(); // krycie
reader.skip(12);
List<Integer> frames = new ArrayList<>(framesCount);
for (int i = 0; i < framesCount; i++) {
frames.add(reader.u16());
}
for (int i = 0; i < framesCount; i++) {
skipFrameData(reader);
}
return new AnimEvent(name, framesCount, loopStart, loopEnd, frames);
}
/** Dane klatki opisują ruch i dźwięk — do podglądu nam niepotrzebne, ale trzeba je przejść. */
private static void skipFrameData(Reader reader) throws IOException {
reader.skip(4); // bajty startowe
reader.skip(4);
reader.i16(); // przesunięcie X
reader.i16(); // przesunięcie Y
reader.skip(4);
int sfxRandomSeed = reader.i32();
reader.skip(4);
reader.u8(); // krycie
reader.skip(5);
reader.bytes(reader.i32()); // nazwa
if (sfxRandomSeed > 0) {
reader.bytes(reader.i32()); // opis efektu dźwiękowego
}
}
// ---------- piksele ----------
/**
* Składa obraz z warstwy koloru i osobnej warstwy przezroczystości.
*
* <p>Kolor to dwa bajty na piksel (RGB565 albo RGB555, zależnie od głębi),
* alfa — jeden bajt. Odwzorowanie kanałów jest takie samo jak
* w {@code Image.convertRgbToRgb888}.
*/
static BufferedImage compose(int width, int height, int colorDepth, int compression,
byte[] imageData, byte[] alphaData) throws IOException {
if (width <= 0 || height <= 0) {
throw new IOException("nieprawdopodobne wymiary: " + width + "x" + height);
}
if (compression == JPEG) {
return composeJpeg(imageData, alphaData, width, height);
}
byte[] pixels = imageData;
byte[] alpha = alphaData;
if (compression == CRLE) {
pixels = CRLECompression.decodeCRLE(pixels, 2);
if (alpha != null) {
alpha = CRLECompression.decodeCRLE(alpha);
}
}
if (compression == CLZW || compression == CRLE_IN_CLZW) {
pixels = CLZW2Compression.decompress(pixels);
if (alpha != null) {
alpha = CLZW2Compression.decompress(alpha);
}
if (compression == CRLE_IN_CLZW) {
pixels = CRLECompression.decodeCRLE(pixels, 2);
if (alpha != null) {
alpha = CRLECompression.decodeCRLE(alpha);
}
}
}
BufferedImage out = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
int count = width * height;
for (int i = 0; i < count; i++) {
int at = i * 2;
if (at + 1 >= pixels.length) {
break;
}
int packed = (pixels[at] & 0xFF) | ((pixels[at + 1] & 0xFF) << 8);
int a = alpha != null && i < alpha.length ? alpha[i] & 0xFF : 255;
out.setRGB(i % width, i / width, (a << 24) | rgb(packed, colorDepth));
}
return out;
}
private static BufferedImage composeJpeg(byte[] imageData, byte[] alphaData,
int width, int height) throws IOException {
BufferedImage jpeg = ImageIO.read(new ByteArrayInputStream(imageData));
if (jpeg == null) {
throw new IOException("nieczytelny strumień JPEG");
}
byte[] alpha = alphaData == null ? null : CLZW2Compression.decompress(alphaData);
BufferedImage out = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int rgb = x < jpeg.getWidth() && y < jpeg.getHeight()
? jpeg.getRGB(x, y) & 0xFFFFFF : 0;
int index = y * width + x;
int a = alpha != null && index < alpha.length ? alpha[index] & 0xFF : 255;
out.setRGB(x, y, (a << 24) | rgb);
}
}
return out;
}
/** RGB565 albo RGB555 rozciągnięte na pełne osiem bitów na kanał. */
private static int rgb(int packed, int colorDepth) {
int r;
int g;
int b;
if (colorDepth == 15) {
r = (packed >> 10) & 0x1F;
g = (packed >> 5) & 0x1F;
b = packed & 0x1F;
r = (r << 3) | (r >> 2);
g = (g << 3) | (g >> 2);
b = (b << 3) | (b >> 2);
} else {
r = (packed >> 11) & 0x1F;
g = (packed >> 5) & 0x3F;
b = packed & 0x1F;
r = (r << 3) | (r >> 2);
g = (g << 2) | (g >> 4);
b = (b << 3) | (b >> 2);
}
return (r << 16) | (g << 8) | b;
}
// ---------- odczyt ----------
/** Czytnik po tablicy bajtów; strumieniowy z {@code :core} nie umie cofać. */
private static final class Reader {
private final byte[] data;
private int at;
Reader(byte[] data) {
this.data = data;
}
int u8() throws IOException {
need(1);
return data[at++] & 0xFF;
}
int u16() throws IOException {
need(2);
int value = (data[at] & 0xFF) | ((data[at + 1] & 0xFF) << 8);
at += 2;
return value;
}
int i16() throws IOException {
return (short) u16();
}
int i32() throws IOException {
need(4);
int value = (data[at] & 0xFF) | ((data[at + 1] & 0xFF) << 8)
| ((data[at + 2] & 0xFF) << 16) | ((data[at + 3] & 0xFF) << 24);
at += 4;
return value;
}
void skip(int count) throws IOException {
need(count);
at += count;
}
byte[] bytes(int count) throws IOException {
if (count < 0) {
throw new IOException("ujemna długość bloku: " + count);
}
need(count);
byte[] out = new byte[count];
System.arraycopy(data, at, out, 0, count);
at += count;
return out;
}
/** Magiczne bajty kończy zero, więc porównujemy je wprost, a nie jako tekst. */
void expectMagic(String magic, String what) throws IOException {
byte[] raw = bytes(4);
boolean ok = raw[3] == 0;
for (int i = 0; ok && i < 3; i++) {
ok = raw[i] == (byte) magic.charAt(i);
}
if (!ok) {
throw new IOException("to nie jest " + what + " " + magic
+ " (nagłówek: " + new String(raw, LATIN1).replace("\0", "\\0") + ")");
}
}
String fixedString(int count) throws IOException {
byte[] raw = bytes(count);
int end = 0;
while (end < raw.length && raw[end] != 0) {
end++;
}
return new String(raw, 0, end, LATIN1);
}
String lengthPrefixed() throws IOException {
return new String(bytes(i32()), LATIN1).replace("\0", "");
}
private void need(int count) throws IOException {
if (at + count > data.length) {
throw new IOException("plik urwany na pozycji " + at);
}
}
}
}
@@ -0,0 +1,169 @@
package pl.genschu.rexcatalog.media;
import java.io.EOFException;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
/**
* Archiwum {@code .snd} — płaski kontener na kwestie mówione.
*
* <p>Używa go <i>Reksio i Kapitan Nemo</i>: zamiast katalogu z tysiącami plików
* leży jeden {@code wavs/wav.snd} na 79 MB, a w środku 3274 wpisy zakodowane
* w Ogg Vorbisie (na płycie leży {@code oggenc.exe}, więc zgadza się i narzędzie).
* Nazwy wpisów kończą się na {@code .wav} mimo vorbisowej zawartości.
*
* <p>Układ wpisu, potwierdzony przejściem całego pliku — parser dochodzi
* dokładnie do jego końca, bajt w bajt:
* <pre>
* u32 znacznik (zawsze 0 w zbadanych płytach)
* u32 długość payloadu
* u32 długość nazwy razem z bajtem zerowym
* u8[] nazwa w CP1250, zakończona zerem
* u8[] payload
* </pre>
*
* <p>{@code :core} tego formatu nie zna, więc parser jest tutaj, a nie w emulatorze.
*/
public final class SndArchive {
private static final Charset CP1250 = Charset.forName("windows-1250");
/** Górny limit długości nazwy — chroni przed próbą alokacji na śmieciach. */
private static final int MAX_NAME = 512;
private SndArchive() {
}
/**
* Jeden wpis archiwum. {@code offset} wskazuje na payload, nie na nagłówek,
* więc wystarcza do odczytania zasobu bez ponownego parsowania kontenera.
*/
public record Entry(String name, String nameRaw, long offset, int size) {
}
/** Odbiorca wpisów; {@code payload} jest null, gdy przechodzimy tylko po nagłówkach. */
public interface Visitor {
void entry(Entry entry, byte[] payload) throws IOException;
}
/**
* Zwraca same wpisy, przeskakując payloady — do odnalezienia zasobu,
* gdy trzeba go tylko podać przeglądarce.
*/
public static List<Entry> index(InputStream in) throws IOException {
List<Entry> entries = new ArrayList<>();
walk(in, false, (entry, payload) -> entries.add(entry));
return entries;
}
/** Przechodzi archiwum wczytując payloady — do liczenia odcisków i nagłówków. */
public static void walk(InputStream in, Visitor visitor) throws IOException {
walk(in, true, visitor);
}
private static void walk(InputStream in, boolean withPayload, Visitor visitor)
throws IOException {
long position = 0;
byte[] header = new byte[12];
while (true) {
int read = readFully(in, header, 0, 12);
if (read == 0) {
return;
}
if (read < 12) {
throw new IOException("obcięty nagłówek wpisu na pozycji " + position);
}
position += 12;
long payloadSize = u32(header, 4);
long nameLength = u32(header, 8);
if (nameLength <= 0 || nameLength > MAX_NAME) {
throw new IOException("nieprawdopodobna długość nazwy (" + nameLength
+ ") na pozycji " + (position - 12));
}
byte[] rawName = new byte[(int) nameLength];
if (readFully(in, rawName, 0, rawName.length) < rawName.length) {
throw new IOException("obcięta nazwa wpisu na pozycji " + position);
}
position += nameLength;
String nameRaw = new String(rawName, CP1250).replace("\0", "");
Entry entry = new Entry(nameRaw.toLowerCase(java.util.Locale.ROOT), nameRaw,
position, (int) payloadSize);
if (withPayload) {
byte[] payload = new byte[(int) payloadSize];
if (readFully(in, payload, 0, payload.length) < payload.length) {
throw new IOException("obcięty payload wpisu " + nameRaw);
}
visitor.entry(entry, payload);
} else {
visitor.entry(entry, null);
skipFully(in, payloadSize);
}
position += payloadSize;
}
}
/** Czy strumień w ogóle wygląda na archiwum — po pierwszym wpisie. */
public static boolean looksLikeArchive(byte[] head) {
if (head.length < 17) {
return false;
}
long nameLength = u32(head, 8);
if (nameLength < 2 || nameLength > MAX_NAME || 12 + nameLength > head.length) {
return false;
}
// nazwa musi być zakończona zerem i mieścić się w drukowalnym ASCII
if (head[(int) (12 + nameLength - 1)] != 0) {
return false;
}
for (int i = 12; i < 12 + nameLength - 1; i++) {
if (head[i] < 0x20) {
return false;
}
}
return true;
}
private static long u32(byte[] buffer, int offset) {
return (buffer[offset] & 0xFFL)
| ((buffer[offset + 1] & 0xFFL) << 8)
| ((buffer[offset + 2] & 0xFFL) << 16)
| ((buffer[offset + 3] & 0xFFL) << 24);
}
private static int readFully(InputStream in, byte[] buffer, int offset, int length)
throws IOException {
int total = 0;
while (total < length) {
int read = in.read(buffer, offset + total, length - total);
if (read < 0) {
break;
}
total += read;
}
return total;
}
private static void skipFully(InputStream in, long count) throws IOException {
long remaining = count;
while (remaining > 0) {
long skipped = in.skip(remaining);
if (skipped <= 0) {
// skip() na niektórych strumieniach nie rusza do przodu na końcu bufora
if (in.read() < 0) {
throw new EOFException("payload krótszy niż deklarowany o " + remaining + " B");
}
remaining--;
} else {
remaining -= skipped;
}
}
}
}
@@ -0,0 +1,213 @@
package pl.genschu.rexcatalog.script;
import pl.genschu.rexcatalog.db.Database;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.LinkedHashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Wyciąga ze skryptów odwołania do zasobów i zapisuje je w {@code script_ref}.
*
* <p>Skrypty Piklib są liniowe — {@code OBIEKT:POLE=WARTOŚĆ} — również dla
* zachowań, bo dekoder zwija całe {@code CODE={...}} do jednej linii. Dzięki temu
* ten sam przebieg łapie i deklaracje ({@code SNDQUESTION:FILENAME=BUREKTOR_M030.WAV}),
* i nazwy wplecione w kod ({@code DBDIALOG^LOAD("$COMMON\DIALOGS.DTA")}), a obiekt
* przypisuje się poprawnie w obu przypadkach.
*
* <p>Odwołania zapisujemy niezależnie od tego, czy plik da się znaleźć: nazwa,
* której w wydaniu nie ma, jest sama w sobie informacją.
*/
public final class RefIndexer {
public static final String VERSION = "refs-1";
/** Rozszerzenia, które w tej kolekcji są zasobami, a nie przypadkowym tekstem. */
private static final Set<String> EXTENSIONS = Set.of(
"WAV", "OGG", "SND", "IMG", "ANN", "SEQ", "ARR", "DTA", "SEK", "FNT",
"CNV", "CLASS", "DEF", "AVI", "MPG", "TGA", "DDS", "BMP", "SCN", "LUA",
"KIMODEL", "KIANIM");
/**
* Nazwa pola bywa zakończona daszkiem i numerem — {@code VARIWST:ONCHANGED^2=...}
* to kolejny wariant tego samego zdarzenia. Bez daszka w zbiorze znaków takie
* linie w ogóle nie trafiałyby do przeszukania, a siedzą w nich prawdziwe
* odwołania, choćby {@code SETBACKGROUND("PARYZ\BKG1.IMG")}.
*/
private static final Pattern ASSIGNMENT =
Pattern.compile("^([A-Za-z0-9_]+):([A-Za-z0-9_^]+)=(.*)$");
/**
* Nazwa nie może być doklejona plusem: {@code "$COMMON\"+ZMIENNA+"_DEF.DTA"}
* buduje ścieżkę w czasie działania, więc {@code _DEF.DTA} nie jest nazwą pliku.
* Podkreślenie na początku samo w sobie niczego nie przesądza — {@code SNDX:FILENAME=
* _WZIECIE_JABLKA.WAV} to prawdziwy plik.
*/
private static final Pattern FILENAME = Pattern.compile(
// dopasowanie musi zaczynać się na granicy tokenu: bez tego zablokowanie
// startu na podkreśleniu sprawia, że silnik rusza znak dalej i wycina
// HIST0.ARR ze środka +"_HIST0.ARR"
"(?<![A-Za-z0-9_$])(?<!\\+)(?<!\\+\")"
+ "[A-Za-z0-9_$][A-Za-z0-9_$\\\\/.-]*\\.("
+ String.join("|", EXTENSIONS) + ")\\b",
Pattern.CASE_INSENSITIVE);
private final Database db;
private final Path dataDir;
public RefIndexer(Database db, Path dataDir) {
this.db = db;
this.dataDir = dataDir;
}
public record Stats(int scripts, int refs, int failed) {
}
public Stats run(boolean force) throws Exception {
Map<String, String> pending = pendingWork(force);
if (pending.isEmpty()) {
return new Stats(0, 0, 0);
}
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
int scripts = 0;
int refs = 0;
int failed = 0;
try (PreparedStatement delete = conn.prepareStatement(
"DELETE FROM script_ref WHERE script_sha1 = ?");
PreparedStatement insert = conn.prepareStatement("""
INSERT INTO script_ref (script_sha1, object, field, value, name, format, line)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(script_sha1, object, field, value) DO NOTHING
""");
// ślad przebiegu — bez niego skrypty bez odwołań (a takich jest ponad sto)
// przechodziłyby przez parser przy każdym uruchomieniu
PreparedStatement mark = conn.prepareStatement("""
INSERT INTO media (sha1, kind, note, tool_version, probed_at)
VALUES (?, 'script', ?, ?, ?)
ON CONFLICT(sha1) DO UPDATE SET
kind = excluded.kind, note = excluded.note,
tool_version = excluded.tool_version, probed_at = excluded.probed_at
""")) {
for (Map.Entry<String, String> entry : pending.entrySet()) {
String sha1 = entry.getKey();
try {
String body = Files.readString(dataDir.resolve(entry.getValue()),
StandardCharsets.UTF_8);
delete.setString(1, sha1);
delete.executeUpdate();
int written = extract(sha1, body, insert);
refs += written;
scripts++;
mark.setString(1, sha1);
mark.setString(2, written + " odwołań do zasobów");
mark.setString(3, VERSION);
mark.setString(4, java.time.Instant.now().toString());
mark.executeUpdate();
} catch (Exception e) {
System.err.printf(" ! %s: %s%n", sha1, e.getMessage());
failed++;
}
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return new Stats(scripts, refs, failed);
}
private int extract(String sha1, String body, PreparedStatement insert) throws Exception {
String[] lines = body.split("\n", -1);
int written = 0;
for (int number = 0; number < lines.length; number++) {
Matcher assignment = ASSIGNMENT.matcher(lines[number].trim());
if (!assignment.matches()) {
continue;
}
String object = assignment.group(1).toUpperCase(Locale.ROOT);
String field = assignment.group(2).toUpperCase(Locale.ROOT);
String rest = assignment.group(3);
Matcher filename = FILENAME.matcher(rest);
while (filename.find()) {
String value = filename.group();
insert.setString(1, sha1);
insert.setString(2, object);
insert.setString(3, field);
insert.setString(4, value);
insert.setString(5, basename(value));
insert.setString(6, filename.group(1).toUpperCase(Locale.ROOT));
insert.setInt(7, number + 1);
written += insert.executeUpdate();
}
}
return written;
}
/** Ścieżki w skryptach bywają z makrem katalogu ({@code $COMMON\...}) — liczy się nazwa. */
static String basename(String value) {
String path = value.replace('\\', '/');
int slash = path.lastIndexOf('/');
return (slash < 0 ? path : path.substring(slash + 1)).toLowerCase(Locale.ROOT);
}
/** Skrypty z cache'u, dla których nie mamy jeszcze odwołań. */
private Map<String, String> pendingWork(boolean force) throws Exception {
String sql = """
SELECT a.blob_sha1, a.path
FROM artifact a
LEFT JOIN media m ON m.sha1 = a.blob_sha1 AND m.kind = 'script'
WHERE a.kind = 'script'
""" + (force ? "" : """
AND (m.sha1 IS NULL OR m.tool_version <> ?)
""");
Map<String, String> out = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
if (!force) {
st.setString(1, VERSION);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
out.put(rs.getString("blob_sha1"), rs.getString("path"));
}
}
}
return out;
}
public int pendingCount() throws Exception {
return pendingWork(false).size();
}
/** Czyści odwołania skryptów, których nie ma już w cache'u. */
public int prune() throws Exception {
try (Statement st = db.connection().createStatement()) {
return st.executeUpdate("""
DELETE FROM script_ref
WHERE script_sha1 NOT IN (SELECT blob_sha1 FROM artifact WHERE kind = 'script')
""");
}
}
}
@@ -0,0 +1,213 @@
package pl.genschu.rexcatalog.script;
import pl.genschu.bloomooemulator.encoding.ScriptDecypher;
import pl.genschu.bloomooemulator.engine.filesystem.AssetSourceDispatcher;
import pl.genschu.bloomooemulator.engine.filesystem.IFileSystem;
import pl.genschu.rexcatalog.db.Database;
import java.io.File;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Odszyfrowuje skrypty przez {@link ScriptDecypher} z {@code :core} i buduje z nich
* cache adresowany treścią plus indeks pełnotekstowy.
*
* <p>Jednostką pracy jest blob, nie plik: identyczny skrypt leżący w pięciu obrazach
* odszyfrowujemy raz. Powtórne uruchomienie przerabia tylko to, czego brakuje albo
* co zostało zrobione inną wersją {@code :core} — {@code artifact.tool_version}
* jest kluczem unieważnienia.
*/
public final class ScriptDecoder {
/** Formaty, które {@code ScriptDecypher} potrafi rozwinąć w tekst. */
private static final Set<String> SCRIPT_FORMATS = Set.of("CNV", "CLASS", "DEF", "SCRIPT");
private static final String KIND = "script";
private final Database db;
private final Path dataDir;
private final String coreVersion;
public ScriptDecoder(Database db, Path dataDir, String coreVersion) {
this.db = db;
this.dataDir = dataDir;
this.coreVersion = coreVersion;
}
public record Stats(int decoded, int skipped, int failed) {
}
/** Jeden skrypt do odszyfrowania: gdzie leży w kontenerze i jaka jest jego treść. */
private record Pending(String pathInContainer, String sha1) {
}
public Stats run(boolean force) throws Exception {
Map<String, List<Pending>> byContainer = pendingWork(force);
if (byContainer.isEmpty()) {
return new Stats(0, 0, 0);
}
Connection conn = db.connection();
boolean previousAutoCommit = conn.getAutoCommit();
conn.setAutoCommit(false);
int decoded = 0;
int failed = 0;
int skipped = 0;
Set<String> done = new HashSet<>();
try (PreparedStatement artifactStmt = conn.prepareStatement("""
INSERT INTO artifact (blob_sha1, kind, path, tool_version, created_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(blob_sha1, kind) DO UPDATE SET
path = excluded.path, tool_version = excluded.tool_version,
created_at = excluded.created_at
""");
PreparedStatement ftsDelete = conn.prepareStatement(
"DELETE FROM script_fts WHERE blob_sha1 = ?");
PreparedStatement ftsInsert = conn.prepareStatement(
"INSERT INTO script_fts (body, blob_sha1) VALUES (?, ?)")) {
for (Map.Entry<String, List<Pending>> entry : byContainer.entrySet()) {
File container = new File(entry.getKey());
IFileSystem fs;
try {
fs = AssetSourceDispatcher.openAssets(container);
} catch (Exception e) {
System.err.printf(" ! nie otwieram %s: %s%n",
container.getName(), e.getMessage());
failed += entry.getValue().size();
continue;
}
for (Pending pending : entry.getValue()) {
// ten sam blob w kolejnym obrazie — treść jest ta sama z definicji
if (!done.add(pending.sha1())) {
skipped++;
continue;
}
try {
byte[] raw = readAll(fs, pending.pathInContainer());
String text = ScriptDecypher.decypherIfNeeded(raw);
String relative = cachePath(pending.sha1());
Path target = dataDir.resolve(relative);
Files.createDirectories(target.getParent());
Files.writeString(target, text, StandardCharsets.UTF_8);
artifactStmt.setString(1, pending.sha1());
artifactStmt.setString(2, KIND);
artifactStmt.setString(3, relative);
artifactStmt.setString(4, coreVersion);
artifactStmt.setString(5, Instant.now().toString());
artifactStmt.executeUpdate();
ftsDelete.setString(1, pending.sha1());
ftsDelete.executeUpdate();
ftsInsert.setString(1, text);
ftsInsert.setString(2, pending.sha1());
ftsInsert.executeUpdate();
decoded++;
} catch (Exception e) {
System.err.printf(" ! %s (%s): %s%n", pending.pathInContainer(),
container.getName(), e.getMessage());
failed++;
}
}
}
conn.commit();
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.setAutoCommit(previousAutoCommit);
}
return new Stats(decoded, skipped, failed);
}
/**
* Zbiera skrypty wymagające przerobienia, pogrupowane po kontenerze — żeby
* każdy obraz otwierać raz, a nie raz na plik.
*/
private Map<String, List<Pending>> pendingWork(boolean force) throws Exception {
String formats = String.join(",", java.util.Collections.nCopies(SCRIPT_FORMATS.size(), "?"));
String sql = """
SELECT c.path AS container, COALESCE(f.path_raw, f.path) AS in_container,
f.blob_sha1
FROM copy c
JOIN file f ON f.copy_id = c.id
JOIN blob b ON b.sha1 = f.blob_sha1
LEFT JOIN artifact a ON a.blob_sha1 = b.sha1 AND a.kind = 'script'
WHERE c.status <> 'unreadable'
AND b.format IN (%s)
""".formatted(formats)
+ (force ? "" : " AND (a.blob_sha1 IS NULL OR a.tool_version <> ?)\n")
+ "ORDER BY c.id";
Map<String, List<Pending>> byContainer = new LinkedHashMap<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
int index = 1;
for (String format : SCRIPT_FORMATS) {
st.setString(index++, format);
}
if (!force) {
st.setString(index, coreVersion);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
byContainer.computeIfAbsent(rs.getString("container"), k -> new ArrayList<>())
.add(new Pending(rs.getString("in_container"),
rs.getString("blob_sha1")));
}
}
}
return byContainer;
}
/** Rozgałęzienie po dwóch pierwszych znakach — bez tego jeden katalog z tysiącami plików. */
private static String cachePath(String sha1) {
return "artifacts/" + KIND + "/" + sha1.substring(0, 2) + "/" + sha1 + ".txt";
}
private static byte[] readAll(IFileSystem fs, String path) throws Exception {
try (InputStream in = fs.open(path)) {
return in.readAllBytes();
}
}
/** Ile skryptów czeka na przerobienie — do komunikatu przed startem. */
public int pendingCount() throws Exception {
int total = 0;
for (List<Pending> list : pendingWork(false).values()) {
total += list.size();
}
return total;
}
/** Usuwa z cache'u i indeksu wszystko, czego nie ma już w bazie blobów. */
public int prune() throws Exception {
try (Statement st = db.connection().createStatement()) {
return st.executeUpdate("""
DELETE FROM artifact
WHERE kind = 'script'
AND blob_sha1 NOT IN (SELECT blob_sha1 FROM file)
""");
}
}
}
@@ -0,0 +1,136 @@
package pl.genschu.rexcatalog.script;
import pl.genschu.rexcatalog.db.Database;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
/**
* Odpytywanie indeksu skryptów. Wynik jest zawsze wiązany z kopiami, w których
* dany skrypt występuje — inaczej trafienie w blob nic nie mówi o tym, której gry dotyczy.
*/
public final class ScriptSearch {
private final Database db;
private final Path dataDir;
public ScriptSearch(Database db, Path dataDir) {
this.db = db;
this.dataDir = dataDir;
}
public record Hit(String sha1, String snippet, List<String> paths, List<String> games) {
}
public record Result(String effectiveQuery, boolean quoted, List<Hit> hits) {
}
/**
* Szuka najpierw dosłownie, żeby nie odbierać składni FTS5 (operatory, {@code *},
* {@code NEAR}). Dopiero gdy zapytanie się nie parsuje — a tak kończy każde
* wyrażenie ze skryptu, choćby {@code TYPE=EPISODE} — powtarza je jako frazę.
*/
public Result find(String query, int limit) throws Exception {
try {
return new Result(query, false, search(query, limit));
} catch (java.sql.SQLException e) {
String phrase = '"' + query.replace("\"", "\"\"") + '"';
return new Result(phrase, true, search(phrase, limit));
}
}
private List<Hit> search(String query, int limit) throws Exception {
List<Hit> hits = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT blob_sha1, snippet(script_fts, 0, '»', '«', ' … ', 12) AS snip
FROM script_fts
WHERE script_fts MATCH ?
ORDER BY rank
LIMIT ?
""")) {
st.setString(1, query);
st.setInt(2, limit);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String sha1 = rs.getString("blob_sha1");
hits.add(new Hit(sha1, rs.getString("snip"),
pathsOf(sha1), gamesOf(sha1)));
}
}
}
return hits;
}
public long indexedCount() throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT COUNT(*) FROM script_fts");
ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getLong(1) : 0;
}
}
private List<String> pathsOf(String sha1) throws Exception {
return oneColumn("SELECT DISTINCT COALESCE(path_raw, path) FROM file "
+ "WHERE blob_sha1 = ? ORDER BY 1", sha1);
}
private List<String> gamesOf(String sha1) throws Exception {
return oneColumn("SELECT DISTINCT c.display_name FROM file f "
+ "JOIN copy c ON c.id = f.copy_id WHERE f.blob_sha1 = ? ORDER BY 1", sha1);
}
private List<String> oneColumn(String sql, String argument) throws Exception {
List<String> out = new ArrayList<>();
try (PreparedStatement st = db.connection().prepareStatement(sql)) {
st.setString(1, argument);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
out.add(rs.getString(1));
}
}
}
return out;
}
/**
* Zamienia to, co podał użytkownik, na blob: albo pełny SHA-1, albo fragment
* ścieżki. Niejednoznaczny fragment nie jest zgadywany — wołający dostaje
* wszystkich kandydatów i decyduje sam.
*/
public List<String> resolve(String identifier) throws Exception {
if (identifier.matches("[0-9a-fA-F]{40}")) {
return List.of(identifier.toLowerCase(Locale.ROOT));
}
return oneColumn("SELECT DISTINCT blob_sha1 FROM file WHERE path LIKE ? ORDER BY 1",
"%" + identifier.toLowerCase(Locale.ROOT).replace('\\', '/') + "%");
}
/** Zwraca odszyfrowaną treść z cache'u albo {@code null}, jeśli blob nie był dekodowany. */
public String body(String sha1) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT path FROM artifact WHERE blob_sha1 = ? AND kind = 'script'")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
if (!rs.next()) {
return null;
}
Path file = dataDir.resolve(rs.getString("path"));
return Files.exists(file) ? Files.readString(file, StandardCharsets.UTF_8) : null;
}
}
}
public List<String> pathsFor(String sha1) throws Exception {
return pathsOf(sha1);
}
public List<String> gamesFor(String sha1) throws Exception {
return gamesOf(sha1);
}
}
@@ -0,0 +1,762 @@
package pl.genschu.rexcatalog.transcribe;
import pl.genschu.rexcatalog.db.Database;
import pl.genschu.rexcatalog.media.MediaIndex;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* Transkrypcja mowy przez whisper.cpp — uruchamiana ręcznie, nigdy sama z siebie.
*
* <p>Transkrypt jest <b>wygenerowany</b>, a nie odczytany z płyty, więc trafia do
* osobnej tabeli razem z nazwą modelu i narzędzia. Nie miesza się z metadanymi
* wydania ani z faktami detektorów: to hipoteza maszyny o tym, co słychać, a nie
* zapis z nośnika.
*
* <p>Kwestie są <b>sklejane w okna</b>, a nie podawane po jednej. Whisper koduje
* zawsze pełne trzydzieści sekund, niezależnie od tego, czy nagranie trwa dwie
* sekundy, czy dwadzieścia — a w tej kolekcji średnia kwestia ma 4,4 s. Podawane
* pojedynczo, dziewiętnaście godzin mowy kosztowałoby tyle, co sto trzydzieści
* pięć godzin ciągłego nagrania. Sklejone z odstępami ciszy mieszczą się po kilka
* w jednym oknie, a wynik rozcinamy z powrotem po znacznikach czasu.
*
* <p>Okno musi mieć jeden język, bo {@code -l} dotyczy całego wywołania.
*
* <p>Praca idzie w tle i da się ją przerwać w dowolnym momencie —
* to, co już policzone, zostaje w bazie. Dzięki temu można puścić kolekcję na noc
* i zatrzymać rano bez straty.
*/
public final class Transcriber {
/** Whisper przyjmuje 16 kHz mono; nagrania z płyt mają 22 050 Hz, więc idą przez ffmpeg. */
private static final int WHISPER_RATE = 16000;
/**
* Ile okien na jedno uruchomienie whisper.cpp. Każde uruchomienie wczytuje
* model od nowa, więc kilka okien naraz amortyzuje ten koszt; przerwanie
* działa dopiero między wsadami.
*/
private static final int BATCH = Integer.getInteger("catalog.whisper.batch", 8);
/**
* Ile dźwięku upychamy w jedno okno. Whisper i tak liczy trzydzieści sekund,
* ale zostawiamy zapas, żeby ostatnia kwestia nie wypadła poza okno.
*/
private static final int WINDOW_MS = Integer.getInteger("catalog.whisper.window", 27_000);
/**
* Cisza między sklejonymi kwestiami. Musi wystarczyć, żeby model potraktował
* je jako osobne wypowiedzi — przy zbyt krótkiej przerwie zlewa dwie kwestie
* w jeden segment i tekst trafia pod niewłaściwy plik.
*/
private static final int GAP_MS = Integer.getInteger("catalog.whisper.gap", 1000);
/** Wyłącznik sklejania — do porównania wyników z trybem plik po pliku. */
private static final boolean CONCAT =
!"false".equals(System.getProperty("catalog.whisper.concat"));
private static final int BYTES_PER_MS = WHISPER_RATE * 2 / 1000;
/**
* Wątki dla whisper.cpp. Zostawiamy jego własną wartość domyślną: na laptopie
* bez wentylatora podniesienie jej przyspiesza kosztem grzania i po chwili
* i tak wchodzi throttling.
*/
private static final String THREADS = System.getProperty("catalog.whisper.threads", "4");
/**
* Zdania, które Whisper dopisuje z siebie na ciszy i muzyce — podpisy ekip
* tworzących napisy, wyuczone z materiału treningowego. To nie jest zapis
* tego, co słychać, tylko wymysł modelu, więc do katalogu archiwalnego
* trafić nie może. Znaczniki w rodzaju {@code [muzyka]} zostawiamy: one
* opisują nagranie zgodnie z prawdą.
*/
private static final List<String> WYMYSLY = List.of(
"napisy stworzone przez", "napisy: ", "subtitles by", "amara.org",
"zdjęcia i napisy", "dziękuję za uwagę", "thanks for watching");
/** Nazwy, pod którymi whisper.cpp bywa instalowany. */
private static final List<String> BINARIES =
List.of("whisper-cli", "whisper-cpp", "whisper", "main");
private final Database db;
private final MediaIndex media;
private final Object lock;
private final AtomicBoolean cancelled = new AtomicBoolean();
private volatile Thread worker;
private volatile Progress progress = Progress.idle();
public Transcriber(Database db, Object lock) {
this.db = db;
this.media = new MediaIndex(db);
this.lock = lock;
}
/** Gdzie leży whisper.cpp i model; {@code ready} mówi, czy da się w ogóle ruszyć. */
public record Tool(String binary, String model, boolean ready, String note) {
}
public record Progress(String state, int total, int done, int failed, String current,
long elapsedMs, String note) {
static Progress idle() {
return new Progress("bezczynny", 0, 0, 0, null, 0, null);
}
}
// ---------- wykrywanie narzędzi ----------
/**
* Szuka whisper.cpp i modelu. Ścieżki da się wskazać wprost właściwościami
* {@code catalog.whisper.bin} i {@code catalog.whisper.model} — bo model
* to kilkaset megabajtów i każdy trzyma go gdzie indziej.
*/
public static Tool detect() {
String binary = System.getProperty("catalog.whisper.bin",
System.getenv("WHISPER_BIN"));
if (binary == null) {
binary = BINARIES.stream().map(Transcriber::onPath)
.filter(java.util.Objects::nonNull).findFirst().orElse(null);
}
String model = System.getProperty("catalog.whisper.model",
System.getenv("WHISPER_MODEL"));
if (model == null) {
model = findModel();
}
if (binary == null) {
return new Tool(null, model, false,
"nie znalazłem whisper.cpp — zainstaluj (brew install whisper-cpp) "
+ "albo wskaż przez -Dcatalog.whisper.bin=");
}
if (model == null) {
return new Tool(binary, null, false,
"nie znalazłem modelu — wskaż przez -Dcatalog.whisper.model=ścieżka.bin");
}
// w kontenerze model jest montowany z zewnątrz, więc ścieżka bywa wskazana,
// a pliku pod nią nie ma; lepiej powiedzieć to wprost niż wywalić się w trakcie
if (!Files.isReadable(Path.of(model))) {
return new Tool(binary, model, false,
"pod wskazaną ścieżką nie ma modelu: " + model);
}
if (!ffmpegAvailable()) {
return new Tool(binary, model, false,
"brakuje ffmpeg, a nagrania trzeba przepróbkować do 16 kHz mono");
}
return new Tool(binary, model, true, null);
}
private static String onPath(String name) {
String path = System.getenv("PATH");
if (path == null) {
return null;
}
for (String directory : path.split(java.io.File.pathSeparator)) {
Path candidate = Path.of(directory, name);
if (Files.isExecutable(candidate)) {
return candidate.toString();
}
}
return null;
}
private static String findModel() {
List<Path> places = List.of(
Path.of(System.getProperty("user.home"), ".cache", "whisper.cpp"),
Path.of(System.getProperty("user.home"), "Library", "Application Support",
"whisper.cpp"),
Path.of("/opt/homebrew/share/whisper.cpp/models"),
Path.of("/usr/local/share/whisper.cpp/models"),
Path.of("models"));
for (Path place : places) {
if (!Files.isDirectory(place)) {
continue;
}
try (var stream = Files.list(place)) {
// większy model bywa dokładniejszy, a przy polskim to widać
return stream.filter(p -> p.getFileName().toString().endsWith(".bin"))
.max(java.util.Comparator.comparingLong(Transcriber::sizeOf))
.map(Path::toString).orElse(null);
} catch (IOException ignored) {
// katalog może zniknąć między sprawdzeniem a listowaniem
}
}
return null;
}
private static long sizeOf(Path path) {
try {
return Files.size(path);
} catch (IOException e) {
return 0;
}
}
private static boolean ffmpegAvailable() {
return onPath("ffmpeg") != null;
}
// ---------- praca ----------
/** Co przetranskrybować: cała kolekcja albo jedno wydanie. */
public record Scope(Integer editionId, int limit, boolean redo) {
}
private record Job(String sha1, String name, String lang) {
}
public Progress progress() {
return progress;
}
public boolean running() {
Thread current = worker;
return current != null && current.isAlive();
}
public synchronized void start(Scope scope) throws Exception {
if (running()) {
throw new IllegalStateException("transkrypcja już trwa");
}
Tool tool = detect();
if (!tool.ready()) {
throw new IllegalStateException(tool.note());
}
List<Job> jobs;
synchronized (lock) {
jobs = pending(scope);
}
if (jobs.isEmpty()) {
progress = new Progress("bezczynny", 0, 0, 0, null, 0,
"nie ma czego transkrybować");
return;
}
cancelled.set(false);
long started = System.currentTimeMillis();
progress = new Progress("pracuje", jobs.size(), 0, 0, null, 0,
"model: " + Path.of(tool.model()).getFileName());
worker = new Thread(() -> run(tool, jobs, started), "transkrypcja");
worker.setDaemon(true);
worker.start();
}
public void stop() {
cancelled.set(true);
}
private void run(Tool tool, List<Job> jobs, long started) {
int done = 0;
int failed = 0;
Path workDir = null;
try {
workDir = Files.createTempDirectory("rex-transkrypcja");
List<Window> pending = new ArrayList<>();
for (List<Job> group : byLanguage(jobs)) {
for (Job job : group) {
if (cancelled.get()) {
break;
}
byte[] pcm;
try {
byte[] raw;
synchronized (lock) {
MediaIndex.Clip clip = media.read(job.sha1());
raw = clip == null ? null : clip.bytes();
}
if (raw == null) {
failed++;
continue;
}
pcm = toPcm(raw);
} catch (Exception e) {
System.err.printf(" ! %s: %s%n", job.name(), e.getMessage());
failed++;
continue;
}
Window last = pending.isEmpty() ? null : pending.get(pending.size() - 1);
if (last == null || !last.accepts(job, pcm)) {
last = new Window(job.lang());
pending.add(last);
}
last.add(job, pcm);
if (pending.size() >= BATCH) {
int[] result = flush(tool, workDir, pending, jobs.size(), done, failed,
started);
done = result[0];
failed = result[1];
pending.clear();
}
}
// język się zmienia, więc niedomknięte okno musi pójść teraz
if (!pending.isEmpty()) {
int[] result = flush(tool, workDir, pending, jobs.size(), done, failed,
started);
done = result[0];
failed = result[1];
pending.clear();
}
if (cancelled.get()) {
break;
}
}
} catch (Exception e) {
progress = new Progress("błąd", jobs.size(), done, failed, null,
System.currentTimeMillis() - started, e.getMessage());
return;
} finally {
deleteQuietly(workDir);
}
progress = new Progress(cancelled.get() ? "przerwany" : "zakończony", jobs.size(),
done, failed, null, System.currentTimeMillis() - started, progress.note());
}
/** Przetwarza zebrane okna i zwraca zaktualizowane liczniki {@code {zrobione, błędy}}. */
private int[] flush(Tool tool, Path workDir, List<Window> windows, int total, int done,
int failed, long started) {
progress = new Progress("pracuje", total, done, failed,
windows.get(0).first(), System.currentTimeMillis() - started, progress.note());
List<Path> inputs = new ArrayList<>();
for (int i = 0; i < windows.size(); i++) {
try {
inputs.add(windows.get(i).write(workDir, i));
} catch (Exception e) {
System.err.printf(" ! okno %d: %s%n", i, e.getMessage());
failed += windows.get(i).jobs().size();
return new int[]{done, failed};
}
}
try {
runWhisper(tool, inputs, windows.get(0).lang());
} catch (Exception e) {
System.err.printf(" ! whisper: %s%n", e.getMessage());
for (Window window : windows) {
failed += window.jobs().size();
}
clean(inputs);
return new int[]{done, failed};
}
for (int i = 0; i < windows.size(); i++) {
Window window = windows.get(i);
Map<String, String> texts;
try {
texts = window.split(readSegments(inputs.get(i)));
} catch (Exception e) {
System.err.printf(" ! okno %d: %s%n", i, e.getMessage());
failed += window.jobs().size();
continue;
}
for (Job job : window.jobs()) {
try {
synchronized (lock) {
store(job, withoutHallucinations(texts.getOrDefault(job.sha1(), "")),
tool);
}
done++;
} catch (Exception e) {
System.err.printf(" ! %s: %s%n", job.name(), e.getMessage());
failed++;
}
}
}
clean(inputs);
return new int[]{done, failed};
}
private static List<List<Job>> byLanguage(List<Job> jobs) {
Map<String, List<Job>> grouped = new LinkedHashMap<>();
for (Job job : jobs) {
grouped.computeIfAbsent(job.lang(), k -> new ArrayList<>()).add(job);
}
return new ArrayList<>(grouped.values());
}
/**
* Okno: kilka kwestii sklejonych ciszą w jeden strumień, razem z zapisem tego,
* gdzie która się zaczyna. Bez tych pozycji nie dałoby się przypisać wyniku
* z powrotem do pliku.
*/
private static final class Window {
private final String lang;
private final List<Job> jobs = new ArrayList<>();
private final List<int[]> spans = new ArrayList<>();
private final ByteArrayOutputStream pcm = new ByteArrayOutputStream();
Window(String lang) {
this.lang = lang;
}
String lang() {
return lang;
}
List<Job> jobs() {
return jobs;
}
String first() {
return jobs.isEmpty() ? null : jobs.get(0).name();
}
/** Puste okno przyjmie wszystko — także kwestię dłuższą niż samo okno. */
boolean accepts(Job job, byte[] audio) {
if (!lang.equals(job.lang())) {
return false;
}
if (jobs.isEmpty()) {
return true;
}
// z wyłączonym sklejaniem każde okno niesie jedną kwestię; reszta
// ścieżki zostaje ta sama, więc oba tryby da się porównać wprost
return CONCAT && lengthMs() + GAP_MS + msOf(audio) <= WINDOW_MS;
}
void add(Job job, byte[] audio) {
try {
if (!jobs.isEmpty()) {
pcm.write(new byte[GAP_MS * BYTES_PER_MS]);
}
int start = lengthMs();
pcm.write(audio);
spans.add(new int[]{start, lengthMs()});
jobs.add(job);
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
private int lengthMs() {
return pcm.size() / BYTES_PER_MS;
}
private static int msOf(byte[] audio) {
return audio.length / BYTES_PER_MS;
}
Path write(Path workDir, int index) throws IOException {
Path wav = workDir.resolve("okno" + index + ".wav");
byte[] data = pcm.toByteArray();
try (OutputStream out = Files.newOutputStream(wav)) {
out.write(wavHeader(data.length));
out.write(data);
}
return wav;
}
/**
* Rozdziela segmenty whispera z powrotem na kwestie. Segment trafia tam,
* gdzie ma największe pokrycie — jedna kwestia bywa rozbita na kilka
* segmentów, a segment potrafi zahaczyć o ciszę.
*/
Map<String, String> split(List<int[]> segments, List<String> texts) {
Map<String, StringBuilder> out = new LinkedHashMap<>();
for (int i = 0; i < segments.size(); i++) {
int best = -1;
int bestOverlap = 0;
for (int j = 0; j < spans.size(); j++) {
int overlap = Math.min(segments.get(i)[1], spans.get(j)[1])
- Math.max(segments.get(i)[0], spans.get(j)[0]);
if (overlap > bestOverlap) {
bestOverlap = overlap;
best = j;
}
}
if (best < 0) {
continue;
}
out.computeIfAbsent(jobs.get(best).sha1(), k -> new StringBuilder())
.append(texts.get(i).trim()).append(' ');
}
Map<String, String> result = new LinkedHashMap<>();
out.forEach((sha1, text) -> result.put(sha1, text.toString().trim()));
return result;
}
Map<String, String> split(Segments segments) {
return split(segments.ranges(), segments.texts());
}
}
/** Nagłówek RIFF dla 16 kHz mono, 16 bitów — dokładnie tego oczekuje whisper.cpp. */
private static byte[] wavHeader(int dataLength) {
ByteBuffer header = ByteBuffer.allocate(44).order(ByteOrder.LITTLE_ENDIAN);
header.put("RIFF".getBytes(StandardCharsets.US_ASCII));
header.putInt(36 + dataLength);
header.put("WAVEfmt ".getBytes(StandardCharsets.US_ASCII));
header.putInt(16);
header.putShort((short) 1);
header.putShort((short) 1);
header.putInt(WHISPER_RATE);
header.putInt(WHISPER_RATE * 2);
header.putShort((short) 2);
header.putShort((short) 16);
header.put("data".getBytes(StandardCharsets.US_ASCII));
header.putInt(dataLength);
return header.array();
}
/** Segmenty odczytane z wyniku whispera: zakresy czasu i odpowiadający im tekst. */
record Segments(List<int[]> ranges, List<String> texts) {
}
private static Segments readSegments(Path window) throws IOException {
Path json = jsonFor(window);
if (!Files.isReadable(json)) {
throw new IOException("whisper nie zapisał wyniku dla " + window.getFileName());
}
JsonObject root = JsonParser
.parseString(Files.readString(json, StandardCharsets.UTF_8))
.getAsJsonObject();
List<int[]> ranges = new ArrayList<>();
List<String> texts = new ArrayList<>();
for (JsonElement element : root.getAsJsonArray("transcription")) {
JsonObject segment = element.getAsJsonObject();
JsonObject offsets = segment.getAsJsonObject("offsets");
ranges.add(new int[]{offsets.get("from").getAsInt(), offsets.get("to").getAsInt()});
texts.add(segment.get("text").getAsString());
}
return new Segments(ranges, texts);
}
/**
* Bez {@code -of} whisper.cpp dopisuje rozszerzenie do pełnej nazwy wejścia:
* {@code okno0.wav} → {@code okno0.wav.json}. Starsze wydania podmieniały je
* zamiast dopisywać, więc sprawdzamy obie postacie.
*/
private static Path jsonFor(Path window) {
String name = window.getFileName().toString();
Path dopisane = window.resolveSibling(name + ".json");
if (Files.isReadable(dopisane)) {
return dopisane;
}
return window.resolveSibling(name.substring(0, name.lastIndexOf('.')) + ".json");
}
private static void clean(List<Path> inputs) {
for (Path input : inputs) {
try {
Files.deleteIfExists(input);
Files.deleteIfExists(jsonFor(input));
} catch (IOException ignored) {
// katalog tymczasowy i tak zniknie na końcu
}
}
}
/**
* Usuwa zdania, których model nie usłyszał, tylko je dopisał. Jeśli po
* odsianiu nie zostaje nic, zapisujemy pustkę — plik jest wtedy oznaczony
* jako przerobiony i nie wraca przy kolejnym przebiegu.
*/
static String withoutHallucinations(String text) {
if (text == null || text.isBlank()) {
return "";
}
StringBuilder out = new StringBuilder();
for (String line : text.split("\n")) {
String probe = line.toLowerCase(Locale.ROOT);
if (WYMYSLY.stream().noneMatch(probe::contains)) {
out.append(line).append('\n');
}
}
return out.toString().trim();
}
/** ffmpeg przepróbkowuje do 16 kHz mono; surowe próbki sklejamy sami. */
private byte[] toPcm(byte[] raw) throws Exception {
Process ffmpeg = new ProcessBuilder("ffmpeg", "-hide_banner", "-loglevel", "error",
"-i", "pipe:0", "-ar", String.valueOf(WHISPER_RATE), "-ac", "1",
"-f", "s16le", "pipe:1")
.start();
Thread feeder = new Thread(() -> {
try (OutputStream in = ffmpeg.getOutputStream()) {
in.write(raw);
} catch (IOException ignored) {
// ffmpeg potrafi zamknąć wejście, gdy nagłówek mu wystarczy
}
});
feeder.setDaemon(true);
feeder.start();
byte[] pcm = ffmpeg.getInputStream().readAllBytes();
String errors = new String(ffmpeg.getErrorStream().readAllBytes(), StandardCharsets.UTF_8);
if (ffmpeg.waitFor() != 0) {
throw new IOException("ffmpeg: " + errors.trim());
}
feeder.join(1000);
return pcm;
}
/**
* Jedno uruchomienie whisper.cpp na kilka okien. Wynik zapisujemy w JSON-ie
* ({@code -oj}), bo tylko tam są znaczniki czasu potrzebne do rozcięcia okna
* z powrotem na kwestie.
*/
private void runWhisper(Tool tool, List<Path> inputs, String lang) throws Exception {
List<String> command = new ArrayList<>(List.of(tool.binary(),
"-m", tool.model(), "-l", lang, "-t", THREADS, "-np", "-oj"));
inputs.forEach(path -> command.add(path.toString()));
Process whisper = new ProcessBuilder(command).start();
whisper.getInputStream().readAllBytes();
String errors = new String(whisper.getErrorStream().readAllBytes(), StandardCharsets.UTF_8);
if (whisper.waitFor() != 0) {
throw new IOException(errors.trim());
}
}
private void store(Job job, String text, Tool tool) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement("""
INSERT INTO transcript (sha1, text, lang, model, tool, duration_ms, created_at)
VALUES (?, ?, ?, ?, 'whisper.cpp',
(SELECT duration_ms FROM media WHERE sha1 = ?), ?)
ON CONFLICT(sha1) DO UPDATE SET
text = excluded.text, lang = excluded.lang, model = excluded.model,
tool = excluded.tool, created_at = excluded.created_at
""")) {
st.setString(1, job.sha1());
st.setString(2, text);
st.setString(3, job.lang());
st.setString(4, Path.of(tool.model()).getFileName().toString());
st.setString(5, job.sha1());
st.setString(6, Instant.now().toString());
st.executeUpdate();
}
try (PreparedStatement delete = db.connection().prepareStatement(
"DELETE FROM transcript_fts WHERE sha1 = ?");
PreparedStatement insert = db.connection().prepareStatement(
"INSERT INTO transcript_fts (body, sha1) VALUES (?, ?)")) {
delete.setString(1, job.sha1());
delete.executeUpdate();
if (!text.isBlank()) {
insert.setString(1, text);
insert.setString(2, job.sha1());
insert.executeUpdate();
}
}
}
/**
* Wybiera mowę, pomijając efekty dźwiękowe — katalog {@code sfx/} i nazwy
* zaczynające się od {@code sfx_} to nie kwestie, a model i tak wypisałby
* dla nich przypadkowe słowa.
*/
private List<Job> pending(Scope scope) throws Exception {
List<Job> jobs = new ArrayList<>();
StringBuilder sql = new StringBuilder("""
SELECT klip.sha1 AS sha1, klip.name AS name FROM (
SELECT f.blob_sha1 AS sha1, f.path AS name, g.edition_id AS edition_id
FROM file f
JOIN blob b ON b.sha1 = f.blob_sha1
JOIN game_root g ON g.copy_id = f.copy_id
WHERE b.format = 'WAV'
UNION
SELECT ae.sha1, ae.name, g.edition_id
FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
JOIN game_root g ON g.copy_id = f.copy_id
) AS klip
JOIN media m ON m.sha1 = klip.sha1
LEFT JOIN transcript t ON t.sha1 = klip.sha1
WHERE m.kind = 'audio' AND COALESCE(m.duration_ms, 0) >= 300
AND klip.name NOT LIKE '%/sfx/%'
AND klip.name NOT LIKE 'sfx!_%' ESCAPE '!'
AND klip.name NOT LIKE '%/sfx!_%' ESCAPE '!'
""");
if (!scope.redo()) {
sql.append(" AND t.sha1 IS NULL\n");
}
if (scope.editionId() != null) {
sql.append(" AND klip.edition_id = ?\n");
}
sql.append("GROUP BY klip.sha1 ORDER BY klip.name LIMIT ?");
try (PreparedStatement st = db.connection().prepareStatement(sql.toString())) {
int index = 1;
if (scope.editionId() != null) {
st.setInt(index++, scope.editionId());
}
st.setInt(index, scope.limit() > 0 ? scope.limit() : 1_000_000);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
String name = rs.getString("name");
jobs.add(new Job(rs.getString("sha1"), name, languageOf(name)));
}
}
}
return jobs;
}
/**
* Wersja językowa wynika ze ścieżki: {@code wavs/cze/} to czeski dubbing.
* Kody katalogów są te same, których używa silnik.
*/
static String languageOf(String path) {
String lower = path.toLowerCase(Locale.ROOT);
if (lower.contains("/cze/")) {
return "cs";
}
if (lower.contains("/hun/")) {
return "hu";
}
if (lower.contains("/slo/")) {
return "sk";
}
if (lower.contains("/niem/")) {
return "de";
}
return "pl";
}
private static void deleteQuietly(Path directory) {
if (directory == null) {
return;
}
try (var stream = Files.walk(directory)) {
stream.sorted(Comparator.reverseOrder()).forEach(path -> {
try {
Files.deleteIfExists(path);
} catch (IOException ignored) {
// katalog tymczasowy, system i tak go w końcu sprzątnie
}
});
} catch (IOException ignored) {
// jw.
}
}
}
@@ -0,0 +1,942 @@
package pl.genschu.rexcatalog.web;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import pl.genschu.rexcatalog.db.Database;
import pl.genschu.rexcatalog.media.MediaIndex;
import pl.genschu.rexcatalog.media.PikGraphics;
import pl.genschu.rexcatalog.script.ScriptSearch;
import pl.genschu.rexcatalog.transcribe.Transcriber;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
/**
* Przeglądarkowy front katalogu: statyczna strona z zasobów oraz API JSON.
*
* <p>Osobno od narzędzi MCP, bo odbiorca jest inny — model czyta tekst, przeglądarka
* potrzebuje struktur. Wspólna jest baza, nie warstwa prezentacji.
*/
public final class WebServer {
private final Database db;
private final Path dataDir;
private final ScriptSearch search;
private final MediaIndex media;
private final Transcriber transcriber;
private final Object lock;
public WebServer(Database db, Path dataDir, Object lock) {
this.db = db;
this.dataDir = dataDir;
this.search = new ScriptSearch(db, dataDir);
this.media = new MediaIndex(db);
this.transcriber = new Transcriber(db, lock);
this.lock = lock;
}
public void register(HttpServer http) {
http.createContext("/api/", this::handleApi);
http.createContext("/", this::handleStatic);
}
// ---------- statyczne ----------
private void handleStatic(HttpExchange exchange) throws IOException {
try {
String path = exchange.getRequestURI().getPath();
if (path.equals("/") || path.isEmpty()) {
path = "/index.html";
}
// zasoby są wbudowane w jar, ale ścieżka i tak przychodzi z sieci
if (path.contains("..")) {
respond(exchange, 400, "text/plain", "Nieprawidłowa ścieżka".getBytes(UTF8));
return;
}
try (InputStream in = WebServer.class.getResourceAsStream("/web" + path)) {
if (in == null) {
respond(exchange, 404, "text/plain", "Nie ma takiej strony".getBytes(UTF8));
return;
}
respond(exchange, 200, contentType(path), in.readAllBytes());
}
} finally {
exchange.close();
}
}
private static String contentType(String path) {
if (path.endsWith(".html")) {
return "text/html";
}
if (path.endsWith(".css")) {
return "text/css";
}
if (path.endsWith(".js")) {
return "application/javascript";
}
return "application/octet-stream";
}
// ---------- API ----------
private void handleApi(HttpExchange exchange) throws IOException {
try {
String path = exchange.getRequestURI().getPath().substring("/api".length());
Map<String, String> query = parseQuery(exchange.getRequestURI().getRawQuery());
// dźwięk i obrazy wychodzą bajtami, a nie JSON-em
if (path.startsWith("/audio/")) {
serveAudio(exchange, path.substring("/audio/".length()));
return;
}
if (path.startsWith("/preview/")) {
servePreview(exchange, path.substring("/preview/".length()));
return;
}
if (path.startsWith("/render/")) {
serveRender(exchange, path.substring("/render/".length()),
parseInt(query.get("frame"), 0));
return;
}
Object payload;
synchronized (lock) {
payload = route(path, query, exchange.getRequestMethod());
}
if (payload == null) {
respond(exchange, 404, "application/json",
"{\"error\":\"nieznany zasób\"}".getBytes(UTF8));
return;
}
respond(exchange, 200, "application/json", payload.toString().getBytes(UTF8));
} catch (Exception e) {
JsonObject error = new JsonObject();
error.addProperty("error", e.getMessage() == null ? e.toString() : e.getMessage());
respond(exchange, 500, "application/json", error.toString().getBytes(UTF8));
} finally {
exchange.close();
}
}
private Object route(String path, Map<String, String> query, String method)
throws Exception {
if (path.equals("/stats")) {
return stats();
}
if (path.equals("/titles")) {
return titles();
}
if (path.equals("/search")) {
return searchScripts(query.getOrDefault("q", ""),
parseInt(query.get("limit"), 30));
}
if (path.startsWith("/editions/")) {
String rest = path.substring("/editions/".length());
if (rest.endsWith("/files")) {
int id = parseInt(rest.substring(0, rest.length() - "/files".length()), -1);
return files(id, query.get("format"), query.get("q"),
parseInt(query.get("limit"), 300));
}
return edition(parseInt(rest, -1));
}
if (path.startsWith("/scripts/")) {
return script(path.substring("/scripts/".length()));
}
if (path.equals("/audio-search")) {
return audioSearch(query.getOrDefault("q", ""), parseInt(query.get("limit"), 60));
}
if (path.startsWith("/media/")) {
return mediaInfo(path.substring("/media/".length()));
}
if (path.equals("/transcribe/status")) {
return transcribeStatus();
}
// start i stop zmieniają stan, więc tylko POST — GET bywa wywołany przez
// podgląd odsyłacza albo wczytanie z historii, a to godziny liczenia
if (path.equals("/transcribe/start")) {
return "POST".equals(method) ? transcribeStart(query) : methodNotAllowed();
}
if (path.equals("/transcribe/stop")) {
if (!"POST".equals(method)) {
return methodNotAllowed();
}
transcriber.stop();
return transcribeStatus();
}
return null;
}
private JsonObject methodNotAllowed() {
JsonObject out = new JsonObject();
out.addProperty("error", "ten zasób wymaga metody POST");
return out;
}
// ---------- dźwięk ----------
/**
* Podaje nagranie prosto z obrazu płyty. Nic nie ląduje na dysku — kopia
* zostaje jednym plikiem, a przeglądarka dostaje wycięty z niej fragment.
*/
private void serveAudio(HttpExchange exchange, String sha1) throws IOException {
MediaIndex.Clip clip;
try {
synchronized (lock) {
clip = media.read(sha1);
}
} catch (Exception e) {
respond(exchange, 500, "text/plain",
("Nie odczytałem nagrania: " + e.getMessage()).getBytes(UTF8));
return;
}
if (clip == null) {
respond(exchange, 404, "text/plain", "Nie ma takiego zasobu".getBytes(UTF8));
return;
}
exchange.getResponseHeaders().set("Content-Type", clip.contentType());
exchange.getResponseHeaders().set("Cache-Control", "public, max-age=86400");
exchange.sendResponseHeaders(200, clip.bytes().length);
exchange.getResponseBody().write(clip.bytes());
}
// ---------- grafika ----------
/** Miniatura z cache'u — mała i gotowa, do przeglądania list. */
private void servePreview(HttpExchange exchange, String sha1) throws IOException {
String relative;
try {
synchronized (lock) {
relative = previewPath(sha1);
}
} catch (Exception e) {
respond(exchange, 500, "text/plain", String.valueOf(e.getMessage()).getBytes(UTF8));
return;
}
if (relative == null) {
respond(exchange, 404, "text/plain", "Brak miniatury".getBytes(UTF8));
return;
}
java.nio.file.Path file = dataDir.resolve(relative);
if (!java.nio.file.Files.isReadable(file)) {
respond(exchange, 404, "text/plain",
"Miniatura zniknęła z cache'u — uruchom probe".getBytes(UTF8));
return;
}
byte[] bytes = java.nio.file.Files.readAllBytes(file);
exchange.getResponseHeaders().set("Content-Type", "image/png");
exchange.getResponseHeaders().set("Cache-Control", "public, max-age=86400");
exchange.sendResponseHeaders(200, bytes.length);
exchange.getResponseBody().write(bytes);
}
/**
* Pełny obraz odtwarzany z płyty na żądanie. W cache'u trzymamy tylko miniatury,
* bo rozpakowana grafika zajęłaby więcej niż sama kolekcja, a odczyt z obrazu
* i rozpakowanie trwają milisekundy.
*/
private void serveRender(HttpExchange exchange, String sha1, int frame) throws IOException {
byte[] png;
try {
MediaIndex.Clip clip;
String kind;
synchronized (lock) {
clip = media.read(sha1);
kind = mediaKind(sha1);
}
if (clip == null) {
respond(exchange, 404, "text/plain", "Nie ma takiego zasobu".getBytes(UTF8));
return;
}
java.awt.image.BufferedImage image;
if ("anim".equals(kind)) {
PikGraphics.Animation animation = PikGraphics.readAnimation(clip.bytes());
if (animation.frames().isEmpty()) {
respond(exchange, 404, "text/plain", "Animacja bez klatek".getBytes(UTF8));
return;
}
int index = Math.max(0, Math.min(frame, animation.frames().size() - 1));
image = animation.frames().get(index).image();
} else {
image = PikGraphics.readImage(clip.bytes()).image();
}
java.io.ByteArrayOutputStream buffer = new java.io.ByteArrayOutputStream();
javax.imageio.ImageIO.write(image, "png", buffer);
png = buffer.toByteArray();
} catch (Exception e) {
respond(exchange, 500, "text/plain",
("Nie zdekodowałem: " + e.getMessage()).getBytes(UTF8));
return;
}
exchange.getResponseHeaders().set("Content-Type", "image/png");
exchange.getResponseHeaders().set("Cache-Control", "public, max-age=3600");
exchange.sendResponseHeaders(200, png.length);
exchange.getResponseBody().write(png);
}
private String previewPath(String sha1) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT path FROM artifact WHERE blob_sha1 = ? AND kind = 'preview'")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getString(1) : null;
}
}
}
private String mediaKind(String sha1) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT kind FROM media WHERE sha1 = ?")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getString(1) : null;
}
}
}
// ---------- transkrypcja ----------
private JsonObject transcribeStatus() {
Transcriber.Tool tool = Transcriber.detect();
Transcriber.Progress progress = transcriber.progress();
JsonObject out = new JsonObject();
out.addProperty("ready", tool.ready());
out.addProperty("tool_note", tool.note());
out.addProperty("binary", tool.binary());
out.addProperty("model", tool.model());
out.addProperty("running", transcriber.running());
out.addProperty("state", progress.state());
out.addProperty("total", progress.total());
out.addProperty("done", progress.done());
out.addProperty("failed", progress.failed());
out.addProperty("current", progress.current());
out.addProperty("elapsed_ms", progress.elapsedMs());
out.addProperty("note", progress.note());
return out;
}
private JsonObject transcribeStart(Map<String, String> query) throws Exception {
Integer edition = query.containsKey("edition")
? parseInt(query.get("edition"), -1) : null;
if (edition != null && edition < 0) {
edition = null;
}
try {
transcriber.start(new Transcriber.Scope(edition,
parseInt(query.get("limit"), 0),
"1".equals(query.get("redo"))));
} catch (IllegalStateException e) {
JsonObject out = transcribeStatus();
out.addProperty("error", e.getMessage());
return out;
}
return transcribeStatus();
}
private JsonObject stats() throws Exception {
JsonObject out = new JsonObject();
out.addProperty("copies", scalar("SELECT COUNT(*) FROM copy"));
out.addProperty("titles", scalar("SELECT COUNT(*) FROM title"));
out.addProperty("editions", scalar("SELECT COUNT(*) FROM edition"));
out.addProperty("files", scalar("SELECT COUNT(*) FROM file"));
out.addProperty("blobs", scalar("SELECT COUNT(*) FROM blob"));
out.addProperty("scripts", scalar("SELECT COUNT(*) FROM script_fts"));
out.addProperty("bytes", scalar("SELECT COALESCE(SUM(size), 0) FROM blob"));
out.addProperty("audio", scalar("SELECT COUNT(*) FROM media WHERE kind = 'audio'"));
out.addProperty("audio_ms",
scalar("SELECT COALESCE(SUM(duration_ms), 0) FROM media WHERE kind = 'audio'"));
out.addProperty("archive_entries", scalar("SELECT COUNT(*) FROM archive_entry"));
out.addProperty("dialogue", scalar("SELECT COUNT(*) FROM dialogue_line"));
out.addProperty("refs", scalar("SELECT COUNT(*) FROM script_ref"));
out.addProperty("transcripts", scalar("SELECT COUNT(*) FROM transcript"));
out.addProperty("images", scalar("SELECT COUNT(*) FROM media WHERE kind = 'image'"));
out.addProperty("animations", scalar("SELECT COUNT(*) FROM media WHERE kind = 'anim'"));
out.addProperty("anim_frames",
scalar("SELECT COALESCE(SUM(frames), 0) FROM media WHERE kind = 'anim'"));
JsonArray formats = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT format, COUNT(*) n, SUM(size) bytes FROM blob "
+ "GROUP BY format ORDER BY n DESC LIMIT 20");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("format", rs.getString("format"));
row.addProperty("n", rs.getInt("n"));
row.addProperty("bytes", rs.getLong("bytes"));
formats.add(row);
}
}
out.add("formats", formats);
return out;
}
private JsonArray titles() throws Exception {
JsonArray out = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT e.id AS edition_id, t.name AS title, t.series, e.label,
COALESCE(e.release_date_override, (
SELECT d.value FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id AND d.root_path = g.root_path
WHERE g.edition_id = e.id AND d.field = 'release_date' LIMIT 1)) AS release_date,
COALESCE(e.engine_override, (
SELECT g.engine_detected FROM game_root g
WHERE g.edition_id = e.id LIMIT 1)) AS engine,
(SELECT GROUP_CONCAT(lang, ' ') FROM edition_language
WHERE edition_id = e.id) AS langs,
(SELECT COUNT(*) FROM game_root WHERE edition_id = e.id) AS copies
FROM edition e LEFT JOIN title t ON t.id = e.title_id
ORDER BY t.series IS NULL, t.series, t.name, release_date
""");
ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("edition_id", rs.getInt("edition_id"));
row.addProperty("title", rs.getString("title"));
row.addProperty("series", rs.getString("series"));
row.addProperty("label", rs.getString("label"));
row.addProperty("release_date", rs.getString("release_date"));
row.addProperty("engine", rs.getString("engine"));
row.addProperty("langs", rs.getString("langs"));
row.addProperty("copies", rs.getInt("copies"));
out.add(row);
}
}
return out;
}
private JsonObject edition(int id) throws Exception {
JsonObject out = new JsonObject();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT e.id, t.name AS title, t.series, t.publisher, e.label, e.fingerprint,
e.dll_sha1, e.app_def_sha1, e.distributor, e.notes,
e.engine_override, e.engine_version_override,
e.compiler_override, e.release_date_override
FROM edition e LEFT JOIN title t ON t.id = e.title_id WHERE e.id = ?
""")) {
st.setInt(1, id);
try (ResultSet rs = st.executeQuery()) {
if (!rs.next()) {
return null;
}
for (String column : List.of("title", "series", "publisher", "label",
"fingerprint", "dll_sha1", "app_def_sha1", "distributor", "notes",
"engine_override", "engine_version_override", "compiler_override",
"release_date_override")) {
out.addProperty(column, rs.getString(column));
}
out.addProperty("edition_id", rs.getInt("id"));
}
}
JsonArray languages = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement(
"SELECT lang, role FROM edition_language WHERE edition_id = ? ORDER BY lang, role")) {
st.setInt(1, id);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("lang", rs.getString("lang"));
row.addProperty("role", rs.getString("role"));
languages.add(row);
}
}
}
out.add("languages", languages);
JsonArray copies = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT c.id, c.display_name, c.path, c.container, c.fs_type, c.status,
c.status_note, c.file_count, c.size, c.sha256, c.source_kind,
c.source_url, c.source_ref, c.rip_tool, c.acquired_at, c.verified,
c.notes, g.root_path
FROM game_root g JOIN copy c ON c.id = g.copy_id
WHERE g.edition_id = ? ORDER BY c.display_name
""")) {
st.setInt(1, id);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
for (String column : List.of("display_name", "path", "container", "fs_type",
"status", "status_note", "sha256", "source_kind", "source_url",
"source_ref", "rip_tool", "acquired_at", "notes", "root_path")) {
row.addProperty(column, rs.getString(column));
}
row.addProperty("copy_id", rs.getInt("id"));
row.addProperty("file_count", rs.getInt("file_count"));
row.addProperty("size", rs.getLong("size"));
row.addProperty("verified", rs.getInt("verified") == 1);
copies.add(row);
}
}
}
out.add("copies", copies);
JsonArray facts = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT DISTINCT d.field, d.value, d.confidence, d.evidence, d.detector
FROM game_root g
JOIN detection d ON d.copy_id = g.copy_id AND d.root_path = g.root_path
WHERE g.edition_id = ? ORDER BY d.detector, d.field
""")) {
st.setInt(1, id);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("field", rs.getString("field"));
row.addProperty("value", rs.getString("value"));
row.addProperty("confidence", rs.getDouble("confidence"));
row.addProperty("evidence", rs.getString("evidence"));
row.addProperty("detector", rs.getString("detector"));
facts.add(row);
}
}
}
out.add("facts", facts);
return out;
}
private JsonArray files(int editionId, String format, String pathContains, int limit)
throws Exception {
StringBuilder sql = new StringBuilder("""
SELECT DISTINCT COALESCE(f.path_raw, f.path) AS path, b.format, b.size,
b.encrypted, f.blob_sha1,
(SELECT 1 FROM artifact a WHERE a.blob_sha1 = f.blob_sha1
AND a.kind = 'script') AS decoded
FROM game_root g
JOIN file f ON f.copy_id = g.copy_id
JOIN blob b ON b.sha1 = f.blob_sha1
WHERE g.edition_id = ?
""");
List<Object> params = new ArrayList<>();
params.add(editionId);
if (format != null && !format.isBlank()) {
sql.append(" AND b.format = ?\n");
params.add(format.toUpperCase(Locale.ROOT));
}
if (pathContains != null && !pathContains.isBlank()) {
sql.append(" AND f.path LIKE ?\n");
params.add("%" + pathContains.toLowerCase(Locale.ROOT) + "%");
}
sql.append("ORDER BY path LIMIT ?");
params.add(Math.min(limit, 2000));
JsonArray out = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement(sql.toString())) {
for (int i = 0; i < params.size(); i++) {
st.setObject(i + 1, params.get(i));
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("path", rs.getString("path"));
row.addProperty("format", rs.getString("format"));
row.addProperty("size", rs.getLong("size"));
row.addProperty("encrypted", rs.getInt("encrypted") == 1);
row.addProperty("sha1", rs.getString("blob_sha1"));
row.addProperty("decoded", rs.getInt("decoded") == 1);
out.add(row);
}
}
}
return out;
}
private JsonObject searchScripts(String query, int limit) throws Exception {
JsonObject out = new JsonObject();
if (query.isBlank()) {
out.addProperty("query", "");
out.add("hits", new JsonArray());
return out;
}
ScriptSearch.Result result = search.find(query, Math.min(limit, 200));
out.addProperty("query", result.effectiveQuery());
out.addProperty("quoted", result.quoted());
JsonArray hits = new JsonArray();
for (ScriptSearch.Hit hit : result.hits()) {
JsonObject row = new JsonObject();
row.addProperty("sha1", hit.sha1());
row.addProperty("snippet", hit.snippet());
row.add("paths", toArray(hit.paths()));
row.add("games", toArray(hit.games()));
hits.add(row);
}
out.add("hits", hits);
return out;
}
private JsonObject script(String sha1) throws Exception {
String body = search.body(sha1);
if (body == null) {
return null;
}
JsonObject out = new JsonObject();
out.addProperty("sha1", sha1);
out.addProperty("body", body);
out.add("paths", toArray(search.pathsFor(sha1)));
out.add("games", toArray(search.gamesFor(sha1)));
out.add("refs", refs(sha1));
return out;
}
/**
* Zasoby, do których odwołuje się skrypt, rozwiązane do konkretnych plików.
*
* <p>To jest sedno przeglądania: przy {@code SNDQUESTION:FILENAME=BUREKTOR_M030.WAV}
* widać od razu, ile ta kwestia trwa, kto ją wypowiada i przy jakim zdarzeniu
* pada — bez otwierania gry.
*/
private JsonArray refs(String scriptSha1) throws Exception {
JsonArray out = new JsonArray();
MediaIndex.ScriptPlace place = media.place(scriptSha1);
if (place == null) {
return out;
}
try (PreparedStatement st = db.connection().prepareStatement(
MediaIndex.RESOLVE_SQL + "ORDER BY r.line, r.object")) {
MediaIndex.bindResolve(st, place, scriptSha1);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
for (String column : List.of("object", "field", "value", "name", "format",
"target", "speaker", "trigger")) {
row.addProperty(column, rs.getString(column));
}
row.addProperty("line", rs.getInt("line"));
// trafienie spoza reguł silnika, przy kilku plikach o tej nazwie,
// jest domysłem, nie faktem — front musi to pokazać
row.addProperty("uncertain", rs.getInt("pewne") == 0
&& rs.getInt("wariantow") > 1);
row.addProperty("variants", rs.getInt("wariantow"));
out.add(row);
}
}
}
// fakty o zasobach dokładamy jednym przebiegiem, żeby nie pytać bazy po razie na wiersz
java.util.Set<String> targets = new java.util.LinkedHashSet<>();
for (var element : out) {
String target = element.getAsJsonObject().get("target").isJsonNull()
? null : element.getAsJsonObject().get("target").getAsString();
if (target != null) {
targets.add(target);
}
}
Map<String, JsonObject> facts = mediaFacts(targets);
for (var element : out) {
JsonObject row = element.getAsJsonObject();
if (!row.get("target").isJsonNull()) {
JsonObject fact = facts.get(row.get("target").getAsString());
if (fact != null) {
row.add("media", fact);
}
}
}
return out;
}
private Map<String, JsonObject> mediaFacts(java.util.Set<String> sha1s) throws Exception {
Map<String, JsonObject> out = new LinkedHashMap<>();
if (sha1s.isEmpty()) {
return out;
}
String placeholders = String.join(",", java.util.Collections.nCopies(sha1s.size(), "?"));
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT m.sha1, m.kind, m.codec, m.duration_ms, m.sample_rate, m.channels,
m.width, m.height, m.frames, m.fps, m.author, m.note,
t.text AS transcript, t.model AS transcript_model
FROM media m
LEFT JOIN transcript t ON t.sha1 = m.sha1
WHERE m.sha1 IN (%s)
""".formatted(placeholders))) {
int index = 1;
for (String sha1 : sha1s) {
st.setString(index++, sha1);
}
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("kind", rs.getString("kind"));
row.addProperty("codec", rs.getString("codec"));
row.addProperty("duration_ms", rs.getInt("duration_ms"));
row.addProperty("sample_rate", rs.getInt("sample_rate"));
row.addProperty("channels", rs.getInt("channels"));
row.addProperty("width", rs.getInt("width"));
row.addProperty("height", rs.getInt("height"));
row.addProperty("frames", rs.getInt("frames"));
row.addProperty("fps", rs.getInt("fps"));
row.addProperty("author", rs.getString("author"));
row.addProperty("note", rs.getString("note"));
row.addProperty("transcript", rs.getString("transcript"));
row.addProperty("transcript_model", rs.getString("transcript_model"));
out.put(rs.getString("sha1"), row);
}
}
}
return out;
}
/**
* Szuka nagrań po nazwie, a gdy transkrypty już są — również po tym, co słychać.
*
* <p>Nazwa i treść to dwa różne źródła, więc trafienia z transkryptu są osobno
* oznaczone: jedno jest zapisane na płycie, drugie zgadnięte przez model.
*/
private JsonObject audioSearch(String query, int limit) throws Exception {
JsonObject out = new JsonObject();
JsonArray hits = new JsonArray();
out.addProperty("query", query);
out.add("hits", hits);
if (query.isBlank()) {
return out;
}
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT klip.sha1, klip.name, klip.copy, m.duration_ms, m.codec,
(SELECT d.speaker FROM dialogue_line d
WHERE d.audio_name = klip.short LIMIT 1) AS speaker,
t.text AS transcript,
CASE WHEN instr(lower(klip.name), lower(?)) > 0 THEN 1 ELSE 0 END AS by_name
FROM (
SELECT f.blob_sha1 AS sha1, COALESCE(f.path_raw, f.path) AS name,
CASE WHEN instr(f.path, '/') > 0
THEN replace(f.path, rtrim(f.path, replace(f.path, '/', '')), '')
ELSE f.path END AS short,
c.display_name AS copy
FROM file f JOIN blob b ON b.sha1 = f.blob_sha1
JOIN copy c ON c.id = f.copy_id
WHERE b.format = 'WAV'
UNION
SELECT ae.sha1, ae.name_raw, ae.name, c.display_name
FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
JOIN copy c ON c.id = f.copy_id
) AS klip
JOIN media m ON m.sha1 = klip.sha1
LEFT JOIN transcript t ON t.sha1 = klip.sha1
WHERE instr(lower(klip.name), lower(?)) > 0
OR (t.text IS NOT NULL AND instr(lower(t.text), lower(?)) > 0)
GROUP BY klip.sha1
ORDER BY by_name DESC, klip.name
LIMIT ?
""")) {
st.setString(1, query);
st.setString(2, query);
st.setString(3, query);
st.setInt(4, Math.min(limit, 300));
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("sha1", rs.getString("sha1"));
row.addProperty("name", rs.getString("name"));
row.addProperty("copy", rs.getString("copy"));
row.addProperty("duration_ms", rs.getInt("duration_ms"));
row.addProperty("codec", rs.getString("codec"));
row.addProperty("speaker", rs.getString("speaker"));
row.addProperty("transcript", rs.getString("transcript"));
row.addProperty("by_name", rs.getInt("by_name") == 1);
hits.add(row);
}
}
}
return out;
}
/** Wszystko, co katalog wie o jednym zasobie: fakty, miejsca, dialog, transkrypt. */
private JsonObject mediaInfo(String sha1) throws Exception {
JsonObject out = new JsonObject();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT m.sha1, m.kind, m.codec, m.duration_ms, m.sample_rate, m.channels,
m.bits, m.note, m.width, m.height, m.frames, m.fps, m.author,
t.text AS transcript, t.model AS transcript_model,
t.lang AS transcript_lang, t.created_at AS transcript_at
FROM media m
LEFT JOIN transcript t ON t.sha1 = m.sha1
WHERE m.sha1 = ?
""")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
if (!rs.next()) {
return null;
}
for (String column : List.of("kind", "codec", "note", "author", "transcript",
"transcript_model", "transcript_lang", "transcript_at")) {
out.addProperty(column, rs.getString(column));
}
out.addProperty("sha1", sha1);
out.addProperty("duration_ms", rs.getInt("duration_ms"));
out.addProperty("sample_rate", rs.getInt("sample_rate"));
out.addProperty("channels", rs.getInt("channels"));
out.addProperty("bits", rs.getInt("bits"));
out.addProperty("width", rs.getInt("width"));
out.addProperty("height", rs.getInt("height"));
out.addProperty("frames", rs.getInt("frames"));
out.addProperty("fps", rs.getInt("fps"));
}
}
JsonArray places = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT COALESCE(f.path_raw, f.path) AS path, c.display_name, 0 AS in_archive
FROM file f JOIN copy c ON c.id = f.copy_id WHERE f.blob_sha1 = ?
UNION ALL
SELECT ae.name_raw, c.display_name, 1
FROM archive_entry ae
JOIN file f ON f.blob_sha1 = ae.container_sha1
JOIN copy c ON c.id = f.copy_id
WHERE ae.sha1 = ?
""")) {
st.setString(1, sha1);
st.setString(2, sha1);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("path", rs.getString(1));
row.addProperty("copy", rs.getString(2));
row.addProperty("in_archive", rs.getInt(3) == 1);
places.add(row);
}
}
}
out.add("places", places);
JsonArray dialogue = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT DISTINCT d.speaker, d.trigger, d.section, d.extra
FROM dialogue_line d
WHERE d.audio_name IN (
SELECT CASE WHEN instr(f.path, '/') > 0
THEN replace(f.path, rtrim(f.path, replace(f.path, '/', '')), '')
ELSE f.path END
FROM file f WHERE f.blob_sha1 = ?
UNION SELECT ae.name FROM archive_entry ae WHERE ae.sha1 = ?
)
""")) {
st.setString(1, sha1);
st.setString(2, sha1);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("speaker", rs.getString("speaker"));
row.addProperty("trigger", rs.getString("trigger"));
row.addProperty("section", rs.getString("section"));
row.addProperty("extra", rs.getString("extra"));
dialogue.add(row);
}
}
}
out.add("dialogue", dialogue);
JsonArray usedBy = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT DISTINCT r.script_sha1, r.object, r.field, a.path
FROM script_ref r
JOIN artifact a ON a.blob_sha1 = r.script_sha1 AND a.kind = 'script'
WHERE r.name IN (
SELECT CASE WHEN instr(f.path, '/') > 0
THEN replace(f.path, rtrim(f.path, replace(f.path, '/', '')), '')
ELSE f.path END
FROM file f WHERE f.blob_sha1 = ?
UNION SELECT ae.name FROM archive_entry ae WHERE ae.sha1 = ?
)
LIMIT 50
""")) {
st.setString(1, sha1);
st.setString(2, sha1);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("script_sha1", rs.getString("script_sha1"));
row.addProperty("object", rs.getString("object"));
row.addProperty("field", rs.getString("field"));
usedBy.add(row);
}
}
}
out.add("used_by", usedBy);
JsonArray events = new JsonArray();
try (PreparedStatement st = db.connection().prepareStatement("""
SELECT name, frames_count, loop_start, loop_end
FROM anim_event WHERE ann_sha1 = ? ORDER BY rowid
""")) {
st.setString(1, sha1);
try (ResultSet rs = st.executeQuery()) {
while (rs.next()) {
JsonObject row = new JsonObject();
row.addProperty("name", rs.getString("name"));
row.addProperty("frames_count", rs.getInt("frames_count"));
row.addProperty("loop_start", rs.getInt("loop_start"));
row.addProperty("loop_end", rs.getInt("loop_end"));
events.add(row);
}
}
}
out.add("events", events);
return out;
}
// ---------- pomocnicze ----------
private static final java.nio.charset.Charset UTF8 = StandardCharsets.UTF_8;
private static JsonArray toArray(List<String> values) {
JsonArray array = new JsonArray();
values.forEach(array::add);
return array;
}
private long scalar(String sql) throws Exception {
try (PreparedStatement st = db.connection().prepareStatement(sql);
ResultSet rs = st.executeQuery()) {
return rs.next() ? rs.getLong(1) : 0;
}
}
private static int parseInt(String value, int fallback) {
try {
return value == null ? fallback : Integer.parseInt(value.trim());
} catch (NumberFormatException e) {
return fallback;
}
}
private static Map<String, String> parseQuery(String raw) {
Map<String, String> out = new LinkedHashMap<>();
if (raw == null || raw.isBlank()) {
return out;
}
for (String pair : raw.split("&")) {
int eq = pair.indexOf('=');
if (eq < 0) {
continue;
}
out.put(URLDecoder.decode(pair.substring(0, eq), UTF8),
URLDecoder.decode(pair.substring(eq + 1), UTF8));
}
return out;
}
private static void respond(HttpExchange exchange, int status, String contentType, byte[] body)
throws IOException {
exchange.getResponseHeaders().set("Content-Type", contentType + "; charset=utf-8");
exchange.sendResponseHeaders(status, body.length == 0 ? -1 : body.length);
if (body.length > 0) {
exchange.getResponseBody().write(body);
}
}
}
+690
View File
@@ -0,0 +1,690 @@
<!doctype html>
<html lang="pl">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>rex-catalog</title>
<style>
:root {
--bg: #f6f5f2; --panel: #fff; --line: #ddd8cf; --ink: #2b2822; --dim: #7a7368;
--accent: #8a5a2b; --hl: #f2e6c9; --mono: ui-monospace, SFMono-Regular, Menlo, monospace;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #1a1815; --panel: #232019; --line: #3a352c; --ink: #e8e2d6; --dim: #948b7c;
--accent: #d3a05e; --hl: #3d3323;
}
}
* { box-sizing: border-box; }
body {
margin: 0; background: var(--bg); color: var(--ink);
font: 14px/1.5 system-ui, -apple-system, "Segoe UI", sans-serif;
}
header {
padding: 14px 20px; border-bottom: 1px solid var(--line); background: var(--panel);
display: flex; gap: 20px; align-items: baseline; flex-wrap: wrap;
}
h1 { margin: 0; font-size: 17px; letter-spacing: .02em; }
h1 span { color: var(--accent); }
#stats { color: var(--dim); font-size: 12.5px; }
nav { margin-left: auto; display: flex; gap: 4px; }
nav button {
border: 1px solid var(--line); background: transparent; color: var(--ink);
padding: 5px 12px; border-radius: 5px; cursor: pointer; font-size: 13px;
}
nav button.on { background: var(--accent); border-color: var(--accent); color: #fff; }
main { padding: 18px 20px; max-width: 1500px; }
/* własna reguła display bije [hidden] z arkusza przeglądarki, więc sekcja
z klasą .cols pokazywałaby się mimo ukrycia — stąd to wymuszenie */
[hidden] { display: none !important; }
.cols { display: grid; grid-template-columns: minmax(320px, 460px) 1fr; gap: 18px; align-items: start; }
@media (max-width: 900px) { .cols { grid-template-columns: 1fr; } }
.card {
background: var(--panel); border: 1px solid var(--line); border-radius: 8px;
padding: 14px 16px; margin-bottom: 14px;
}
table { border-collapse: collapse; width: 100%; font-size: 13px; }
th {
text-align: left; font-weight: 600; color: var(--dim); font-size: 11.5px;
text-transform: uppercase; letter-spacing: .04em;
padding: 6px 8px; border-bottom: 1px solid var(--line); white-space: nowrap;
}
td { padding: 6px 8px; border-bottom: 1px solid var(--line); vertical-align: top; }
tbody tr { cursor: pointer; }
tbody tr:hover { background: var(--hl); }
tbody tr.on { background: var(--hl); box-shadow: inset 3px 0 0 var(--accent); }
.series { color: var(--dim); font-size: 11.5px; text-transform: uppercase; letter-spacing: .05em; }
.label { color: var(--dim); font-size: 12.5px; }
.mono { font-family: var(--mono); font-size: 12px; }
.dim { color: var(--dim); }
input[type=search], input[type=text], select {
width: 100%; padding: 8px 10px; border: 1px solid var(--line); border-radius: 6px;
background: var(--bg); color: var(--ink); font-size: 13px; font-family: inherit;
}
.row { display: flex; gap: 8px; margin-bottom: 12px; }
.row > * { flex: 1; }
.row > .narrow { flex: 0 0 130px; }
dl { display: grid; grid-template-columns: auto 1fr; gap: 3px 14px; margin: 0; font-size: 13px; }
dt { color: var(--dim); white-space: nowrap; }
dd { margin: 0; overflow-wrap: anywhere; }
pre {
font-family: var(--mono); font-size: 12px; line-height: 1.45; margin: 0;
white-space: pre-wrap; overflow-wrap: anywhere; max-height: 62vh; overflow-y: auto;
}
mark { background: var(--accent); color: #fff; border-radius: 2px; padding: 0 1px; }
.snip { font-family: var(--mono); font-size: 11.5px; color: var(--dim); margin-top: 3px; }
.pill {
display: inline-block; padding: 1px 7px; border: 1px solid var(--line);
border-radius: 10px; font-size: 11px; color: var(--dim); margin-right: 4px;
}
.bar { height: 6px; background: var(--hl); border-radius: 3px; overflow: hidden; }
.bar > i { display: block; height: 100%; background: var(--accent); }
.empty { color: var(--dim); padding: 20px 0; text-align: center; }
h2 { font-size: 14px; margin: 0 0 10px; }
h3 { font-size: 12px; margin: 16px 0 6px; color: var(--dim);
text-transform: uppercase; letter-spacing: .05em; }
a { color: var(--accent); }
.zasob { display: grid; grid-template-columns: 1fr auto; gap: 6px 12px;
padding: 7px 8px; border-bottom: 1px solid var(--line); align-items: center; }
.zasob:last-child { border-bottom: 0; }
.zasob audio { height: 30px; width: 240px; }
.kwestia { grid-column: 1 / -1; font-size: 12.5px; padding: 2px 0 0 2px;
border-left: 2px solid var(--accent); padding-left: 8px; }
.maszyna { color: var(--dim); font-size: 11px; font-style: italic; }
/* szachownica pod przezroczystością — inaczej nie widać, gdzie kończy się obrazek */
.kratka { background-image:
linear-gradient(45deg, var(--line) 25%, transparent 25%, transparent 75%, var(--line) 75%),
linear-gradient(45deg, var(--line) 25%, transparent 25%, transparent 75%, var(--line) 75%);
background-size: 12px 12px; background-position: 0 0, 6px 6px; }
.galeria { display: grid; grid-template-columns: repeat(auto-fill, minmax(96px, 1fr));
gap: 8px; margin-top: 6px; }
.kafel { border: 1px solid var(--line); border-radius: 6px; overflow: hidden;
cursor: pointer; background: var(--bg); }
.kafel:hover { border-color: var(--accent); }
.kafel img { display: block; width: 100%; height: 74px; object-fit: contain; }
.kafel .pod { font-size: 10.5px; padding: 3px 4px; color: var(--dim);
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
font-family: var(--mono); }
#zaslona { position: fixed; inset: 0; background: rgba(0,0,0,.72); z-index: 10;
display: flex; align-items: center; justify-content: center; padding: 24px; }
#zaslona > div { background: var(--panel); border: 1px solid var(--line);
border-radius: 10px; padding: 16px; max-width: min(920px, 94vw);
max-height: 92vh; overflow: auto; }
#zaslona img.duzy { max-width: 100%; max-height: 58vh; object-fit: contain;
display: block; margin: 0 auto; }
.postep { display: flex; align-items: center; gap: 10px; margin: 10px 0; }
.postep .bar { flex: 1; height: 10px; }
button.akcja { border: 1px solid var(--line); background: var(--panel); color: var(--ink);
padding: 6px 14px; border-radius: 6px; cursor: pointer; font: inherit; font-size: 12.5px; }
button.akcja:hover:not(:disabled) { border-color: var(--accent); color: var(--accent); }
button.akcja:disabled { opacity: .45; cursor: default; }
</style>
</head>
<body>
<header>
<h1>rex<span>·</span>catalog</h1>
<div id="stats">wczytuję…</div>
<nav>
<button data-view="katalog" class="on">Katalog</button>
<button data-view="skrypty">Skrypty</button>
<button data-view="dzwiek">Dźwięk</button>
<button data-view="formaty">Formaty</button>
</nav>
</header>
<main>
<section id="katalog" class="cols">
<div>
<div class="card">
<input type="search" id="filtr" placeholder="Filtruj tytuły…" autocomplete="off">
</div>
<div class="card"><table>
<thead><tr><th>Tytuł</th><th>Silnik</th><th>Data</th><th>Kopie</th></tr></thead>
<tbody id="tytuly"></tbody>
</table></div>
</div>
<div id="szczegoly"><div class="card empty">Wybierz tytuł z listy.</div></div>
</section>
<section id="skrypty" hidden class="cols">
<div>
<div class="card">
<input type="search" id="szukaj" placeholder="Szukaj w skryptach, np. CANVAS_OBSERVER"
autocomplete="off">
<div class="dim" style="margin-top:8px;font-size:12px">
Składnia FTS5. Identyfikatory z podkreśleniem są jednym tokenem;
użyj <span class="mono">*</span> do przedrostka.
</div>
</div>
<div class="card" id="wyniki"><div class="empty">Wpisz zapytanie.</div></div>
</div>
<div id="podglad"><div class="card empty">Wybierz trafienie, żeby zobaczyć skrypt.</div></div>
</section>
<section id="dzwiek" hidden class="cols">
<div>
<div class="card">
<h2>Transkrypcja mowy</h2>
<div class="dim" style="font-size:12.5px;margin:6px 0 10px">
Rozpoznawanie mowy przez whisper.cpp. Wynik jest <b>wygenerowany maszynowo</b>,
a nie odczytany z płyty — trzymamy go osobno od metadanych wydania.
</div>
<div id="whisper"></div>
</div>
<div class="card">
<input type="search" id="szukajdzwiek"
placeholder="Szukaj nagrania po nazwie, np. kret_e5" autocomplete="off">
<div class="dim" style="margin-top:8px;font-size:12px">
Można też szukać po treści transkryptu, jeśli już jest.
</div>
</div>
<div class="card" id="wynikidzwiek"><div class="empty">Wpisz zapytanie.</div></div>
</div>
<div id="podgladdzwiek"><div class="card empty">Wybierz nagranie.</div></div>
</section>
<section id="formaty" hidden>
<div class="card" style="max-width:620px">
<h2>Rozkład formatów</h2>
<table><thead><tr><th>Format</th><th>Plików</th><th>Rozmiar</th><th></th></tr></thead>
<tbody id="tabformaty"></tbody></table>
</div>
</section>
</main>
<div id="zaslona" hidden></div>
<script>
const $ = s => document.querySelector(s);
const esc = s => String(s ?? '').replace(/[&<>"]/g, c =>
({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
const mb = b => b >= 1048576 ? (b/1048576).toFixed(1)+' MB'
: b >= 1024 ? (b/1024).toFixed(1)+' kB' : b+' B';
const get = async u => { const r = await fetch(u); if (!r.ok) throw new Error(await r.text());
return r.json(); };
const post = async u => { const r = await fetch(u, {method: 'POST'});
if (!r.ok) throw new Error(await r.text()); return r.json(); };
let tytuly = [];
// ---- nawigacja ----
document.querySelectorAll('nav button').forEach(b => b.onclick = () => {
document.querySelectorAll('nav button').forEach(x => x.classList.toggle('on', x === b));
['katalog','skrypty','dzwiek','formaty'].forEach(v => $('#'+v).hidden = v !== b.dataset.view);
});
// ---- katalog ----
function rysujTytuly(filtr = '') {
const f = filtr.toLowerCase();
const widoczne = tytuly.filter(t =>
!f || (t.title||'').toLowerCase().includes(f) || (t.label||'').toLowerCase().includes(f)
|| (t.engine||'').toLowerCase().includes(f) || (t.series||'').toLowerCase().includes(f));
if (!widoczne.length) { $('#tytuly').innerHTML =
'<tr><td colspan="4" class="empty">Nic nie pasuje.</td></tr>'; return; }
$('#tytuly').innerHTML = widoczne.map(t => `
<tr data-id="${t.edition_id}">
<td>${t.series ? `<div class="series">${esc(t.series)}</div>` : ''}
${esc(t.title)}
${t.label ? `<div class="label">${esc(t.label)}</div>` : ''}</td>
<td>${esc(t.engine || '—')}</td>
<td class="mono">${esc(t.release_date || '—')}</td>
<td>${t.copies}</td>
</tr>`).join('');
$('#tytuly').querySelectorAll('tr[data-id]').forEach(tr => tr.onclick = () => {
$('#tytuly').querySelectorAll('tr').forEach(x => x.classList.remove('on'));
tr.classList.add('on');
pokazWydanie(tr.dataset.id);
});
}
async function pokazWydanie(id) {
$('#szczegoly').innerHTML = '<div class="card empty">Wczytuję…</div>';
const e = await get('/api/editions/' + id);
const pliki = await get(`/api/editions/${id}/files?limit=400`);
const jezyki = e.languages.length
? e.languages.map(l => `<span class="pill">${esc(l.lang)} · ${esc(l.role)}</span>`).join('')
: '<span class="dim">nie uzupełniono</span>';
const pole = (k, v) => v ? `<dt>${k}</dt><dd>${esc(v)}</dd>` : '';
$('#szczegoly').innerHTML = `
<div class="card">
<h2>${esc(e.title)}${e.label ? ` <span class="label">${esc(e.label)}</span>` : ''}</h2>
<dl>
${pole('Seria', e.series)}${pole('Wydawca', e.publisher)}
${pole('Dystrybutor', e.distributor)}${pole('Notatki', e.notes)}
${pole('Override silnika', e.engine_override)}
${pole('Override wersji', e.engine_version_override)}
${pole('Override kompilera', e.compiler_override)}
${pole('Override daty', e.release_date_override)}
<dt>Języki</dt><dd>${jezyki}</dd>
</dl>
</div>
<div class="card">
<h3>Kopie na dysku</h3>
${e.copies.map(c => `
<dl style="margin-bottom:12px">
<dt>Plik</dt><dd>${esc(c.display_name)}
${c.root_path ? `<span class="dim">(korzeń ${esc(c.root_path)}/)</span>` : ''}</dd>
<dt>Ścieżka</dt><dd class="mono">${esc(c.path)}</dd>
<dt>Nośnik</dt><dd>${esc(c.container)} / ${esc(c.fs_type || '—')},
${c.file_count} plików, ${mb(c.size)}, status ${esc(c.status)}</dd>
<dt>Źródło</dt><dd>${esc(c.source_kind)}${c.rip_tool ? ` · ${esc(c.rip_tool)}` : ''}
${c.verified ? '<span class="pill">zweryfikowane</span>' : ''}
${c.source_url ? `<br><a href="${esc(c.source_url)}">${esc(c.source_url)}</a>` : ''}</dd>
<dt>SHA-256</dt><dd class="mono">${esc(c.sha256 || '—')}</dd>
</dl>`).join('')}
</div>
<div class="card">
<h3>Wykryte fakty</h3>
<table><thead><tr><th>Pole</th><th>Wartość</th><th>Pewność</th></tr></thead><tbody>
${e.facts.map(f => `<tr style="cursor:default">
<td>${esc(f.field)}</td>
<td>${esc(f.value)}<div class="snip">${esc(f.evidence)}</div></td>
<td class="mono">${f.confidence.toFixed(2)}</td></tr>`).join('')}
</tbody></table>
</div>
<div class="card">
<h3>Pliki (${pliki.length}${pliki.length >= 400 ? ', obcięte' : ''})</h3>
<div class="row">
<input type="search" id="filtrpliki" placeholder="Filtruj po ścieżce…">
<select id="filtrformat" class="narrow"></select>
</div>
<table><thead><tr><th>Ścieżka</th><th>Format</th><th>Rozmiar</th></tr></thead>
<tbody id="tabpliki"></tbody></table>
</div>`;
const formaty = [...new Set(pliki.map(p => p.format).filter(Boolean))].sort();
$('#filtrformat').innerHTML = '<option value="">wszystkie</option>' +
formaty.map(f => `<option>${esc(f)}</option>`).join('');
const rysujPliki = () => {
const q = $('#filtrpliki').value.toLowerCase(), fmt = $('#filtrformat').value;
const w = pliki.filter(p => (!q || p.path.toLowerCase().includes(q))
&& (!fmt || p.format === fmt));
$('#tabpliki').innerHTML = w.length ? w.map(p => `
<tr ${p.decoded ? `data-sha1="${p.sha1}"` : 'style="cursor:default"'}>
<td class="mono">${esc(p.path)}
${p.encrypted ? '<span class="pill">szyfr</span>' : ''}
${p.decoded ? '<span class="pill">podgląd</span>' : ''}</td>
<td>${esc(p.format || '—')}</td>
<td class="mono">${mb(p.size)}</td></tr>`).join('')
: '<tr><td colspan="3" class="empty">Nic nie pasuje.</td></tr>';
$('#tabpliki').querySelectorAll('tr[data-sha1]').forEach(tr =>
tr.onclick = () => otworzSkrypt(tr.dataset.sha1));
};
$('#filtrpliki').oninput = rysujPliki;
$('#filtrformat').onchange = rysujPliki;
rysujPliki();
}
// ---- skrypty ----
let czekaj;
$('#szukaj').oninput = e => {
clearTimeout(czekaj);
const q = e.target.value.trim();
czekaj = setTimeout(() => szukajSkryptow(q), 250);
};
async function szukajSkryptow(q) {
if (!q) { $('#wyniki').innerHTML = '<div class="empty">Wpisz zapytanie.</div>'; return; }
$('#wyniki').innerHTML = '<div class="empty">Szukam…</div>';
const r = await get('/api/search?limit=40&q=' + encodeURIComponent(q));
if (!r.hits.length) {
$('#wyniki').innerHTML = `<div class="empty">Brak trafień dla ${esc(r.query)}.</div>`;
return;
}
// snippet przychodzi z FTS5 z własnymi znacznikami » «, zamieniamy je po ucieczce
const podswietl = s => esc(s).replace(/»/g, '<mark>').replace(/«/g, '</mark>');
$('#wyniki').innerHTML =
`<div class="dim" style="margin-bottom:10px">${r.hits.length} trafień
dla <span class="mono">${esc(r.query)}</span>
${r.quoted ? '— potraktowane jako fraza' : ''}</div>` +
r.hits.map(h => `
<div class="wynik" data-sha1="${h.sha1}"
style="padding:8px;border-radius:6px;cursor:pointer">
<div class="mono">${esc(h.paths[0])}</div>
<div class="dim" style="font-size:12px">${esc(h.games.join(', '))}</div>
<div class="snip">${podswietl(h.snippet)}</div>
</div>`).join('');
$('#wyniki').querySelectorAll('.wynik').forEach(d => {
d.onmouseenter = () => d.style.background = 'var(--hl)';
d.onmouseleave = () => d.style.background = '';
d.onclick = () => otworzSkrypt(d.dataset.sha1);
});
}
async function otworzSkrypt(sha1) {
document.querySelector('nav button[data-view=skrypty]').click();
$('#podglad').innerHTML = '<div class="card empty">Wczytuję…</div>';
const s = await get('/api/scripts/' + sha1);
$('#podglad').innerHTML = `
<div class="card">
<h2 class="mono" style="font-size:13px">${esc(s.paths[0])}</h2>
<div class="dim" style="font-size:12px;margin-bottom:4px">
${esc(s.games.join(', '))}</div>
<div class="mono dim" style="font-size:11px">sha1: ${esc(s.sha1)}</div>
${s.paths.length > 1 ? `<div class="dim" style="font-size:12px;margin-top:6px">
Ten sam plik pod ${s.paths.length} ścieżkami:<br>
<span class="mono">${s.paths.map(esc).join('<br>')}</span></div>` : ''}
</div>
${rysujZasoby(s.refs || [])}
<div class="card"><pre>${esc(s.body)}</pre></div>`;
$('#podglad').querySelectorAll('[data-zasob]').forEach(k =>
k.onclick = () => otworzZasob(k.dataset.zasob));
}
const czas = ms => ms == null ? '' :
ms >= 60000 ? Math.floor(ms/60000) + ' min ' + Math.round(ms%60000/1000) + ' s'
: (ms/1000).toFixed(1) + ' s';
/**
* Zasoby, do których odwołuje się skrypt. Dźwięk dostaje odtwarzacz, bo o to
* chodzi: widząc SNDQUESTION:FILENAME=KRET_E511.WAV chce się od razu wiedzieć,
* co w tym pliku jest — kto mówi, jak długo i co konkretnie.
*/
function rysujZasoby(refs) {
if (!refs.length) return '';
const audio = refs.filter(r => r.format === 'WAV' || r.format === 'OGG');
const grafika = refs.filter(r => r.target && (r.format === 'IMG' || r.format === 'ANN'));
const rysowalne = new Set(grafika.map(r => r.target));
const reszta = refs.filter(r => !(r.format === 'WAV' || r.format === 'OGG')
&& !(r.target && rysowalne.has(r.target) && (r.format === 'IMG' || r.format === 'ANN')));
const brak = refs.filter(r => !r.target).length;
const wiersz = r => {
const m = r.media || {};
const opis = [r.speaker, r.trigger].filter(Boolean).map(esc).join(' · ');
return `<div class="zasob">
<div>
<span class="mono">${esc(r.value)}</span>
<span class="dim" style="font-size:11.5px"> ${esc(r.object)}:${esc(r.field)}</span>
${opis ? `<div class="dim" style="font-size:12px">${opis}</div>` : ''}
${r.uncertain ? `<div class="dim" style="font-size:11px">dopasowanie niepewne —
${r.variants} pliki o tej nazwie</div>` : ''}
</div>
<div>${r.target
? `<audio controls preload="none" src="/api/audio/${r.target}"></audio>
<div class="dim" style="font-size:11px;text-align:right">
${czas(m.duration_ms)} ${m.codec ? '· ' + esc(m.codec) : ''}</div>`
: '<span class="dim" style="font-size:12px">brak na płycie</span>'}</div>
${m.transcript ? `<div class="kwestia">${esc(m.transcript)}
<div class="maszyna">rozpoznane maszynowo — ${esc(m.transcript_model)}</div></div>` : ''}
</div>`;
};
return `<div class="card">
<h2>Zasoby ${refs.length}
${brak ? `<span class="dim" style="font-weight:400;font-size:12px">
· ${brak} bez pliku na płycie</span>` : ''}</h2>
${audio.length ? `<h3>Dźwięk (${audio.length})</h3>${audio.map(wiersz).join('')}` : ''}
${grafika.length ? `<h3>Grafika (${grafika.length})</h3>
<div class="galeria">${grafika.map(kafel).join('')}</div>` : ''}
${reszta.length ? `<h3>Pozostałe (${reszta.length})</h3>
<div style="columns:2;column-gap:20px;font-size:12.5px">
${reszta.map(r => `<div class="mono" style="break-inside:avoid;padding:1px 0">
${r.target ? '' : '<span class="dim">·</span> '}${esc(r.value)}
<span class="dim" style="font-size:11px">${esc(r.format)}</span></div>`).join('')}
</div>` : ''}
</div>`;
}
const kafel = r => {
const m = r.media || {};
const opis = m.kind === 'anim' ? `${m.frames} kl.` : `${m.width}×${m.height}`;
// przy kilku plikach o tej nazwie poza katalogiem skryptu to tylko domysł
const tytul = esc(r.value) + ' · ' + opis
+ (r.uncertain ? ` · niepewne, ${r.variants} pliki o tej nazwie` : '');
return `<div class="kafel" data-zasob="${r.target}" title="${tytul}">
<img class="kratka" loading="lazy" src="/api/preview/${r.target}" alt="${esc(r.value)}">
<div class="pod">${r.uncertain ? '<span title="dopasowanie niepewne">?</span> ' : ''}${esc(r.name)}</div>
</div>`;
};
// ---- dźwięk ----
let czekajD;
$('#szukajdzwiek').oninput = e => {
clearTimeout(czekajD);
const q = e.target.value.trim();
czekajD = setTimeout(() => szukajDzwieku(q), 250);
};
async function szukajDzwieku(q) {
if (!q) { $('#wynikidzwiek').innerHTML = '<div class="empty">Wpisz zapytanie.</div>'; return; }
$('#wynikidzwiek').innerHTML = '<div class="empty">Szukam…</div>';
const r = await get('/api/audio-search?q=' + encodeURIComponent(q));
if (!r.hits.length) {
$('#wynikidzwiek').innerHTML = '<div class="empty">Nic nie pasuje.</div>';
return;
}
$('#wynikidzwiek').innerHTML =
`<div class="dim" style="margin-bottom:8px">${r.hits.length} nagrań</div>` +
r.hits.map(h => `
<div class="wynik" data-sha1="${h.sha1}" style="padding:7px;border-radius:6px;cursor:pointer">
<div class="mono">${esc(h.name)}
<span class="dim" style="font-size:11px">${czas(h.duration_ms)}</span></div>
<div class="dim" style="font-size:12px">${esc(h.copy)}
${h.speaker ? '· ' + esc(h.speaker) : ''}
${h.by_name ? '' : '· trafienie w transkrypcie'}</div>
${h.transcript ? `<div class="snip">${esc(h.transcript)}</div>` : ''}
</div>`).join('');
$('#wynikidzwiek').querySelectorAll('.wynik').forEach(d => {
d.onmouseenter = () => d.style.background = 'var(--hl)';
d.onmouseleave = () => d.style.background = '';
d.onclick = () => otworzNagranie(d.dataset.sha1);
});
}
async function otworzNagranie(sha1) {
$('#podgladdzwiek').innerHTML = '<div class="card empty">Wczytuję…</div>';
const m = await get('/api/media/' + sha1);
const d = m.dialogue[0] || {};
$('#podgladdzwiek').innerHTML = `
<div class="card">
<h2 class="mono" style="font-size:13px">${esc(m.places[0]?.path || sha1)}</h2>
<div class="dim" style="font-size:12px">${esc(m.places[0]?.copy || '')}
${m.places[0]?.in_archive ? '· wewnątrz archiwum wav.snd' : ''}</div>
<div style="margin:12px 0"><audio controls preload="metadata"
src="/api/audio/${sha1}" style="width:100%"></audio></div>
<dl>
<dt>Długość</dt><dd>${czas(m.duration_ms)}</dd>
<dt>Format</dt><dd>${esc(m.codec)} · ${m.sample_rate} Hz ·
${m.channels === 1 ? 'mono' : m.channels + ' kanały'}</dd>
${d.speaker ? `<dt>Mówi</dt><dd>${esc(d.speaker)}</dd>` : ''}
${d.trigger ? `<dt>Zdarzenie</dt><dd class="mono">${esc(d.trigger)}</dd>` : ''}
${d.section ? `<dt>Scena</dt><dd>${esc(d.section)}</dd>` : ''}
<dt>sha1</dt><dd class="mono" style="font-size:11px">${esc(sha1)}</dd>
</dl>
</div>
${m.transcript ? `<div class="card">
<h2>Co słychać</h2>
<div style="margin-top:8px">${esc(m.transcript)}</div>
<div class="maszyna" style="margin-top:8px">rozpoznane maszynowo przez
${esc(m.transcript_model)}, język ${esc(m.transcript_lang)}
to hipoteza modelu, nie zapis z płyty</div>
</div>` : ''}
${m.used_by.length ? `<div class="card">
<h3>Używane przez ${m.used_by.length} obiektów w skryptach</h3>
${m.used_by.slice(0, 12).map(u => `<div style="font-size:12.5px;padding:2px 0">
<a href="#" data-skrypt="${u.script_sha1}" class="mono">${esc(u.object)}</a>
<span class="dim">:${esc(u.field)}</span></div>`).join('')}
</div>` : ''}`;
$('#podgladdzwiek').querySelectorAll('[data-skrypt]').forEach(a =>
a.onclick = e => { e.preventDefault(); otworzSkrypt(a.dataset.skrypt); });
}
// ---- podgląd grafiki ----
let klatki = null;
async function otworzZasob(sha1) {
const m = await get('/api/media/' + sha1);
const anim = m.kind === 'anim';
klatki = anim ? {sha1, ile: m.frames, teraz: 0} : null;
$('#zaslona').innerHTML = `<div>
<div style="display:flex;align-items:baseline;gap:10px;margin-bottom:10px">
<h2 class="mono" style="font-size:13px">${esc(m.places[0]?.path || sha1)}</h2>
<span class="dim" style="font-size:12px;margin-left:auto">${esc(m.places[0]?.copy || '')}</span>
<button class="akcja" id="zamknij">Zamknij</button>
</div>
${m.note && m.note.startsWith('nieczytelny')
? `<div class="empty">${esc(m.note)}</div>`
: `<img class="duzy kratka" src="/api/render/${sha1}${anim ? '?frame=0' : ''}" alt="">`}
${anim && m.frames > 1 ? `<div class="postep" style="margin-top:12px">
<button class="akcja" id="poprz"></button>
<input type="range" id="suwak" min="0" max="${m.frames - 1}" value="0" style="flex:1">
<button class="akcja" id="nast"></button>
<span class="mono" style="font-size:12px" id="numer">1 / ${m.frames}</span>
</div>` : ''}
<dl style="margin-top:12px">
<dt>Wymiary</dt><dd>${m.width}×${m.height}</dd>
${anim ? `<dt>Klatki</dt><dd>${m.frames}${m.fps ? ' · ' + m.fps + ' kl./s' : ''}</dd>` : ''}
<dt>Format</dt><dd>${esc(m.codec)}${m.note && !m.note.startsWith('nieczytelny')
? ' · ' + esc(m.note) : ''}</dd>
${m.author ? `<dt>Autor</dt><dd>${esc(m.author)}</dd>` : ''}
<dt>sha1</dt><dd class="mono" style="font-size:11px">${esc(sha1)}</dd>
</dl>
${m.events && m.events.length ? `<h3>Zdarzenia (${m.events.length})</h3>
<div style="columns:3;column-gap:18px;font-size:12.5px">
${m.events.map(e => `<div style="break-inside:avoid;padding:1px 0">
<span class="mono">${esc(e.name)}</span>
<span class="dim">${e.frames_count} kl.</span></div>`).join('')}
</div>` : ''}
${m.used_by && m.used_by.length ? `<h3>Używane przez ${m.used_by.length} obiektów</h3>
<div class="dim" style="font-size:12px">${m.used_by.slice(0, 10)
.map(u => esc(u.object)).join(', ')}</div>` : ''}
</div>`;
$('#zaslona').hidden = false;
$('#zamknij').onclick = zamknijZaslone;
$('#zaslona').onclick = e => { if (e.target.id === 'zaslona') zamknijZaslone(); };
const suwak = $('#suwak');
if (suwak) {
const pokaz = n => {
klatki.teraz = Math.max(0, Math.min(n, klatki.ile - 1));
suwak.value = klatki.teraz;
$('#numer').textContent = (klatki.teraz + 1) + ' / ' + klatki.ile;
$('#zaslona').querySelector('img.duzy').src =
`/api/render/${klatki.sha1}?frame=${klatki.teraz}`;
};
suwak.oninput = () => pokaz(+suwak.value);
$('#poprz').onclick = () => pokaz(klatki.teraz - 1);
$('#nast').onclick = () => pokaz(klatki.teraz + 1);
}
}
function zamknijZaslone() {
$('#zaslona').hidden = true;
$('#zaslona').innerHTML = '';
klatki = null;
}
document.addEventListener('keydown', e => {
if ($('#zaslona').hidden) return;
if (e.key === 'Escape') zamknijZaslone();
if (klatki && $('#suwak')) {
if (e.key === 'ArrowLeft') $('#poprz').click();
if (e.key === 'ArrowRight') $('#nast').click();
}
});
// ---- transkrypcja ----
let pytajOStan = null;
function rysujWhisper(st) {
const pasek = st.total
? `<div class="postep">
<div class="bar"><i style="width:${(st.done + st.failed) / st.total * 100}%"></i></div>
<span class="mono" style="font-size:12px">${st.done + st.failed} / ${st.total}</span>
</div>
<div class="dim" style="font-size:12px">
${st.current ? 'teraz: <span class="mono">' + esc(st.current) + '</span>' : ''}
${st.failed ? ' · nieudanych: ' + st.failed : ''}
${st.elapsed_ms ? ' · ' + czas(st.elapsed_ms) : ''}
</div>` : '';
$('#whisper').innerHTML = st.ready
? `<div class="dim" style="font-size:12px">
model: <span class="mono">${esc((st.model || '').split('/').pop())}</span>
</div>
${pasek}
<div style="margin-top:10px;display:flex;gap:8px">
<button class="akcja" id="start" ${st.running ? 'disabled' : ''}>
${st.running ? 'Pracuje…' : 'Transkrybuj kolekcję'}</button>
<button class="akcja" id="stop" ${st.running ? '' : 'disabled'}>Zatrzymaj</button>
</div>
${st.state !== 'bezczynny' && !st.running
? `<div class="dim" style="font-size:12px;margin-top:8px">stan: ${esc(st.state)}</div>`
: ''}`
: `<div class="dim" style="font-size:12.5px">${esc(st.tool_note)}</div>
<div class="dim" style="font-size:12px;margin-top:8px">
Katalog działa bez tego — mówcę i kontekst kwestii bierzemy z
<span class="mono">dialogi.dta</span>, a nie z rozpoznawania mowy.
</div>`;
const start = $('#start'), stop = $('#stop');
if (start) start.onclick = async () => {
start.disabled = true;
rysujWhisper(await post('/api/transcribe/start'));
sledzPostep();
};
if (stop) stop.onclick = async () => {
stop.disabled = true;
rysujWhisper(await post('/api/transcribe/stop'));
};
}
function sledzPostep() {
clearInterval(pytajOStan);
pytajOStan = setInterval(async () => {
const st = await get('/api/transcribe/status');
rysujWhisper(st);
if (!st.running) clearInterval(pytajOStan);
}, 1000);
}
// ---- start ----
(async () => {
const s = await get('/api/stats');
$('#stats').textContent =
`${s.editions} wydań · ${s.copies} kopii · ${s.files.toLocaleString('pl')} plików `
+ `· ${s.blobs.toLocaleString('pl')} unikalnych · ${s.scripts} skryptów `
+ `· ${s.audio.toLocaleString('pl')} nagrań (${(s.audio_ms/3600000).toFixed(1)} h) `
+ `· ${s.images.toLocaleString('pl')} obrazów · ${s.animations.toLocaleString('pl')} animacji `
+ `(${s.anim_frames.toLocaleString('pl')} klatek) `
+ `· ${s.refs.toLocaleString('pl')} powiązań · ${mb(s.bytes)}`;
const max = Math.max(...s.formats.map(f => f.n));
$('#tabformaty').innerHTML = s.formats.map(f => `
<tr style="cursor:default"><td>${esc(f.format || '—')}</td>
<td class="mono">${f.n.toLocaleString('pl')}</td>
<td class="mono">${mb(f.bytes)}</td>
<td style="width:170px"><div class="bar"><i style="width:${f.n/max*100}%"></i></div></td>
</tr>`).join('');
rysujWhisper(await get('/api/transcribe/status'));
if ((await get('/api/transcribe/status')).running) sledzPostep();
tytuly = await get('/api/titles');
rysujTytuly();
$('#filtr').oninput = e => rysujTytuly(e.target.value);
})().catch(e => $('#stats').textContent = 'Błąd: ' + e.message);
</script>
</body>
</html>