Python API¶
The command line is built on the modules below; scripts can use them directly. The names each module lists in its
__all__ are its documented API, and the most used ones are also importable from the top-level package:
from nnnotes import Config, Session
session = Session(Config.load()) # the config file and the NNNOTES_* environment variables
catalog = session.catalog() # the region's catalog, with its bundles
master = session.master_dir() # the decoded master data directory
A missing or invalid setting raises ConfigError, naming the setting and never its value; a failed game API call
raises GameApiError. Other names are internal and can change in any version.
Configuration and sources¶
Settings from a TOML file, environment variables and command-line flags.
Sources, lowest to highest precedence:
- the TOML file:
--config <file>, else the file named byNNNOTES_CONFIG, else./nnnotes.tomlwhen present, else the per-user file (user_file) when present; - environment variables
NNNOTES_<SECTION>_<KEY>: the setting's dotted name upper-cased, dots and dashes as underscores (servers.tw.cdn->NNNOTES_SERVERS_TW_CDN,bundle.nonce_seed->NNNOTES_BUNDLE_NONCE_SEED); - command-line flags.
No setting has a default and an empty value counts as unset. A command that needs a setting nobody gave stops with a ConfigError naming the TOML key and the environment variable. Values are never printed, logged or put into an error message. Relative paths in the TOML file are relative to the file's directory; relative paths from the environment or the command line are relative to the working directory.
Bases: Exception
A setting is missing or malformed (the message names the setting, never its value).
Merged settings. overrides: {(section, key): value} from command-line flags (None = not given).
origin(section: str, key: str) -> str | None
Where a setting comes from: 'flag', 'env', 'file', or None when it is unset.
flag(section: str, key: str) -> str | None
The command-line flag of a setting, when it has one.
file_keys() -> list[tuple[str, str]]
(section, key) of every value of the TOML file, a nested table as a dotted section ('' at the top).
env_names() -> list[str]
The names of the NNNOTES_ environment variables that set something (never their values).
get_list(section: str, key: str) -> list[str]
A list of strings (TOML array; comma-separated in the environment or on the command line).
hex(section: str, key: str, size: int | None = None) -> bytes
A required hex string as bytes (size: the exact byte length).
regions() -> list[str]
Region names: the [servers.
catalog_pin(region: str | None = None) -> tuple[str, str, str] | None
The pinned resource version of an international catalog, (value, section, key) of the setting that pins
it: the --catalog-version flag, else [servers.<region>] catalog_version, else [catalog] version; None
when nothing pins it (session.Session.catalog_version asks the API then).
master(region: str | None) -> tuple[str, str]
The setting that names the decoded master data of region: the --master flag, else
[servers.<region>] master, else [paths] master -> (section, key).
user_file(environ: dict[str, str] | None = None) -> Path | None
The per-user config file: %APPDATA%\nnnotes\nnnotes.toml on Windows, else
$XDG_CONFIG_HOME/nnnotes/nnnotes.toml (~/.config when XDG_CONFIG_HOME is unset); None when the environment
names no such directory.
config_files(path: str | Path | None = None, environ: dict[str, str] | None = None) -> list[tuple[str, Path | None]]
The config file candidates in lookup order, (how it is named, path or None): --config, NNNOTES_CONFIG, the
working directory's nnnotes.toml, the per-user file. The first two are used when they are given (and must
exist), the last two when they exist.
The data sources of one configuration: the catalog, the decoded master data, the APK and the keys.
A Session reads a setting when a command first needs it, and names the setting in a ConfigError when it is missing
or wrong. The game API answers a Session needs (the master version, the resource version of an international
catalog) come from the region's [servers.<region>] api; JP regions go through jp.Client.
The data sources cfg names. region arguments select another configured region; None: [catalog] region.
path(section: str, key: str, kind: str = 'file') -> Path | None
The path a setting names, checked to exist (kind "file" or "directory"; an APK may be a directory of
splits); None when the setting is unset.
store_root() -> Path
The store directory: [paths] store, else the store/ directory of [paths] cache.
master_dir(region: str | None = None) -> Path
The decoded master data of region: the --master flag, else [servers.<region>] master, else
[paths] master.
master_version(region: str | None = None, *, timeout: float = gameapi.TIMEOUT) -> gameapi.MasterVersion
The master data version region serves now (its [servers.<region>] api).
catalog_version(region: str | None = None) -> str
The resource version of an international catalog: the --catalog-version flag, the regional or global pin,
the resource version the region's API serves, else main.
catalog(bundles: bool = True, region: str | None = None) -> Catalog
The catalog of [catalog] language (merged with the APK's when [paths] apk is set), fetching from the CDN
of region. International versioned catalogs are isolated by CDN root and resource version. bundles:
bundles will be fetched; else only the catalog is read. An explicit [paths] catalog file bypasses
discovery; a version pin bypasses the API but needs the CDN root to identify the cache. The bundle key is
only read for a missing encrypted bundle.
Addressables catalog access: dependency closures, a bundle cache, remote and APK-local bundles.
Remote bundles are fetched from the region's CDN and decrypted (addressables.decrypt); local bundles (shipped inside the APK) are read from a user-supplied base.apk. Nothing here bundles or redistributes game data: it only reads what the user points it at into a local cache the user controls.
Remote content catalog, merged with the APK's local catalog when an APK is given.
The game loads both: the APK catalog (embedded assets, e.g. ADV settings and shared prefabs) and the remote catalog (downloadable content). Entry offsets are per-catalog, so APK entries are moved into their own offset namespace.
cdn: the region's CDN base (needed only to download what the cache lacks); bundle_key: the bundle
decryption key (needed only for bundles not yet in the cache). Either may be a function that returns it, called
the first time it is needed (a ConfigError it raises is raised naming the file that needed the setting).
apk_catalog: the stored catalog to replay without opening an APK; JP local reads verify it before extraction.
property
¶
cdn: str | None
The CDN base as given (None while it is given as a function not called yet).
check_apk(archive) -> None
Before extracting a JP local file, verify the APK against this catalog's snapshot.
local_cache_dir() -> Path
JP embedded files also depend on the APK catalog, independently of the CDN snapshot.
classmethod
¶
load(language: str, cache_dir: Path, *, cdn=None, bundle_key=None, apk: Path | None = None, version: str = 'main') -> Catalog
The remote catalog of language from the cache, downloaded from the CDN on first use (cdn /
bundle_key: as for Catalog). A versioned catalog is isolated by CDN root and version; main retains
its legacy offline cache path. Missing versioned files never fall back to main.
staticmethod
¶
cache_file(language: str, cache_dir: Path, *, version: str = 'main') -> Path
Where the remote catalog of language is cached.
entry(key: str) -> dict
The location of key: its asset location (internal id under Assets/ or Packages/) when it has one, else
its first location. KeyError when the catalog lacks the key.
dependencies(key: str) -> list[dict]
The locations the location of key (entry) depends on, in its dependency order (those the catalogs
hold).
staticmethod
¶
as_bundle(e: dict) -> Bundle | None
The bundle file a location names (a remote or APK-local .bundle), else None.
sources() -> dict[str, bytes]
The catalog files this catalog was made of: {"remote": bytes, "apk": bytes (with an APK)}.
locations() -> list[dict]
Every location of the remote catalog and (with an APK) of the APK catalog, fully decoded
(addressables.parse_locations), with catalog ("remote" / "apk") and kind (location_kind). Offsets and
dependencies are those of entries (APK ones in their own namespace).
bundles() -> list[Bundle]
Every bundle file the catalogs name (remote and APK-local), one per file name, sorted by name. A file the APK catalog lists gets the APK catalog's location (the remote catalog's references to APK files may describe another build), else its first location.
raw_files() -> list[dict]
Every raw file the catalogs name (a CDN or APK file that is not a bundle, e.g. CRI ACB/AWB/USM data), one entry per file name (file_name), sorted by name, preferring the APK catalog's as bundles() does. Remote ones are what fetch_raw takes.
resolve(key: str) -> list[Bundle]
Full bundle closure for an addressable key (BFS over dependencies).
cached(b: Bundle) -> Path | None
The file fetch(b) returns when the bundle is in the cache already, else None.
cached_raw(e: dict) -> Path | None
The file fetch_raw(e) returns when it is in the cache already, else None.
fetch_key(key: str, skip=frozenset()) -> list[Path]
Every bundle of the key's closure but those named in skip; APK-local ones only when an APK is set.
bundle_of(key: str) -> Bundle
The bundle that holds the asset key: the first bundle among its location's dependencies (a bundle key:
that bundle).
fetch_raw(e: dict) -> Path
A non-bundle CDN asset (e.g. CRI ACB/AWB data) as stored -- no decryption.
apk_bundle(name_contains: str) -> Path
An APK-local addressable bundle whose file name contains the substring.
Addressables content catalogs, asset bundle decryption and a local catalog browser.
A remote content catalog (catalog_main_<language>.bin, Addressables binary catalog format version 2) lists every
addressable key with its location: an asset path inside a bundle, a bundle, or a raw file. Remote locations are
absolute URLs on a placeholder host that the game replaces with its CDN base at load time. parse reads what the
extractors need (keys, internal ids, dependencies); parse_locations decodes every field of every location,
including the AssetBundleRequestOptions of bundle and raw file locations (content hash, CRC, size, internal bundle
name), and parse_header / parse_keys the header and the key table.
Asset bundles on the CDN and inside the APK are encrypted in their first 16 KiB with AES-128 in CTR mode: the nonce
is the first 8 bytes of SHA-256(nonce seed + UTF-8 bundle file name), the counter block is nonce + 64-bit big-endian
counter starting at 0. A file that already starts with the UnityFS signature is not encrypted. The key and the
nonce seed come from the configuration ([bundle] key, [bundle] nonce_seed).
dataclass
¶
AES-128 key and nonce seed of the bundle encryption (never shown in a repr).
decrypt(data: bytes, filename: str, key: BundleKey) -> bytes
A bundle file as stored -> UnityFS bytes. filename: the bundle's bare file name (it seeds the nonce).
parse(data: bytes) -> list[dict]
Every location of a binary catalog: {offset, primary_key, internal_id, dependencies (location offsets)}.
parse_header(data: bytes) -> dict
The header of a binary catalog: magic, version, keysOffset, locatorId, instanceProvider, sceneProvider ({id, assembly, type, data}), initObjects (the same, the providers the catalog initializes), buildResultHash.
parse_keys(data: bytes) -> list[dict]
The key table of a binary catalog in stored order (ContentCatalogData.ResourceLocator.KeyData {u32 key object, u32 location set}): {"key": value, "type": the key's type name, "locations": location offsets}.
parse_locations(data: bytes) -> list[dict]
Every location of a binary catalog, fully decoded, by offset: what parse() gives (offset, primary_key, internal_id, dependencies) plus provider, dependency_hash, resource_type (class names) and extra_data.
A location record (ResourceLocation.Serializer.Data) is a fixed 28-byte value with no length prefix: primary key and internal id (strings read with '/'), provider (read with '.'), dependency set (u32 array of location offsets), dependency hash (i32), extra data (an ObjectTypeData) and resource type (a TypeSerializer.Data). extra_data of bundle and raw file locations is an AssetBundleRequestOptions: {type, hash, bundleName, crc, bundleSize, timeout, redirectLimit, retryCount, flags and the flag bits by name}.
Master data files -> JSON tables.
A master data file (<Table>.bin) is a 64-byte prefix followed by the table encrypted with Rijndael (256-bit
block, 256-bit key, CBC, PKCS7 padding); the plaintext is gzip-compressed UTF-8 JSON ({"_allData": [rows]}).
The game reads it in MasterDataModelBox<,>.Load: RijndaelEncryption.FixedDecrypt (the data after the prefix,
the CryptProvider key and IV), then Compression.Decompress (GZipStream), then JsonUtility.FromJson.
AES is Rijndael with 128-bit blocks only, so the 256-bit block cipher is implemented here (numpy, every block of a
file at once). Key and IV come from the configuration ([master] key, [master] iv).
download fetches a master data version from the region's CDN: <cdn>/master/<version>/MasterManifest.json
({"version", "files": [{"name", "hash", "size"}]}) and the files it lists, each checked against its SHA-256.
The version a region serves now comes from the game's API (session.Session.master_version).
dataclass
¶
Key and IV of the master data encryption (never shown in a repr).
Bases: RuntimeError
A CDN download failed (the message names the file, not the URL).
decode(raw: bytes, key: MasterKey, rk: ndarray | None = None) -> bytes
A master data file as stored -> its JSON text (UTF-8 bytes).
decode_files(files: list[Path], out_dir: Path, key: MasterKey, workers: int = 8) -> dict
Decode master data files into
table(master_dir: Path, name: str) -> list[dict]
The rows of a decoded master table: _allData of
has_row(master_dir: Path, name: str, row_id) -> bool
Whether the decoded master table name has a row whose _id is row_id.
download(cdn: str, version: str, out_dir: Path, workers: int = 16, *, get=None, strict=False) -> dict
Master data version from the CDN into
Anonymous calls to the game's API: the master data version a region serves, and the server list.
The game's API is gRPC over HTTP/2 with TLS. Two unary methods can be called anonymously; both take an empty request message:
app.masterdata.MasterdataService/Version, on a region's API root
response string version = 1 (the master data version: the <version> of <cdn>/master/<version>/) and
string resource_version = 2.
GetServerList, on the bootstrap API root
response repeated ServerInfo servers = 1; of ServerInfo only string name = 1, string cdnRoot = 2,
string apiServerRoot = 3, string displayName = 7 and string areaID = 8 are read. A root field can hold
several alternative roots separated by |.
Every call sends the metadata the game client sends with each call: x-request-id (a random UUID), x-platform
(android) and x-client-version (the client version). Requests and responses are raw bytes; the few fields above
are read with a small protobuf wire-format decoder, without generated code. A call has a deadline and is retried when
the server is unreachable, misses the deadline or answers with an empty message. Error messages name the method and
the setting, never the address.
Bases: RuntimeError
A game API call failed (the message names the method and the setting, never the address).
dataclass
¶
An entry of the server list. cdn / api: the alternative roots in the server's order (not in the repr).
fetch_master_version(root: str, client_version: str, *, timeout: float = TIMEOUT, setting: str = 'the API root') -> MasterVersion
The master data version the API root root serves now (an anonymous Version call with client_version);
a GameApiError naming setting when the call fails or the answer has no version.
fetch_server_list(root: str, client_version: str, *, timeout: float = TIMEOUT, setting: str = 'the bootstrap API root') -> list[Server]
The server list the bootstrap API root root serves (an anonymous GetServerList call with
client_version); a GameApiError naming setting when the call fails or the list is empty.
apk_version_name(apk: Path) -> str | None
versionName of the APK's AndroidManifest.xml (None when the APK has none).
Japanese release: anonymous Version discovery and snapshot-bound CDN downloads.
Addresses and client versions come from Config. Credentials remain in a Client, never in source metadata, cache records, task descriptions or exception messages.
The JP game API and CDN of one configured region: observe() asks the API for the current master version,
asset source and CDN credential; get() downloads a file of one source or master version with that credential,
observing again when it is older than five minutes.
origin(value)
The origin (scheme://host[:port], without a default port) of an HTTPS URL with no path, query or
credentials (HTTP only on a loopback host); a ConfigError for any other value.
Music data¶
Music data: one JSON file with every live song and chart of one master data version (format
nnnotes.music-data/2, docs/music-data.md): titles and credits in every language, bands, vocal characters, category,
tags, release time, score ranks, the live BGM's length, the Gekisou catalog (member cards, snaps and their Gekisou
skills), per difficulty the chart facts (level, note counts, BPM, chart times, skill events, fever ranges) and the
chart's deck statistics, what the chart contributes to the live score whatever the deck, with Gekisou on (a Gekisou
live, every rank) and off (a solo live), measured by the deck model ournotes-deck (the extension module
the package ournotes-sim).
Master data is read from the files as served (deckdata.master_files / apk_master: SHA-256 checked against the
manifest, decoded with master.decode). Charts are the TextAssets Live/MusicScore/<file name> converted by
score.runtime_score; the BGM length is read from the cue sheet's ACB (cue Length and the stream's sample count),
without decoding audio. The deck model reads the deck input (deckdata.build: the charts' runtime notes and the master
data tables it needs) in memory; its statistics are checked against the chart facts and the master data. With full
the file also carries that deck input (master, charts), every chart's runtime notes and the tables. With a jackets
directory, every song's jacket (the Texture2D Image/Jacket/<jacket>) is written there as <jacket>.webp, scaled to
at most JACKET_SIZE pixels on its longer side.
The output is canonical (deckdata.encode): minified UTF-8 with one trailing LF, keys in a fixed order, songs sorted by id, master data floats as the shortest decimal of their binary32 value, the deck model's numbers as it writes them. The same inputs and deck model give the same bytes. The file is written only when every table, chart and cue sheet was read and every chart measured; any missing or unreadable input is a MusicDataError naming it.
Bases: ValueError
An input the music data file cannot be made from (the message names it).
The deck model (the package ournotes-sim): native chart expectations. seeds selects replay examples; it does
not sample expectations.
With cache, the native model persists reusable simulation programs and runs. Python passes the complete
current input on every call and publishes the native cache counters separately from the music data.
published_seeds(n: int) -> list[int]
The first n seeds of ournotes-deck's published seed set (a smaller set is a prefix of a larger one).
validate_expectation(stats: dict, kinds: int, where: str) -> None
Check a chart's deck statistics (charts[].deck with the chart's judgedNotes, numbers read by
deckdata.loads): the nominal expectations of the Gekisou lives with their checks, and the deterministic Free Live
measurements. Raises MusicDataError naming where; kinds: the number of skill kinds.
validate_aptitude_statistics(header: dict | None, stats: dict, where: str) -> dict | None
A chart's gekisouAptitude after checking it: null exactly when the aptitude is left out, the master data
has no shape, the chart is unplayable with Gekisou or has no Gekisou range; else a factor per range and a
variant per shape of the chart's missions (both band match results of a shape with a band condition), each
with finite outward estimates, consistent tail identities and its expectation check within
its bound.
build(tables: dict[str, list[dict]], table_sha: dict[str, str], fetch: Callable[[str], bytes], bgm: Callable[[str, str], dict] | None, *, region: str, client: dict, catalog: dict, master_source: str, master_version: str | None, deck: Deck | None = None, full: bool = False) -> dict
The music data document. fetch(file name): a chart TextAsset's bytes (KeyError when there is none);
bgm(cue sheet, cue): the BGM length (catalog_bgm), None to leave every song's bgm.length null; deck: the
deck model measuring the songs' charts (and their Gekisou aptitude), None to leave every chart's deck null;
full: add the deck input (master, charts: deckdata.TABLES and every chart's runtime notes). tables:
tables_of(deck, full).
export(out, src: MasterSource, key, fetch: Callable[[str], bytes], bgm: Callable[[str, str], dict] | None, *, region: str, client: dict, catalog: dict, deck: Deck | None = None, full: bool = False, jacket: Callable[[str], bytes] | None = None, jackets_dir=None, replay_dir=None, replay_engine=None, recommend_engine=None) -> dict
Read the master data, every chart and every BGM cue sheet, measure the charts with deck, then write the file
out (gzip when it ends in .gz) through a temporary file and a rename; with jacket and jackets_dir, first
every song's jacket as <jackets_dir>/<jacket>.webp. Returns the summary.
Deck input: every live chart as the client builds it at runtime and the master data tables the deck model
(ournotes-deck) reads, for one master data version. nnnotes music-data hands it to the deck model in memory
(DECK_FORMAT, the reader's format) and writes it into the music data file with --full (docs/music-data.md).
Master data is decoded from the files as served: a directory with MasterManifest.json and the .bin files it
lists (nnnotes master download), or the same layout inside the APK (assets/Master/). Each file is checked against
the manifest's SHA-256 and decoded with master.decode. Master data decoded elsewhere is read as it is: a directory of
decoded tables (<Table>.json) with the MasterManifest.json of the files they were decoded from, whose version and
SHA-256 are taken as the manifest lists them (decoded_master). Charts are the TextAssets
Live/MusicScore/<file name> of every MasterLiveMusicScore row, read from the catalog and converted by
score.runtime_score; notes are listed in the order the client enumerates them.
The encoding is canonical (encode): minified UTF-8 with one trailing LF, keys in a fixed order, charts sorted by
score id, numbers the master data writes with a fraction or exponent as the shortest decimal that reads back as the
same binary32 value (infinity as 1e999 / -1e999; a NaN is an error). The same inputs give the same bytes; a .gz
output is gzip with no file name and a zero modification time. Any missing or unreadable input is a DeckDataError
naming it.
Bases: ValueError
An input the deck data cannot be made from (the message names it).
dataclass
¶
Master data files as served: version and the SHA-256 of each file name from the manifest, and a reader of
files by name. decoded: the reader has the decoded tables instead (<Table>.json for <Table>.bin).
Bases: str
A number token of a JSON document, written back as it is.
master_files(directory) -> MasterSource
The master data files of a directory written by nnnotes master download (MasterManifest.json + .bin).
decoded_master(directory) -> MasterSource
Decoded master data: a directory of decoded tables, <Table>.json with the _allData rows (as nnnotes master
decode writes them), and the MasterManifest.json of the files they were decoded from (a master data snapshot
published with its manifest). The version and each file's SHA-256 are the manifest's: the decoded tables cannot
be checked against the files as served.
apk_master(apk) -> MasterSource
The master data files the APK ships (assets/Master/: MasterManifest.json + .bin).
loads(text: str | bytes)
The JSON document text with every non-integer number kept as its token (a Num), so that checks and writes
see the number as written.
Site and schemas¶
Live2D models -> the model part of an ournotes-player site (web.py builds the charts of the same site).
<site>/models.json model index: id, manifest path, key, listing facts, sizes
<site>/models/<id>.json model manifest (format sitefiles.MANIFEST_FORMAT): every path of the model's files ->
{asset, size}, or, for a large JSON object, {parts: [[key, asset, size], ...], size}
(as a chart manifest)
<site>/assets/<sha256>.<ext> content-addressed files, shared with the charts (the models share their shaders)
The files of one model (the paths in its manifest), only those the player reads (read_files):
model.json index: format (MODEL_FORMAT), name, key, moc3, prefab, textures, canvas, shaders,
resources, motionSync
<name>.moc3 the moc3 (as live2d.extract_runtime writes it)
<name>.prefab.json the whole prefab, clips, expressions and controllers inlined (the same)
textures/*.png the atlas pages the drawables draw with (the same)
shaders/shaders.json the shader index (shader.py layout, filtered as in the chart sites:
sitefiles.collect),
shaders/<shader>.json the parsed forms and the GLES3 programs (GLSL ES 3.00) the drawables' materials
shaders/<shader>/gles3/*.glsl select by their keywords, each also with STORY_KEYWORDS (the keywords the story
renderer adds to a character draw); the mask shader's only when a drawable is masked
resources in model.json: the materials the Cubism mask pass draws with (the Resources.Load paths of RESOURCES, read
from the APK's boot data). motionSync: the prefab's root has the MotionSync controller with its CRI audio input
(has_motion_sync); a model without it takes its mouth in a story from the voice's CRI Lips analysis.
Models are the catalog keys Character/Live2D/force; an outdated one (outdated: its model.json older than MODEL_FORMAT or without
motionSync, or unreadable) is built again. Same inputs and library versions give byte-identical outputs. A model's
files read no master data and the regions serve the same catalog, so one build serves every region of a site (the
bundles are fetched from the CDN of region).
Names (optional): when master data is configured for region, the listing of each model the master data maps to a
character (model_names: MasterCharacterCostume -> MasterCharacter -> MasterText) gets character, names and
label; the manifests of skipped models get the same fields of this build. Without master data the fields are not
written (and skipped manifests keep theirs).
Model sources: story.build takes the models of an episode from a source, ensure(address) -> {"id", "motionSync"}
plus story_fields(story_dir) (what story.json gets besides models). ModelDir exports each model into
nnnotes export story), SiteModels reads model.json from the model manifests of a site (the site's
stories).
The JSON Schemas (draft 2020-12) of the documents nnnotes writes, shipped with the package: <name>.schema.json,
each with the $id urn:nnnotes:schema:<name> that the others reference.