Skip to content

Conversation

@icex2
Copy link
Contributor

@icex2 icex2 commented Jan 28, 2024

Fundamentally re-think how launcher operates and
bootstrapping the games is managed and configured.

This brings it significantly closer to how the original
bootstrap is doing the job: launcher now utilizes the
data (structures) provided by the bootstrap.xml configuration
file. This creates compatibility with vanilla data dumps
and original stock images. Note that bemanitools does not
include any code or means to run DRM'd data, only decrypted.

But, this allows users to keep decrypted dumps as stock as
possible which means:

  • No copying around of property files anymore
  • Keep the modules/ folder with the binaries
  • Have bemanitools binaries separate in the data
  • No need to edit/customize the original configuration files

A list of key features of the "new" launcher:

  • Boostrap games by following the configuration provided by
    stock game's bootstrap.xml files
  • Custom launcher.xml configuration file that adds further
    launcher configurable features, composability of
    bootstrap.xml configuration(s) as well as configuration
    overriding/stacking of selected types of configurations,
    e.g. eamuse config, avs-config. The latter eliminates
    the need for modifying stock config files in the prop/
    folder
  • Unified logging system: launcher and AVS logging uses
    the same logger, all output can now be in a single file
  • Original features such as various hook types still
    available

Move everything to new launcher.xml configuration
files. Adjust the bootstrapping of launcher in the
.bat files. Features such as copying the default
props/ files to nvram are now handled by launcher.

Using the PATH variable, bemanitools binaries can
live in their own dedicated bemanitools/ subfolder
next to props/ and modules/ now. All original
binaries are expected to be kept in a modules/
folder like on stock data.

Due to the significant architectural changes, this also
breaks with any backwards compatibility to existing
launcher setups. Thus, users need to migrate by re-applying
the new configuration format and migrating their config
parameters accordingly.

Further migration instructions and updated documentation
will be provided upon release.

icex2 added 15 commits January 28, 2024 21:44
We want this to always be visible, so the log level must be
the highest possible. Unfortunately, that's quite the hack
around the existing API and how fatal behaves.

The log API stopped scaling already a while ago and needs
considerable refactoring to consider the various use-cases
that emerged since it was first created on alpha versions
of bemanitools.
Because we are using mingw, we can't just use window's
dbghelp library as the symbols created are in dwarf format.

Fortunately, the dwarfstack library already provides all the
facilities to easily print very descriptive stacktraces,
including function names, file names and line numbers,
when dwarf symbols are available.

This moves the incomplete exception handling portion from
signal to a separate module as well to improve scoping.
Adjust inject to utilize the new feature. This also
requires including the dwarfstack.dll in all distribution
packages.
Expecting to improve debugability significantly. Stacktraces
can be enhanced with function names, file names and line
numbers when symbols are included in all exe and dll files.

Furthermore, this also improves debugger usage as symbols
can be imported to help navigate disassembly/decompiled code.
This has been a source of common error in the past.
It is known that most, or even all, games run into
various issues when not run with elevated privileges.
Use this to share helpers or other extensions to the
original avs API across modules.

Start with including error codes to readable strings
to improve velocity on AVS API error analysis.
Taken from a private eamuse server backend which
had more complete mappings.
Supports improving expressiveness of the API interface
These were previously missing and are required for
various file system related tasks such as iterating
directory trees, reading and writing files through
the AVS file system for the upcoming launcher
rework.

Note that the AVS API broke with some mode flags
after version 2.13.06.
Again, required for the launcher rework when dealing
with property node trees.
Add all functions and their respective ordinals (and mangled names for
documentation purpose) to all currently used AVS version.
This one was missing and is the actual correct
version used for ddr-13. 2.13.04 was compatible,
thus far, but there isn't any guarantee that they
are actually 100% compatible (only konmai knows...).
icex2 added 2 commits January 29, 2024 22:34
Apparently forgotten to get updated to reflect the
currently supported versions correctly.
Apparently also forgotten to reflect currently
supported games.
@icex2 icex2 force-pushed the launcher-refactoring branch from 5bb84c8 to ff9f8d8 Compare January 29, 2024 21:35
@icex2 icex2 changed the title Launcher refactoring "New" launcher, bootstrap.xml compatible Jan 29, 2024
@icex2 icex2 marked this pull request as ready for review January 29, 2024 21:36
@icex2 icex2 force-pushed the launcher-refactoring branch from ff9f8d8 to 4d21f75 Compare January 29, 2024 21:40
@icex2
Copy link
Contributor Author

icex2 commented Jan 29, 2024

Remark: This is currently not rebased to #278 which is pre-work required for the major refactoring. Just the last two commits account for the actual refactoring work relevant in this PR. I will rebase accordingly to clean-up the history once #278 is merged.

@icex2
Copy link
Contributor Author

icex2 commented Jan 29, 2024

These are still open TODOs and feedback that are not addressed by this PR but will be followed-up to once this is merged.

  • Complete addressing concerns of Log system information #57
  • add development build container
  • rework logging system, issues with non-async logging and lagging the game (stdout and file must be written async),
    implement features such as log ration etc. as outlined by the bootstrap logging config
  • have some sort of migration guide to switch to the new launcher. part of that should be proper
    documentation about how to use the launcher the configuration xml file and what you can do
    with it with various use-case examples
  • align layout of distribution files, different for games such as iidx, jubeat and sdvx
  • config format with stacking and overriding doesn't work here:
    <fs>
        <nr_filesys __type="u16">16</nr_filesys>
        <nr_mountpoint __type="u16">1024</nr_mountpoint>
        <nr_mounttable __type="u16">32</nr_mounttable>
        <nr_filedesc __type="u16">1024</nr_filedesc>
        <link_limit __type="u16">8</link_limit>
        <root>
            <device __type="str">.</device>
        </root>
        <mounttable_selector __type="str">boot</mounttable_selector>
        <mounttable>
            <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/raw" dst="/dev/raw" opt="vf=1,posix=1"/>
            <vfs name="boot" fstype="fs" src="D:/LDJ/contents/dev/nvram" dst="/dev/nvram2" opt="vf=0,posix=1"/>
            <vfs name="boot" fstype="nvram2" src="/dev/nvram2" dst="/dev/nvram" opt="nr_mirror=4"/>
        </mounttable>
    </fs>

vfs nodes are being replaced as they are not unique by node name -> add tag __replace="true" or something similar to

icex2 and others added 2 commits February 3, 2024 15:19
Kudos to Shiz for providing the groundwork for this.

Fundamentally re-think how launcher operates and
bootstrapping the games is managed and configured.

This brings it significantly closer to how the original
bootstrap is doing the job: launcher now utilizes the
data (structures) provided by the bootstrap.xml configuration
file. This creates compatibility with vanilla data dumps
and original stock images. Note that bemanitools does not
include any code or means to run DRM'd data, only decrypted.

But, this allows users to keep decrypted dumps as stock as
possible which means:

* No copying around of property files anymore
* Keep the modules/ folder with the binaries
* Have bemanitools binaries separate in the data
* No need to edit/customize the original configuration files

A list of key features of the "new" launcher:

* Boostrap games by following the configuration provided by
  stock game's bootstrap.xml files
* Custom launcher.xml configuration file that adds further
  launcher configurable features, composability of
  bootstrap.xml configuration(s) as well as configuration
  overriding/stacking of selected types of configurations,
  e.g. eamuse config, avs-config. The latter eliminates
  the need for modifying stock config files in the prop/
  folder
* Unified logging system: launcher and AVS logging uses
  the same logger, all output can now be in a single file
* Original features such as various hook types still
  available

Due to the significant architectural changes, this also
breaks with any backwards compatibility to existing
launcher setups. Thus, users need to migrate by re-applying
the new configuration format and migrating their config
parameters accordingly.

Further migration instructions and updated documentation
will be provided upon release.

Co-authored-by: Shiz <hi@shiz.me>
Move everything to new launcher.xml configuration
files. Adjust the bootstrapping of launcher in the
.bat files. Features such as copying the default
props/ files to nvram are now handled by launcher.

Using the PATH variable, bemanitools binaries can
live in their own dedicated bemanitools/ subfolder
next to props/ and modules/ now. All original
binaries are expected to be kept in a modules/
folder like on stock data.
@icex2 icex2 force-pushed the launcher-refactoring branch from 4d21f75 to a237454 Compare February 3, 2024 14:20
@icex2 icex2 changed the base branch from master to development February 19, 2024 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant