feat: bemanitools 6, wip (TBD needs splitting and reviewing) #305

icex2 · 2024-06-12T16:40:17Z

THIS IS A HIGHLY WORK/DEVELOPMENT IN PROGRESS VERSION

THINGS ARE BROKEN AND EVERYTHING IS SUBJECT TO CHANGE

First cut after massive refactoring with most critical and fundamental changes implemented.
We need to start somewhere, and this might be as good as anything else, so we can get started
with testing, bug fixing and iterating for the next releases.

The following list is non-exhaustive, does not guarantee anything does work, yet, and is supposed
to give a high level idea of what all of this is about. Updated documentation will reflect all of
this at some later point in time in more detail.

A common "core" now abstracts logging, thread, property and configuration infrastructure and
provides a common interface. This is used by bemanitools internally as well as all tools, hooks
and APIs provided and don't depend on the game, version of the game or AVS version available
anymore
New bemanitools (public) API
- Versioned API allowing for handling incremental API changes as well as breaking changes by
  providing a new/different version when necessary
- Unified interfaces for bemanitools core API, i.e. logging, threads, configuration
- SDK with examples (TBD)
- Dogfooding approach: Bemanitools uses its own (public) API to implement and provide fundamental
  features like configurable keyboard implementations for IO or hooks for different games and
  versions
All bemanitools hooks and IO libraries have been or are about to be re-worked to use the new APIs
New hook API allows for more fine grained runtime control when stages of the hook are to be
executed, i.e. pre AVS, before main game, iat hooking instead of relying purely on DllMain
(which is still a compatible option though)
launcher as a replacement for bootstrap: Bring it significantly closer to the original bootstrap
by supporting completely vanilla data and bootstrap.xml configurations to run the games. Note
that bemanitools does not include any code or means to run DRM'd data, only decrypted
inject is also being reworked to use as much of the same "infrastructure" as launcher to provide
a more seamless bootstrapping process for games that keeps pre-eapki data as vanilla as possible

This module contains the "core" (API) of bemanitools which includes an abstraction layer for threads and logging at this time. The threads API is very close to what util/thread already was with some structural enhancements which make it easier to understand and work with the API, I hope. Some additional helpers (*-ext module) support in doing common tasks, e.g. setting up the thread API with other modules. The log(ging) part receives a major overhaul to address known limitations and issues with the util/log module: - Cleaner API layer - Separate sinks from actual logging engine - Sinks are composable - Improved and cleaner compatibility layer with AVS logging API Additional "extensions" (*-ext modules) add various helper functions for common tasks like setting up the logging engine with a file and stdout sink. The sinks also improved significantly with the file sink now supporting proper appending and log rotation. Logging to stdout/stderr supports coloring of log messages which works across logging engines. Overall, this refactored foundation is expected to support future developments and removes known limitations at the current scale of bemanitools such as: - Reducing boiler plate code across hooks - Interop of bemanitools and AVS (and setting the foundation for addressing currently missing interop, e.g. for dealing with property structures without AVS) - Addressing performance issues in the logging engine due to incorrect interop with AVS

Doesn't really reduce boiler plate but adds clarity with a more meaningful function name what the operation does.

Split files and add name spacing.

Boils down to: - Include headers - Reduce boiler plate with helpers - Swap out explicit usages with core API layer and ensure the right API is configured beforehand

The log API stopped scaling already a while ago and needs considerable refactoring to consider the various use-cases that emerged since it was first created on alpha versions of bemanitools.

Keep this a separate commit because this also removes inject's own logging engine and replaces it with the streamlined core API. The core API provides all the features of inject's own logging engine which also performed horribly. The entire logging operation was locked which included expensive operations that formatted the log messages and required memory allocations and copying around data. The core API's implementation at least only synchronizes the actual IO operations (though this can be improved further with an actual async logging sink, TBD)

Kudos to Shiz for providing the groundwork for this. Fundamentally re-think how launcher operates and bootstrapping the games is managed and configured. This brings it significantly closer to how the original bootstrap is doing the job: launcher now utilizes the data (structures) provided by the bootstrap.xml configuration file. This creates compatibility with vanilla data dumps and original stock images. Note that bemanitools does not include any code or means to run DRM'd data, only decrypted. But, this allows users to keep decrypted dumps as stock as possible which means: * No copying around of property files anymore * Keep the modules/ folder with the binaries * Have bemanitools binaries separate in the data * No need to edit/customize the original configuration files A list of key features of the "new" launcher: * Boostrap games by following the configuration provided by stock game's bootstrap.xml files * Custom launcher.xml configuration file that adds further launcher configurable features, composability of bootstrap.xml configuration(s) as well as configuration overriding/stacking of selected types of configurations, e.g. eamuse config, avs-config. The latter eliminates the need for modifying stock config files in the prop/ folder * Unified logging system: launcher and AVS logging uses the same logger, all output can now be in a single file * Original features such as various hook types still available Due to the significant architectural changes, this also breaks with any backwards compatibility to existing launcher setups. Thus, users need to migrate by re-applying the new configuration format and migrating their config parameters accordingly. Further migration instructions and updated documentation will be provided upon release. Co-authored-by: Shiz <hi@shiz.me>

Move everything to new launcher.xml configuration files. Adjust the bootstrapping of launcher in the .bat files. Features such as copying the default props/ files to nvram are now handled by launcher. Using the PATH variable, bemanitools binaries can live in their own dedicated bemanitools/ subfolder next to props/ and modules/ now. All original binaries are expected to be kept in a modules/ folder like on stock data.

HRESULT not defined fixed by including windows header

Adjust inject to utilize the new feature. This also requires including the dwarfstack.dll in all distribution packages.

Because we are using mingw, we can't just use window's dbghelp library as the symbols created are in dwarf format. Fortunately, the dwarfstack library already provides all the facilities to easily print very descriptive stacktraces, including function names, file names and line numbers, when dwarf symbols are available. This moves the incomplete exception handling portion from signal to a separate module as well to improve scoping.

Apply a simple heuristic to provide the user with more specific information regarding which vcredist package might not have been found on their system. This is a common problem as different games require different versions and different versions of windows and installations might already come with some versions already pre-installed. Also improve the readme regarding that and provide links to all versions that are required by one game or another today.