Refactor repository layout

[Anaminus]

The content of this comment is unsettled. See edit history for changes.

Packages per repository

To better utilize Go modules, packages that were formerly grouped into a single repository are split into their own separate repositories.

Type equivalence

In general, a package operates on its own version of an element. For example, rbx.Instance is the central representation of a Lua API instance, while rbxfile.Instance represents a generic lower-level serialization of such an instance. The dependent package has operations for converting between these two representations.

Independence

Packages are designed to be as independent as possible. rbxfile.Instance may be converted into an instance that utilizes the generated descriptors, descriptors that are created manually, or not at all.

The rbx package provides the common interface. The ??? package provides an implementation that uses automatically generated descriptors.

Repository names

Generally, a name will have the rbx prefix, followed by something short and disambiguating. This is for the package name: 1) it's easier to understand if the package name closely matches the repo name, 2) The rbx prefix reduces conflicts with variable names (e.g. rbxfile instead of just file).

Repository revamp

/testfiles

Previously: /testfiles (unchanged)

Contains language-independent unit files for checking implementations of the various serialized formats.

The name seems fine; "/test" might contain some sort of testing operations, but "/testfiles" correctly suggests that the repo only contains unit files.

/ref

Previously: /ref (unchanged)

Contains the Roblox API Reference website at https://robloxapi.github.io/ref.

The name shall be unchanged: it becomes apart of the website URL, so it is already established. It is short while being descriptive and disambiguating enough, in terms of github.io subpaths.

/documentation

Previously: /doc

Contains documentation files for the Roblox API.

Name changed to set it out as a non-package repository.

/rbx-api-ref

Previously: /rbxapiref

Command line program that generates the Roblox API Reference as a static website, incorporating documentation from /doc, to be hosted at /ref.

Name changed to set it out as a non-package repository. Requires refactor that moves command to /rbx-api-ref/cmd/rbxapiref.

/dump-archive

Previously: /archive

Contains archived dump files, in the original format, and adapted to the modern JSON format.

Name changed to reduce ambiguity,

Consider including dumps from all versions. GitHub Actions can be used to keep it up to date automatically.

/rbx

Previously: /rbx (unchanged)

An implementation of the Roblox Lua API in Go. The package contains primary interfaces.

The API need not resemble the Lua API precisely. It is more important that it fits Go. However, a requirement is that it is possible to implement a matching Lua API in a Lua implementation (such as gopher-lua).

The API of this package is very unsettled, and the whole thing may end up being scrapped. I feel that it needs to be accompanied by an actual Lua API implementation in order for a proper design to emerge. That is, it needs to actually be used in practice. Right now it's all theoretical, and I'm uncertain about actual use-cases.

Subpackages

I might want to move these outside the package and add the rbx prefix.

At the very least, class and enum should be grouped under one parent package, given their generated nature. Not sure what to name it, though.

• /rbxgen: Generated descriptors.
• /rbxdesc: Easily confused with /rbx/desc.
• /rbxeng: Roblox engine.
• /rbxapi: Roblox API. Probably the best, but conflicts with current /rbxapi.

/rbx/desc

Provides functions to create descriptors.

/rbx/class

Class descriptors generated automatically.

/rbx/enum

Enum descriptors generated automatically.

/rbx/types

The Roblox data types that implement rbx.Value.

All elements related to a type should be put here. e.g. serialization data. The current situation where type information is fragmented across dozens of files is hard to keep track of when Roblox adds a new type. Consolidating everything into one package will make this easier.

Aside

What about the binary format types?

• They represent the serialized data, whereas /rbx/types represents the in-memory data.
• Combining them is not necessarily appropriate, but keeping them split may be too fragmenting.
• Not all types are serialized.
• Putting serialization methods on the type seems to add a lot of clutter.
• The From/ArrayBytes methods are still not great to have on the type itself.
• Maybe a little fragmentation isn't so bad.
• Fragmentation could be alleviated by simplifying the format implementation. e.g. Serializing directly from in-memory types rather than converting to an intermediate representation.

/rbxfile

Previously: /rbxfile (unchanged)

Contains an interface for manipulating serialized instances. This operates on instances at a lower-level than /rbx by providing a rbxfile.Instance type that contains instance data in a generic way.

Has functions for converting between rbx.Instance and rbxfile.Instance.

rbxfile provides a low-level serialization API. It is not meant to consider behaviors of the existing Roblox engine, but rather a generic Roblox engine that has arbitrary descriptors. Higher-level behaviors (e.g. decoding attributes, terrain, tags) are reserved for higher-level packages that may utilize rbxfile.

Currently, rbxfile.Instance is halfway between a serialization type and a full instance. Most of the Lua API-like methods will be removed, favoring methods useful purely for serialization.

Currently, rbxfile includes a dependency on rbxapi for the purpose of optionally
validating elements. This behavior should be moved to a separate package that
uses rbxdump to validate a rbxfile tree.

Some properties encode binary data. These are outside of the scope of serialization implementations. Instead, they have their own separate packages. Such packages should strive to have similar APIs. Should these repos be named to reflect that they involve property encodings?

/rbxfile/testfiles

Previously: /rbxfile/rbxtestfiles

Contains common structures for implementations to check against the /testfiles repo.

Right now this implements checking for all implementations, but only because they currently live in the same repo. When the subpackages are split out, their testing will move with them.

/rbxl

Previously: /rbxfile/bin

Implements serialization of the Roblox binary file format.

/rbxl/testfiles

Previously: /rbxfile/rbxtestfiles

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxlx

Previously: /rbxfile/xml

Implements serialization of the Roblox XML file format.

/rbxlx/testfiles

Previously: /rbxfile/rbxtestfiles

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxmesh

Previously: /rbxmesh (unchanged)

Implements serialization of the Roblox mesh file format.

The API of this package needs to be reconsidered and solidified. It would help if it were actually used somewhere...

/rbxmesh/testfiles

Previously: (none)

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxterrain

Previously: (none)

Implements encoding of terrain binary data.

Currently unimplemented. Doing so will require significant effort. The RobloxFileSpec describes a previous version of the terrain format. While very outdated, it may provide some insight.

/rbxterrain/testfiles

Previously: (none)

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxattr

Previously: (none)

Implements encoding of attribute binary data.

Currently unimplemented, although there is a spec. The format is relatively simple.

/rbxattr/testfiles

Previously: (none)

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxcsg

Previously: (none)

Implements encoding of CSG binary data.

Currently unimplemented. No clue how to approach. Working with CSG at this level doesn't seem very common anyway, so the priority for this is low.

/rbxcsg/testfiles

Previously: (none)

Contains a tool that implements unit testing by checking the implementation against the /testfiles repo.

/rbxclip

Previously: /rbxclip (unchanged)

Provides an interface to the clipboard for Roblox types that is compatible with Studio.

Needs to be updated to reflect changes to rbxfile.

/rbxdump

Previously: /rbxapi

Decodes and encode API dump files.

Done. API is struct-based rather than interfaced-based. Diff and patch are combined into one subpackage.

Renamed to reflect that the package will exclusively involve serializing API dumps, rather than providing some sort of serialization/descriptor hybrid, as it currently does.

/rbxdump/diff

Previously: /rbxapi/diff, /rbxapi/patch, ...

Diffs and patches rbxdump structures.

/rbxdump/json

Previously: /rbxapi/rbxapijson

Serializes between rbxdump and the JSON API dump format.

/rbxdump/legacy

Previously: /rbxapi/rbxapijdump

Serializes between rbxdump and the legacy API dump format.

Weak general support; converted files from /dump-archive should be used instead. Only does enough to allow /dump-archive/dump2json to convert the legacy format.

/rbxdump/histlog

Previously: /rbxdhist

Parses logs involving the deployment of Roblox builds (DeployHistory.txt).