Borderlands ModCabinet Sorter

This is an attempt to create an automated ModCabinet wiki for the BL2/TPS mods housed in the BLCMods github. The current ModCabinet wiki relies on modders manually updating the entries, and as a result the ModCabinet is extremely under-populated (to say nothing of the hassle required for the folks who do keep their mods updated on there).

The util will attempt to loop through a local github checkout, looking for files named cabinet.info, which mostly just contain a line per mod in the same directory which specifies which categories the mod should belong in. The app should be able to read in mod summaries both from the mod files themselves, and from README files stored alongside the mods.

Obviously it's a bit insane to try and write something to parse through a bunch of effectively freeform text all over the place, but hopefully it can be wrangled into something useful, which can then be run every 15 minutes or something.

At the moment, the app works quite well, with the caveats of the items noted in the TODO below. I will eventually get some better documentation in here about how to get this running, in case anyone else ever needs/wants to take over generating these pages.

Requirements

This is a Python 3 app. It may work in Python 2 but no attempt has been made to find out. Required modules (installable via pip, of course):

• gitpython
• Jinja2
• python-Levenshtein
• appdirs
• coverage (only to run coverage on the unit tests, for development purposes. Not needed just to run.)

How To Use

In case Apocalyptech ever gets hit by a bus or something, someone else might end up wanting to take over running this thing. Here's how:

1. Set up a Python virtual env for the app, wherever you like, using python3 -m venv cabinetsorter (or whatever you want to call it). That'll create a new cabinetsorter dir in your current dir, containing the venv.
2. Activate the venv using . cabinetsorter/bin/activate (the dot at the beginning is important). Your command prompt should now have a (cabinetsorter) prefix on it, to indicate that you're operating "in" that venv.
3. Run pip install --upgrade pip to get on the latest version of the python package manager (this step's optional, but it'll nag you until it's up-to-date anyway, so you might as well).
4. Go into the dir where you've checked out this project, and install the dependencies with: pip install -r requirements.txt, or just install them one by one with pip install <module>, from the list above.
   1. requirements.txt doesn't list the coverage module since that's only used in conjunction with unit tests, so if you care about running those, you might want to install coverage too. If you're just running the sorter, though, don't sweat it.
5. Run sorter.py once; it'll complain that you don't have a config file set up, but more importantly it'll tell you the path that it's looking for, for the file.
6. Copy cabinetsorter.ini.example so that the app will find it. On a Linux machine that'll likely be ~/.config/cabinetsorter/cabinetsorter.ini
7. The mods section of the INI file describes the mod repo itself.
   1. clone_url can just be the anonymous HTTPS URL used to clone the repo.
   2. base_url is the base URL used to link directly to files inside the repo (usually the same as clone_url but with .git changed to /tree/master).
   3. download_url is the URL used to embed screenshots in our markdown files, and will have a raw.githubusercontent.com hostname.
   4. repo_dir is the checkout location of the repo on-disk. I keep them inside the repos directory right inside the cabinetsorter checkout.
8. The wiki section of the INI file describes the wiki repo itself (github project wikis are really just separate github repos themselves.)
   1. clone_url is the URL used to check out the wiki. github itself will not actually show the correct URL here -- it'll only tell you the "anonymous" HTTPS url. You need to use the SSH-based URL which lets you push new content, etc. Its form will be git@github.com:BLCM/ModCabinet.wiki.git. On my own system, I'm doing some shenanigans where I upload using an account different from my "main" github account, so I actually specify the hostname bit there as github.com-apocalyptechcabinet, which I've configured SSH to use an alternate IdentityFile, and set the HostName to github.com and User to git. If you don't care about using a separate github user for this, don't bother with that -- just use the URL mentioned earlier.
   2. cabinet_dir is the checkout location of the wiki repo on-disk. Like the mods repo, I keep it inside the repos directory right inside the cabinetsorter checkout.
9. Manually check out both the mods repo and the wiki repo -- the app currently doesn't support doing that automatically. If you're keeping them checked out in the repos subdir, simply go in there, and do a git clone using the clone_urls for both the "mods" and "wiki" section of the INI file you just configured.
   1. Note that if you're starting this up on a brand new github wiki, the wiki needs to have at least the main page, so create the first page from the web UI and then do your checkouts.
   2. Doublecheck that the resulting repos/BLCMods dir contains a checkout of the mod archive, and the repos/ModCabinet.wiki contains a checkout of the current wiki.
10. By default, the sorter will only Do Things if it notices that there's been an update to the mods repo. Since you just checked it out, there probably won't be changes yet, to you'll want to use the -f/--force flag to force it to Do Stuff regardless. Also, for your very first time running the app, you'll want to use the -i/--initial flag, which sets the file modification times for all the files in the BLCMods checkout. (This is useful because it uses the file modification times to populate the "last updated" report on the mod pages. Otherwise the reported "last updated" times will end up being the timestamp when you cloned the repo.)
    1. If you want to just test things out, you can also use the -g/--no-git flag to prevent the app from trying to interact with github in any way. The app also keeps caches of it's "remembered" state of the BLCMods repo in the cache dir. You can clean that dir out whenever you like, or just use the -x/--ignore-cache argument to ignore it. There's a few other options which can be used, use -h/--help to see them all.
11. Once you're confident that it's working properly, you'll want to hook it up an automated process which runs it occasionally. I wouldn't recommend doing it more often than every 10 minutes. My cron line looks like this: */10 * * * * cd /home/pez/git/b2patching/cabinetsorter && /home/pez/virtualenv/mcpcabinetsorter/bin/python /home/pez/git/b2patching/cabinetsorter/sorter.py -q
    1. Using the -q/--quiet option will prevent the app from outputting any text unless there's an actual error with processing. This way your system running the cron won't send you an email unless there's a problem which might need your attention.

TODO

• Support doing initial git clones?
• README/Mod Description parsing could probably use some tweaking, but will have to do that carefully, of course.
• Update unit tests, they're pretty out of date now
  • Make sure to have tests to explicitly check for MUT categories on FT and BLCMM files
  • Make sure to test specifying a filename which doesn't exist
  • Test stripping spaces from category names (silverxpenguin's Revalation, for instance)
  • Test escaped quotes in BLCMM top-level category names
  • Test unicode vs. latin1 mod files
  • Test escaped leading hashes in cabinet.info files
  • ModFile sorting (case-insensitive)
  • Angle brackets in load_unknown
  • wiki_filename processing
  • We've moved to always using full HTML HREFs instead of wiki links, when calling wiki_link
  • In ModFile.load_unknown(), check for FT-style categories
  • Strip square brackets from ASCII art
  • Test __eq__ on ModURL objects for the various tests in there
  • The new @no-mod-comment style directives in cabinet.info files
• Figure out something to do with, for instance, mopioid's Phaseclock. Also Robeth's TimeScale? Also Aaron0000's Pandemonium.
• The first time the app is run, without caches to read from, mods which share the name of an author will error out instead of being generated. That will get fixed on subsequent runs which do read from the cache. I don't care enough to fix that edge case at the moment, but it may bear looking into later. (vWolvenn's "Tsunami" is the only current case of this actually happening.)
• If we get a CRITICAL error midway through processing, the next run will not try again, because the git pull has already happened and it looks like there's nothing to do. I think we'll have to cache the git commit at which we last successfully run, and compare against that.
• See if we can get rid of our full_filename var in ModFile. I bet we can...

License

Borderlands ModCabinet Sorter is licensed under the GPLv3 or later. See COPYING.txt for the full text of the license.