handling directories transparantly

[Dieterbe]

Maybe I'm missing something. But I just read the readme and I have an idea:
If the user creates a file like TO_DIR/.config/somefile, why not do something like:
mkdir FROM_DIR/.config
ln -s TO_DIR/.config/somefile FROM_DIR/.config/somefile

In other words: only symlink the "most specific path component" (basename), and replicate the dirname
I don't see the need for a separate FROM_DIR/TO_DIR pair for a directory like ~/.config
this would also work with other paths, no matter how deep.

[seanh]

I had originally intended to make it recurse into directories just like you describe. When I was implementing it there was some specific reason why that was a pain in the ass to do, so I didn't bother. I fell back on only managing top-level symlinks in FROM_DIR, and they can be links to files or to directories. I can't remember exactly what the problem was, but I think it was just a difficulty not an impossibility. Probably a careful reconsideration of the algorithm could find an easy way of handling it.

One issue that comes to mind is when using dotfilemanager to tidy up (i.e. to delete the symlinks). It doesn't keep any record of which symlinks were created by dotfilemanager and which not, it just finds all (top level) broken symlinks in FROM_DIR and deletes them. It's assuming that any top level broken symlinks in FROM_DIR were created by an earlier dotfilemanager command, or that you don't want them around anyway. So if dotfilemanager were creating directories as well, you'd have to think of some other way to handle cleaning up. Perhaps delete directories that contain nothing but other directories and broken symlinks.

But I've been thinking that maybe dotfilemanager should keep a record of what links (and directories) it has created in a file and make use of it for things like the tidy and report commands. Might simplify somethings, and make some new features possible. It would have to consider what to do if this file got destroyed or damaged or no longer matched up with reality because the user modified the symlinks manually.

I'd be interested to hear your thoughts on it

[seanh]

Another thing comes to mind: I have a _mutt/ directory in my git repo. Dotfilemanager creates a ~/.mutt symlink to it. If I cd into ~/.mutt and create a new mutt config file (my mutt config is spread across several files in the .mutt directory) then this new config file will show up when I run git status in my git repo, because the entire ~/.mutt directory is a symlink to the _mutt dir in my git repo. In fact, I can even do cd ~/.mutt; git status and it will work.

If dotfilemanager had instead recursed into the mutt directory and created symlinks to each of the individual files, then creating a new file in ~/.mutt would not show up in git status.

[Dieterbe]

Yeah i hear you re:cleanup.. it would also be slow to do a search for possible candidates for removal.
What about:

• make dotfilemanager use git to store a versioned snapshot of the symlinks in $FROM_DIR everytime the user invokes a link/tidy command? or similarly, maintain a list of files in $TO_DIR in git everytime the user invokes dotfilemanager link/tidy/report.
  then, to find all directories/files which must be deleted, you would compare the state from the last tidy operation with the last one.
• or even further: tell users they should use git in $TO_DIR if they want the directory cleanups to work

about the mutt thing: yeah that's true. but could be solved: upon dotfilemanager report: for every dir in $TO_DIR, check the corresponding dir in $FROM_DIR and warn if there are "real" (non-symlinked) files in there.

[tgray]

So I have this behavior loosely working as of now:

mkdir FROM_DIR/.config
ln -s TO_DIR/.config/somefile FROM_DIR/.config/somefile

As of now, the tidy mechanism is probably partially broken. Directories that are made are definitely not reported or removed during tidy. I'm also not entirely clear if other broken links are reported properly right now either -- I need to check on that.

I think it would be relatively easy to keep a file with the created symlinks and directories. It could either be a flat file or a small sqlite database. This would allow you to differentiate between broken links and empty directories that are the result of the dotfilemanager script from ones that are leftovers from other user actions.

Lastly, the mutt thing - I think you have to pick your poison. Either you symlink directories, which let's you create a file at .mutt/newfile and have it appear automatically in .dotfiles/mutt/newfile, or you use mkdir instead and symlink the contents. If you do the latter, you just have to know to at some point copy your new file back to the appropriate .dotfile subdirectory. I guess you could also keep track of the contents of each dot folder and have dotfilemanager copy any new folder contents back to the original .dotfile folder...

[seanh]

So I have this behavior loosely working as of now:

mkdir FROM_DIR/.config ln -s TO_DIR/.config/somefile
FROM_DIR/.config/somefile

As of now, the tidy mechanism is probably partially broken.
Directories that are made are definitely not reported or removed
during tidy. I'm also not entirely clear if other broken links are
reported properly right now either -- I need to check on that.

I think it would be relatively easy to keep a file with the created
symlinks and directories. It could either be a flat file or a small
sqlite database. This would allow you to differentiate between broken
links and empty directories that are the result of the dotfilemanager
script from ones that are leftovers from other user actions.

Yeah, I agree with you about keeping a file. The reason I didn't do this
in the first place was that I was trying to keep dotfilemanager as
simple as possible and thought having no saved configuration or state
would be good from that point of view. But if recording state in a file
will enable better functionality, I think it's worth it.

Dieter's suggestion of versioning this file in git is interesting. I
wouldn't want to have dotfilemanager be integrated with git or any
version control system in particular, I prefer to keep it a 'one thing
well' tool that is unaware of version control systems and allows the
user to version, sync, backup, their TO_DIR or not as they please.
However, if this file kept by dotfilemanager could be placed in the
TO_DIR (or, say, in a ~/.dotfilemanager directory that could itself be
symlinked to TO_DIR by dotfilemanager) then the file could be synced and
versioned using whatever tool the user wants.

Lastly, the mutt thing - I think you have to pick your poison. Either
you symlink directories, which let's you create a file at
.mutt/newfile and have it appear automatically in
.dotfiles/mutt/newfile, or you use mkdir instead and symlink the
contents. If you do the latter, you just have to know to at some
point copy your new file back to the appropriate .dotfile
subdirectory. I guess you could also keep track of the contents of
each dot folder and have dotfilemanager copy any new folder contents
back to the original .dotfile folder...

Yeah, I agree there are pros and cons either way on this issue, but I'm
not sure there isn't some third way that would feel 'right'. Homesick
(https://github.com/technicalpickles/homesick) keeps dotfiles in a
~/.homesick directory. Each top-level subdir of ~/.homesick is a
'castle', a directory that contains dotfiles. So if you have
~/.homesick/config you can tell it to make symlinks to that dir with the
command homesick symlink config. homesick list lists all your
castles. I guess there could also be commands to report which castle are
currently symlinked, to unlink a castle, etc.

I think it's basically the same thing that dotfilemanager achieves by
allowing you to specify FROM_DIR and TO_DIR as arguments, but the idea
of keeping all your 'castles' nice and tidy in one place and having
convenient commands to work with them appeals.

I wonder if this could inspire a solution for dotfilemanager?

[seanh]

What do you guys think of this? Below is my README for a proposed new version of dotfilemanager. Since dotfilemanager is kind of a long name, I propose renaming it to dr (short for dotfiles repository manager, inspired by mr).

I'm proposing that dr should still work by creating symlinks to every top-level file and directory in a repo instead of recursing into directories (I don't like the idea of recursing into directories as it prevents commands like git status from reporting new files in my ~/.mutt directory, for example), but it should have commands that make working with multiple repos more convenient. The idea is to make it easy to have multiple repos linked to the same or different source directories at the same time.

Whereas dotfilemanager was configuration-free and stateless, dr will need a ~/.drconfig file and a ~/.drstate file.

The new design is inspired by homesick, but I think that dr should not include commands like homesick has for cloning repos, pulling them, etc. using git (or any VCS). I already use mr which does this stuff really well, so better to keep dr simple and unaware of VCSs.

drconfig file

Lists repos, for each giving:

• name (to identify this repo in dr commands)
• from dir (where symlinks will be created)
• to dir (where the files and dirs to be symlinked to will be stored)

and also identifying one repo as the default.

If dr is run and finds no drconfig file it will offer to create one, and will make a default config file listing one default repo. It should offer to create the to dir for the repo as well.

Commands

dr register --from dir --to dir name
Add a new repo to the drconfig file with the given name, from dir and to dir.

dr track [--repo name] file|dir [file|dir ...]
Move files (or directories) into a repo and symlink them. If no --repo option is given uses the default repo.

dr ls|list
List the names of all repos from the drconfig file. With -l, also list the from dirs and to dirs and a summary of the status of each (is linked or not, has broken links in from dir or not, has unlinked files in to dir or not, has any other problems such as files in the way where dr wants to create symlinks).

dr link [repo ...]
Create symlinks from a repo's from dir to all of the top-level files in a repo's to dir. Run the command again to create new symlinks for any new files in the repo. Should create from dir (and parent directories) as necessary.

dr tidy [repo ...]
Remove any broken symlinks in the from dir(s) of the listed repo(s).

dr sync [repo ...]
Run dr link [repo ...] and then dr tidy [repo ...].

dr unlink [repo ...]
Delete all symlinks for repos (broken or not).

dr status [repo ...]
Print out a status report for a repo, listing any broken symlinks in the from dir, any unlinked files in the to dir, and any other problems (e.g. other files existing where dr wants to create a symlink). If the repo is not linked at all, say so. With the -l option also list all symlinked files.

In all commands that have an optional [repo] or [repo ...] argument, if none is given they will operate on the default repo. This means that if you just want to work on the one repo you can just use dr track, dr link, dr tidy, dr sync and dr status on their own without worrying about the multiple repos support.

Symlinks and broken symlinks should only be removed by link, tidy, sync or unlink or reported by status if they belong to the repo being worked on. Any broken links belonging to other repos linked to the same from dir or not belonging to a dr repo at all should be ignored. To make this possible, dr needs to keep a state file listing all of the symlinks it has created for each repo.

So for the case described in dotfilemanager's README where you have one repo linked to ~/ and another linked to ~/.config, with dr you would:

First run dr and have it create the initial ~/.drconfig file containing one repo called default with ~/ as the from dir and ~/.dr/default/ as the to dir. ~/.dr and ~/.dr/default/ will be created. Now you can run for example dr track .mutt to move ~/.mutt/ to ~/.dr/default/_mutt and symlink it.

Next, to create the repo for the .config directory run dr register --from ~/.config --to ~/.dr/config config. This adds the new repo to the drconfig file and creates the new to dir, you can now use dr tack --repo config file|dir [file|dir ...] to add stuff to the config repo.

Now if you want to create a 'private' repo also linked to ~/ to contain your sensitive files (you might only sync this repo on trusted machines) yu would do dr register --from ~/ --to ~/.dr/private private, and then use track to add files to it.

[tgray]

Lots of points to address:

new name - dr is a bit short and makes me think of old school unix commands. I do agree 'dotfilemanager.py' is a bit of a mouthful. Maybe something like 'dotm' or 'dotman'?

homesick inspired design - I'll need to read up on homesick a bit. It sounds like it is a bit of an overkill solution. I kind of like the relative simplicity of dotfilemanager as it is; just copy over/symlink any file in my .dotfiles directory. A little more flexibility for alternative configurations would be useful, but simple and elegant is key in my mind. And there's obviously no sense in duplicating it in full, not that you are suggesting it.

The spec laid out in the new readme is interesting. Going back to the unix philosophy though, what is gained by having a command like dr track [--repo name] file|dir [file|dir ...] when I can just type mv original .dotfiles/target? But maybe I'm misunderstanding something. Are the repos mentioned meant for individual configurations, or are they for whole groups of files (ex. linux setup vs. OS X setup), or some combination of the two.

It has interesting possibilities, but I think it would be wise and elegant to have any repo management/tracking/configuration happen solely through file names and locations. Much like hostnames are tracked now. One could imagine having a 'default' directory in the .dotfiles directory which acts as base set of files to symlink. Other repos would be designated by putting them in other directories in .dotfiles, and their name would be drawn from the directory name. It would avoid having to register repos and manage them via this tool - all management would occur using normal file system tools.

[seanh]

Thanks for taking an interest in dotfilemanager tgray.

The track command is just a convenience, it's the same as manually moving a file into a repo and then running the link or sync command.

The repos are meant for whole groups of files, e.g. a repo for dotfiles that belong in your home directory, another repo for private/sensitive dotfiles, a repo for dotfiles that belong in ~/, etc. The semantics of multiple repos are up to the user.

There are two reasons for using a config file to list and name repos, rather than just using a directory. First, it's more flexible because it doesn't force you to put all of your repos in the same directory. Second, the config file records the from dir as well as the to dir of each repo. It's also more extensible, we could add more settings to the config file in future if we wanted.

[tgray]

Makes more sense to me now.

I'm game. How do you want to start working on this?

[Dieterbe]

hey guys, i'm very busy lately. unfortunately no time for me to participate in coding or even design discussions. have fun.

[seanh]

Sorry for the late reply tgray, I've been busy messing with my website.

1. Make a new branch to develop the new version in.
2. Update the README file with a full spec for the new version, listing exactly what the commands will be and what each will do, any command-line options, what the configuration file will be and its syntax and options, and anything else. My comment above should be a start.
3. Plan how to implement it. Some of the functions already in dotfilemanager.py might be able to be reused, maybe with some modification. New functions will probably have to be written.
4. Code!

Feel free to make a start on this in your clone, I'm up for receiving pull requests from you.