Radiam Agent

************ WARNING ************

THIS CODE IS NO LONGER MAINTAINED

It contains known security isssues

Use at your own risk

************ WARNING ************

Getting Started

Python 3 is required to run the Radiam agent from the command line. Ensure that the output of python3 --version shows 3.6 or higher.

If Python 3 is not installed on your Mac, the easiest way to get it is:

• Install homebrew; see https://brew.sh/
• brew install python3

On Windows and Mac, install Python dependencies using pip. You may want to run inside a virutal environment (see below) before running this command.

pip3 install -r requirements.txt

If you aren't an administrator and receive an error such as:

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/elasticsearch5'
Consider using the `--user` option or check the permissions.

you will either need to run:

pip install -r requirements.txt --user

or alternatively, run inside a virtual environment:

virtualenv venv
source venv/bin/activate
python --version

at this point you should see (venv) to the right of your command prompt, showing you that you are running inside a virtual environment. You can now check the version of Python inside this environment:

python --version

And if it's indeed version 3, then install the requirements inside the virtual environment:

pip install -r requirements.txt

To exit the virtual environment:

deactivate

Install on Linux is slightly different as there is no binary distribution of libmagic already created for Python -- you'll want to do:

sed -i '' "s/python-magic-bin/python-magic/g" requirements.txt
pip install -r requirements.txt

The Electron GUI in /tray should work for development after doing npm install and npm start.

If on install you receive the error:

error: can't copy 'docx/templates/default-docx-template': doesn't exist or not a regular file

Then run:

pip install -U setuptools

CLI Usage

$ python radiam.py -d /rootpath/you/want/to/crawl -h hostname -n projectname

Options:
  -d --rootdir=<DIR>  Directory to start crawling from
  -m --mtime=<mt>  Minimum days ago for modified time (default: 0) [default: 0]
  -s --minsize=<ms>  Minimum file size in Bytes (default: 0 Bytes) [default: 0]
  -h --hostname Specify hostname if not in config (e.g. https://localhost:8100, https://dev2.radiam.ca:8100)
  -u --username Specify username if connecting without a token
  -p --password Specify password if connecting without a token
  -n --projectname=<pro> Project name

Configuration file is automatically generated after your first run, and stored, along with authorization tokens, in a user-specific directory depending on platform:

~/.local/share/radiam-agent/
~/Library/Application Support/radiam-agent
~/AppData/Local/Compute Canada/radiam-agent

GUI Usage

The crawler GUI will launch as a tray app on Windows, Mac, and Linux. Before running a crawl, you need to obtain an auth token by providing your Radiam username and password with the "Get Login Token" menu option, and select the project directory to be crawled with the "Set Projection Location" menu option. You can optionally configure any advanced settings from the "Settings" menu. After configuring the app, select "Crawl" from the tray menu, and it will perform a full crawl of your project directory and continue monitoring for changes in the background.

Building the crawler and the GUI

The crawler needs to be built on its target platform (Windows/Mac/Linux) using pyinstaller, but you'll need to do some additional work to package libmagic in there, so we've included a pyinstaller.sh which runs the necessary steps on Windows and Mac.

After building the crawler, the GUI can be built from the tray subdirectory with:

electron-packager . --icon=resources/icon.ico (Windows)

electron-packager . --icon=resources/icon.icns (Mac)

Finally, to package for install:

electron-installer-windows --src radiamagent-win32-x64/ --dest install/ --config config.json --certificateFile radiam.pfx --certificatePassword [password] (Windows)

hdiutil create tmp.dmg -ov -volname "RadiamAgent" -fs HFS+ -srcfolder radiamagent-darwin-x64/ && hdiutil convert tmp.dmg -format UDZO -o RadiamAgent.dmg && rm tmp.dmg (Mac)

We aren't actively supporting the GUI on Linux, but anecdotally it works fine, though you need to prepend env XDG_CURRENT_DESKTOP=Unity to your command to spawn the tray app properly. This means env XDG_CURRENT_DESKTOP=Unity npm start when running it for development or Exec=env XDG_CURRENT_DESKTOP=Unity radiamagent in a .desktop shortcut.

Advanced features

Radiam can monitor multiple directories and connect to multiple upstream projects by adding additional project headers to the config file.

By default, the config file will contain:

project_list = project1

[project1]
rootdir =
name =

To configure multiple projects, follow this template:

project_list = project1, project2

[project1]
rootdir =
name =

[project2]
rootdir =
name =

Radiam can also include advanced metadata extracted from files in its search index. This functionality is disabled by default to avoid uploading any potentially sensitive data, but it can be enabled by changing this line in your config file:

#rich_metadata = disabled

to rich_metadata = enabled.

You can also manually provide rich metadata for folders in a file called [foldername].yml in each folder using YML syntax:

raw: no
analyzed: no
sensitive: yes

Which will be indexed as advanced folder metadata in the Radiam interface, similar to advanced file metadata.

Note: The fields "id" and "endpoint" will be automatically filled. Please leave them blank.

Tests

To run tests on the Python code: python -m unittest test