diff --git a/book/modules-software.md b/book/modules-software.md index bde4c64..2935be2 100644 --- a/book/modules-software.md +++ b/book/modules-software.md @@ -30,13 +30,14 @@ module avail - To use a software package, load its module. - Example: ```bash - module load python/3.8 + module load python/3.13.0 ``` + - What happens if you do not specify a version? - **Unloading a Module:** - To remove a software package from your environment. ```bash - module unload python/3.8 + module unload python/3.13.0 ``` - **Listing Loaded Modules:** @@ -49,6 +50,9 @@ module avail **Exercise 2:** Load the `gcc` module and verify it's loaded. :::: +- Note: there are more than one version of gcc available. What is the behaviour if you do not specify a version? + - How should you load a module when running jobs to ensure reproducibility? + :::: solution ```bash module load gcc @@ -67,7 +71,7 @@ module list - Submit a Research Computing Query with details about the software. :::: exercise -**Exercise 3:** Identify a software you need that's not available and draft a request. +**Exercise 3:** Identify a software you need that's not available and draft a request. Can you find the Research Computing Query form via the IT website? :::: :::: solution @@ -76,6 +80,9 @@ Provide the software name, version, and a brief justification for its use in you ```` ````{tab-item} Alternative Software Management + +You also can manage your own software on Aire via a number of different routes. Many users will not need to do this; but it may be neccessary if you want fine-grained control and need to use older versions of software than are available via the module system. + - [**Spack:**](https://spack.io/) - A flexible package manager for HPC systems. - Allows users to install software without admin privileges. @@ -108,8 +115,10 @@ Provide the software name, version, and a brief justification for its use in you - **Environment Management:** - Use modules to manage software environments effectively. - Unload modules when they are no longer needed to avoid conflicts. + - When using software like R or Python, use the Miniforge module to create a conda environment; [read our documentation on how to set this up](https://arcdocs.leeds.ac.uk/aire/usage/dependency_management.html) - **Reproducibility:** + - Specify the version of the module you want to load; the default modules may change as new versions of software are released. - Document the modules and versions used in your research. - Use containers for complex environments to ensure consistency. @@ -124,8 +133,8 @@ Provide the software name, version, and a brief justification for its use in you :::: solution ```bash #!/bin/bash -module load gcc -module load python/3.8 +module load gcc/14.2.0 +module load python/3.13.0 # Add other module load commands as needed ``` :::: diff --git a/book/storage.md b/book/storage.md index e69f1cf..68d90ab 100644 --- a/book/storage.md +++ b/book/storage.md @@ -1,6 +1,6 @@ # Session 3: Storage on Aire -In this session, we will work to understand the main file areas on Aire (home, scratch, etc.), learn the purpose and differences of each storage type, practice navigating between storage areas and checking disk quotas, and learn commands for copying and transferring data to/from Aire. +In this session, we will work to understand the main file areas on Aire (home, scratch, etc.), learn the purpose and differences of each storage type, practice navigating between storage areas and checking disk quotas, and learn commands for copying and transferring data to/from Aire. `````{tab-set} @@ -41,6 +41,20 @@ $ echo $SCRATCH ````{tab-item} 3. Quotas +It's important for you to be able to check how much of your allocated quota you are using in a particular storage area, as exceeding this quota will cause your jobs to fail. + +- `quota` + - This command tells you your disk usage and limits. By default, the unit is kbytes or "blocks". You can add the `-s` flag to make the output "human readable", showing units after the values (`quota -s`). + - If the output is difficult to read, try resizing your terminal windows to accomodate the table. +- `du` + - This command reports the amount of disk space used by different files and directories, by default reporting in 512-byte blocks. + - You can provide the path to a specific directory to focus the results. + - Add the flag `-h` to make the output human readable. + - Add the flag `-s` to summarise the results (and not recursively provide results for subdirectories). + - `du -hs *` will provide you with the space taken up by the data in each directory. This can be a little slow! + + + ::: important **Exercise:** Check your current disk usage and quotas. *(Hint: use the `quota` command.)* ::: @@ -94,7 +108,7 @@ Disk quotas for user yourusername (uid 12345): ::: tip **Solution:** Use the syntax: ```bash -scp username@aire-login.leeds.ac.uk:/users/yourname/results.txt . +scp @target-system:/users/yourname/results.txt . ``` ::: ```` @@ -104,10 +118,15 @@ scp username@aire-login.leeds.ac.uk:/users/yourname/results.txt . - **Navigation:** Use `cd`, `ls`, etc., along with environment variables. - **Check usage:** Regularly run `quota -s` to see your space and file usage. Clean up old files from `$SCRATCH`. - **Data transfer:** Use `scp` or `rsync` via the login nodes to move data. -- **Avoid data loss:** Remember that scratch and flash are *temporary*. Always **backup important data** to home or external storage when done. +- **Avoid data loss:** Remember that scratch and flash are *temporary*. Always **backup important data** to home or external storage when done. + +Read up on [best practises here](https://arcdocs.leeds.ac.uk/aire/usage/file_data_management/best_practices.html). ::: note **Tip:** Keep your home directory organized. Place large data in scratch only while needed. Archive or delete old files in scratch after use. ::: ```` ````` + + +You can read more about the [different storage options on Aire in our documentation](https://arcdocs.leeds.ac.uk/aire/system/storage_filesystem.html#summary-of-storage-types), and find [detailed guidance on organising your files and research data in our data management section](https://arcdocs.leeds.ac.uk/aire/usage/file_data_management/start.html). \ No newline at end of file