Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions book/modules-software.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,14 @@ module avail
- To use a software package, load its module.
- Example:
```bash
module load python/3.8
module load python/3.13.0
```
- What happens if you do not specify a version?

- **Unloading a Module:**
- To remove a software package from your environment.
```bash
module unload python/3.8
module unload python/3.13.0
```

- **Listing Loaded Modules:**
Expand All @@ -49,6 +50,9 @@ module avail
**Exercise 2:** Load the `gcc` module and verify it's loaded.
::::

- Note: there are more than one version of gcc available. What is the behaviour if you do not specify a version?
- How should you load a module when running jobs to ensure reproducibility?

:::: solution
```bash
module load gcc
Expand All @@ -67,7 +71,7 @@ module list
- Submit a Research Computing Query with details about the software.

:::: exercise
**Exercise 3:** Identify a software you need that's not available and draft a request.
**Exercise 3:** Identify a software you need that's not available and draft a request. Can you find the Research Computing Query form via the IT website?
::::

:::: solution
Expand All @@ -76,6 +80,9 @@ Provide the software name, version, and a brief justification for its use in you
````

````{tab-item} Alternative Software Management

You also can manage your own software on Aire via a number of different routes. Many users will not need to do this; but it may be neccessary if you want fine-grained control and need to use older versions of software than are available via the module system.

- [**Spack:**](https://spack.io/)
- A flexible package manager for HPC systems.
- Allows users to install software without admin privileges.
Expand Down Expand Up @@ -108,8 +115,10 @@ Provide the software name, version, and a brief justification for its use in you
- **Environment Management:**
- Use modules to manage software environments effectively.
- Unload modules when they are no longer needed to avoid conflicts.
- When using software like R or Python, use the Miniforge module to create a conda environment; [read our documentation on how to set this up](https://arcdocs.leeds.ac.uk/aire/usage/dependency_management.html)

- **Reproducibility:**
- Specify the version of the module you want to load; the default modules may change as new versions of software are released.
- Document the modules and versions used in your research.
- Use containers for complex environments to ensure consistency.

Expand All @@ -124,8 +133,8 @@ Provide the software name, version, and a brief justification for its use in you
:::: solution
```bash
#!/bin/bash
module load gcc
module load python/3.8
module load gcc/14.2.0
module load python/3.13.0
# Add other module load commands as needed
```
::::
Expand Down
25 changes: 22 additions & 3 deletions book/storage.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Session 3: Storage on Aire

In this session, we will work to understand the main file areas on Aire (home, scratch, etc.), learn the purpose and differences of each storage type, practice navigating between storage areas and checking disk quotas, and learn commands for copying and transferring data to/from Aire.
In this session, we will work to understand the main file areas on Aire (home, scratch, etc.), learn the purpose and differences of each storage type, practice navigating between storage areas and checking disk quotas, and learn commands for copying and transferring data to/from Aire.


`````{tab-set}
Expand Down Expand Up @@ -41,6 +41,20 @@ $ echo $SCRATCH

````{tab-item} 3. Quotas

It's important for you to be able to check how much of your allocated quota you are using in a particular storage area, as exceeding this quota will cause your jobs to fail.

- `quota`
- This command tells you your disk usage and limits. By default, the unit is kbytes or "blocks". You can add the `-s` flag to make the output "human readable", showing units after the values (`quota -s`).
- If the output is difficult to read, try resizing your terminal windows to accomodate the table.
- `du`
- This command reports the amount of disk space used by different files and directories, by default reporting in 512-byte blocks.
- You can provide the path to a specific directory to focus the results.
- Add the flag `-h` to make the output human readable.
- Add the flag `-s` to summarise the results (and not recursively provide results for subdirectories).
- `du -hs *` will provide you with the space taken up by the data in each directory. This can be a little slow!



::: important
**Exercise:** Check your current disk usage and quotas. *(Hint: use the `quota` command.)*
:::
Expand Down Expand Up @@ -94,7 +108,7 @@ Disk quotas for user yourusername (uid 12345):
::: tip
**Solution:** Use the syntax:
```bash
scp username@aire-login.leeds.ac.uk:/users/yourname/results.txt .
scp <username>@target-system:/users/yourname/results.txt .
```
:::
````
Expand All @@ -104,10 +118,15 @@ scp username@aire-login.leeds.ac.uk:/users/yourname/results.txt .
- **Navigation:** Use `cd`, `ls`, etc., along with environment variables.
- **Check usage:** Regularly run `quota -s` to see your space and file usage. Clean up old files from `$SCRATCH`.
- **Data transfer:** Use `scp` or `rsync` via the login nodes to move data.
- **Avoid data loss:** Remember that scratch and flash are *temporary*. Always **backup important data** to home or external storage when done.
- **Avoid data loss:** Remember that scratch and flash are *temporary*. Always **backup important data** to home or external storage when done.

Read up on [best practises here](https://arcdocs.leeds.ac.uk/aire/usage/file_data_management/best_practices.html).

::: note
**Tip:** Keep your home directory organized. Place large data in scratch only while needed. Archive or delete old files in scratch after use.
:::
````
`````


You can read more about the [different storage options on Aire in our documentation](https://arcdocs.leeds.ac.uk/aire/system/storage_filesystem.html#summary-of-storage-types), and find [detailed guidance on organising your files and research data in our data management section](https://arcdocs.leeds.ac.uk/aire/usage/file_data_management/start.html).
Loading