# Using nabu at ESRF

This page contains specific instructions on how to use nabu at ESRF.
The activation commands will also provide the tomwer graphical interface.

**One command to remember**: `source /scisoft/tomotools/activate`


## 1. On the compute cluster 

For large scale reconstructions (multiple volumes), nabu is run on the ESRF compute cluster. Reconstruction can be done either through an interactive session, or through submission of a shell script.

The first thing to do is to connect to a front-end machine, which in turn is able to request computational resources:
```bash
ssh -XC cluster-access
```

### 1.1 - Requesting resources in SLURM

You can open an interactive session on a GPU compute node. Currently, the preferred partition is `p9gpu` (power9 machines with Tesla V100 GPUs).
```bash
salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash -l
```

```{warning}
The important bit is `--gres=gpu:1` - you have to explicitly request a GPU, even if the node is in the GPU partition. 
```

Don't forget the final `-l` in `srun --pty bash -l`.



This command will only request one CPU core. If you wish to do some CPU processing (eg. involving numpy), then the `-c` parameter has to be provided. To ensure that all the cores are on the same CPU socket, a working command is 

```bash
salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G --cores-per-socket 16 -c 64 srun --pty bash
```

```{caution}
The maximum duration of jobs in the `p9gpu` is one hour. For longer jobs, the `p9gpu-long` partition has to be used.
```


### 1.2 - Pick and activate a nabu version

On the compute clusters (and *most* ESRF machines), the command to use is

```bash
source /scisoft/tomotools/activate VERSION
```
where `VERSION` is replaced with a version of nabu. For example:
```bash
source /scisoft/tomotools/activate nabu_2022.2_tomwer_1.0
```

Please see section 2 for instructions specific to machines that cannot access the `/scisoft` disk.


### 1.3 - Run the reconstruction

Once you have a compute node and activated a version of nabu, you can run the [usual command](nabu_cli) `nabu config_file.conf`



## 2. On a local machine (LBS/dedicated machines)

Some machines cannot mount the `/scisoft` disk.

The following machines have special paths for activating nabu (here the `dev` version, but it can be another version):

  - `iccid191`: `source  /nobackup/iccid1911/tomotools/activate dev`
  - `iccbm051`:  `source /nobackup/iccbm0511/tomotools/activate dev`

Please contact us if you wish to install nabu on a specific ESRF machine.


## 3. Troubleshooting

### Nabu fails to start with a `cublasNotInitialized` error

It very likely means that you did not request for a GPU in the SLURM/OAR allocation command. A list of available GPUs can be obtained with `nvidia-smi`. If a `No devices were found` message is shown, then indeed no GPU is available.

## 4. Frequently Asked Questions (FAQ)

### How can I report a problem ?

For any problem with nabu (crash, bad reconstruction results, installation), there are several ways to report it:
  - Ideally, open an [issue on the bug tracker](https://gitlab.esrf.fr/tomotools/nabu/-/issues)
  - Write a mail to [pierre.paleo@esrf.fr](mailto:pierre.paleo@esrf.fr)
  - Use the ESRF instant messaging (rocket.chat)

In any case, you have to provide the **configuration file** giving the error so that the error can be reproduced, and ensure that the dataset to process is accessible by "other" users.

### How can I do a quick check that nabu works correctly ?

To check that nabu is installed, simply type `nabu -V`. It returns the current version of nabu. If it fails, then nabu is not installed/activated.

To check that nabu is actually able to reconstruct, you can [run some tests](tests), the recommended one is `nabu-test reconstruct`. Ensure that all the tests in the line `test_fbp` are passing.