__Ganglia__ (https://uhhpc.herts.ac.uk/ganglia/) can be useful to see the state of nodes. 

If a node goes down while a user’s job is running on it, the job will not terminate properly 
and may flood the user’s inbox with notifications. If `Ganglia` or `showstate` report a node 
is down, consider rebooting it with 

`sudo rebootnode.pl nodexxx` 

This will prompt you for the IDRAC password, which is `rianhs4b`. Once a node has been rebooted, 
wait a few minutes, then check that you can ssh into it as a normal user and view your home 
directory and /beegfs. If so, bring it back on line with 

`sudo pbsnodes –c nodexxx` 

If a node is misbehaving and you don’t want to/can’t reboot it, you can temporarily remove it 
from the pool used the job control system with 

`pbsnodes –o nodexxx` 

– also reversed by 

`pbsnodes –c`