Scyld Commands

The Scyld system on includes special commands that enable administration of Alamo's multiple nodes. The normal Linux administrative commands cover most of what is necessary on the login node. For administration of the compute nodes, a set of special commands is required.

On the main (login, head) node, the glusterfs (Gluster File System) is examined and modified with gluster commands, such as

gluster volume status
gluster volume info
gluster peer status
gluster volume stop gv0
gluster volume start gv0

Overall state of Torque nodes can be examined with

qstat -B

Compute Node Monitoring and Basic Start / Stop

The bpsh command executes shell commands on a list or range of compute nodes. The arguments include the node list/range, an optional "-p" print-node-prefix flag, and the command to execute. Examples are as follows.

bpsh 7 df -h
bpsh 7 mount
bpsh 7 lspci |egrep -i infiniband
bpsh 0 xfs_repair /dev/md0
bpsh 0-15 -p lspci |egrep -i infiniband
bpsh 0-15 -p df -h /home
bpsh 0-15 -p df -h /mnt/glusterfs/gv0

Nodes should normally be restarted or their states changed with "soft" state change bpctl commands. To start a set of nodes that is powered off, the ipmitool command should be used.

bpctl -S all -s reboot
bpctl -S all -s up
ipmitool -H n0-ipmi -U ADMIN -P ADMIN chassis power on
ipmitool -H n1-ipmi -U ADMIN -P ADMIN chassis power status
for N in `seq 0 15`; do ipmitool -H n${N}-ipmi -U ADMIN -P ADMIN chassis power on; done

Node Configuration and Services

Configuration files for nodes are held in the /etc/beowulf directory. Service start/stop scripts are held in /etc/beowulf/init.d.

NODE=15 bash /etc/beowulf/init.d/93glusterd
NODE=0 bash /etc/beowulf/init.d/97glusterfs
for N in `seq 0 15`;do NODE=${N} bash /etc/beowulf/init.d/93glusterd;done
for N in `seq 0 15`;do NODE=${N} bash /etc/beowulf/init.d/97glusterfs;done
beochkconfig --list
beochkconfig 97glusterfs off
ls /etc/beowulf/init.d

Miscellaneous Scyld Commands

less /proc/mdstat
mdstat --help
man mdadm
ls /var/log/glusterfs
ls -lF /glusterd/var/glusterfs/log/glusterfs
Last modified 5 years ago Last modified on Sep 30, 2015 7:14:44 PM