In my job, I have to perform calculations on a remote machine to which I connect through ssh. It happens quite often that I have to run a few jobs overnight, if not for days. I could check manually for their termination, but I came up with a better way.
In a previous post I described how useful it can be to run script in background on the machine without having to be logged-in. Combining the power of screen with the utility of mail we can create a script which will automatically send an email whenever the job is finished.
The Script
#!/bin/bash
#
##
## This scripts sends an automated email as soon as a job is finished
##
## *** Use with screen:
## ***
## *** $ screen -S [job-id] (start screen)
## *** $ mailme [job-id] (run mailme in screen)
## *** Ctrl+a d (detach)
## *** Ctrl+d (kill screen, from session)
## *** $ screen -ls (check running screens)
## *** $ screen -r [screen-id] (resumes session)
## *** screen -X -S [screen-id] kill (kill screen, from outside)
## ***
# Need argument: job id which needs to be checked for completion
if [ -z "${1}" ] ; then
echo "Please insert job ID as argument"
echo "Exiting status 1"
exit 1
fi
# Need second argument: email
if [ -z "${2}" ] ; then
echo "Please insert email as second argument"
echo "Exiting status 1"
exit 1
fi
# Put your email here
zzEMAIL="${2}"
zzJOBID="${1}"
zzFREQ=3600 #Frequency of checks, in ms
zzEND=true
# Loop that checks job status
while "${zzEND}"
do
zzSTRING=$(bhist -d | “${zzJOBID}”)
if [ ! -z “${zzSTRING}” ] ; then
zzEND=false
echo -e “Dear Sir,\n\nThis is an automatic email from Alych Linux indicating that the job ID\# ${zzJOBID} has been terminated (succesfully or not).\n\nThis is the output:\n\n $(bhist -d) \n\nBest, \nAlych Linux | mail -s “Job ${zzJOBID} Completed!” “${zzEMAIL}”
else
sleep “${zzFREQ}”
fi
done
exit 0
The comments should provide a satisfactory explanation, but it could be used straight away for jobs submitted through the routine bsub.
You would just need to run the script from a screen and the game is on. If you do not remember how to do this, I have put some reminders in the header section of the script, or you can check my guide to screen.
You can run multiple of these scripts in multiple screen sessions if more jobs need to be tracked.
Afterthoughts
This script has been designed to check job progress and send an email whenever the job is complete. Through an easy modification of the condition to be checked in the variable zzSTRING in the loop, anything can trigger the delivery of an automated email, potentially making the scope of this script very broad.
Have a play with it and let me know how you use/modify it!
