Cron jobs execution monitoring in slack
This post is in the same vein than the previous one but applies to cron jobs instead of celery tasks.
I used to redirect stderr to a log file for my cron jobs but it took me weeks to figure the jobs were crashing in the first place.
So now I want my jobs to crash loudly, for that purpose I use slacktee util that posts stdin to slack.
1#!/bin/bash 2 3# Take a command to run in argument. Posts stderr to slack if it fails. 4runcheck() { 5 ERR="$($@ 2>&1 > /dev/null)" 6 if [ $? -ne 0 ]; then 7 echo "\`${1##*/}\` failed" | /usr/local/bin/slacktee.sh -p -u "$HOSTNAME" --config /home/ubuntu/.slacktee-alert 8 if [ ! -z $ERR ]; then 9 echo "${ERR}" | sed -e 's/^/>/' | /usr/local/bin/slacktee.sh -p -u "$HOSTNAME" --config /home/ubuntu/.slacktee-alert 10 fi 11 fi 12} 13 14main() { 15 runcheck $@ 16} 17main $@
runcheck.sh
So my cron tasks are declared like so :
In case of failure, a slack alert pops with all needed infos : hostname, command that fails and error message.