Public pages

h1. Function to run a command

{{toc}}

h1. Introduction

When writing bash with full command error trapping including timeouts, very similar error trapping code is written multiple times. Here's a solution.

h1. h2. Globals

* [[Globals used in several of these functions]]
* cmd is set by caller

h1. h2. Called function

msg: [[Generalised messaging function]]

h1. Function run_cmd_with_timeout

<pre>
#!/bin/bash <- Does nothing but triggers editor syntax highlighting

# Copyright (C) 2019 Charles Atkinson
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

#--------------------------
# Name: run_cmd_with_timeout
# Purpose:
# * Optionally checks the configured ssh connection
# * Runs the command in array $cmd
# * Puts any output of the command in file $out_fn
# * Puts any return code from the command in file $rc_fn
# * Sets return code:
# 0 - No warning message was generated by this function
# 1 - A warning message was generated by this function
# 2 - Timed out
#
# Usage:
# * Before calling this function:
# * Array ${cmd[*]} must be loaded with a command.
# It must be a simple command; cannot include | < > &
# * $out_fn and $rc_fn must contain paths of writeable files.
# Any existing content in the files will be removed.
# Syntax:
# run_cmd_with_timeout
# [-o <rc test list>] [-w <rc test list>] [-e <rc test list>]
# [-O <regex>] [-W <regex>] [-E <regex>]
# [-s <ssh host>]
# [-t <timeout>[,timeout>] [-T <msg class>[,<msg_class>]]
# Options:
# -o <rc test list> OK return codes.
# If none of the tests match the return code:
# If -e is also used, message class warning is set
# Otherwise (-w is also used), message class error is set
# -w <rc test list> warning return codes.
# If any of the tests match the return code, message class warning is set
# -e <rc test list> error return codes.
# If any of the tests match the return code, message class error is set
# -O specifies regex for OK output
# -W specifies regex for warning output
# -E specifies regex for error output
# -s <[<user@>]ssh host> Test the ssh connection to the ssh host before
# running the command.
# If <ssh_host> is an entry in the script runner's ~/.ssh/config file, <user@>
# is not required
# -t <timeout>[,timeout>] Duration before timing out the command and any
# remote host connection test
# -T <msg class>[,<msg_class>] Class of message to generate on timeout
#
# rc_test_list format:
# One or more of > < >= <= != == followed by an unsigned integer and
# separated by commas. Example: >8,==2
#
# -o -w and -e usage notes:
# * When none are specified, return code >0 generates an error message
# * When one or more are specified:
# If e is specified and its rc list matches, generates an error message
# Else if w is specified and its rc list matches, generates a warning message
# Else if o is specified and o's rc list does not match:
# If e is specified, generates a warning message
# Else generates an error message
#
# -O -W and -E usage notes ("output" is combined stdout and stderr):
# * When none are specified, output is ignored
# * When one or more are specified:
# If E is specified and its regex matches output,
# generates an error message
# Else if W is specified and its regex matches output,
# generates a warning message
# Else if O is specified and O's regex does not match output,
# If E is specified, generates a warning message
# Else generates an error message
#
# Timeout (-t) list format
# * Comma separated list
# * First member specifies command timeout
# * Second member specifies any remote host connection test timeout (-s option)
# * Members must be empty or an unsigned integer or float with an optional suffix:
# s for seconds (the default)
# m for minutes
# h for hours
# d for days
# * Default: 10 for each unspecified member
#
# Timeout message class (-T) list format
# * Comma separated list
# * First member specifies message class on command timeout
# * Second member specifies message class on any remote host connection test
# * Members must be an empty string, I, W or E
# * Default (when -t not used): two empty strings meaning no message is
# generated on timeouts

# After calling this function caller could:
# * Examine this function's return code
# * Read the command's output from $out_fn
# * Read the command's return code from $rc_fn
#
# Example 1
# cmd=(ssh "$router_FQDN" file print detail)
# run_cmd_with_timeout
# case $? in
# 1 | 2 )
# fct "${FUNCNAME[0]}" 'returning 1'
# return 1
# esac
#
# Example 2
# msg I "Checking the $user@$hostname ssh connection"
# cmd=(ssh "$user@$hostname" 'echo OK')
# run_cmd_with_timeout -t 5
# case $? in
# 0 )
# msg I 'ssh connection OK'
# ;;
# 1 | 2 )
# msg W 'ssh connection check failed'
# fct "${FUNCNAME[0]}" 'returning 1'
# return 1
# esac
#
# Global variables read:
# cmd
# false
# msg_lf
# out_fn
# rc_fn
# true
# Global variables set: none
# Output:
# * stdout and stderr to log or screen, either directly or via msg function
# Returns: described above
#--------------------------
function run_cmd_with_timeout {
fct "${FUNCNAME[0]}" "started with arguments $*"
local OPTIND # Required when getopts is called in a function
local args opt
local opt_e_flag opt_E_flag opt_o_flag opt_O_flag opt_w_flag opt_W_flag
local opt_e_rc_list opt_o_rc_list opt_t_arg opt_w_rc_list
local opt_E_regex opt_O_regex opt_T_arg opt_W_regex
local cmd_timeout cmd_timeout_msg_class ssh_host
local connection_timeout connection_timeout_msg_class
local -r duration_OK_regex='^\+?[[:digit:]]*\.?[[:digit:]]+(|d|h|m|s)$'
local -r timeout_msg_class_OK_regex='^(|E|I|W)$'
local array buf emsg msg msg_class my_rc oldIFS out rc rc_for_msg timed_out_flag

# Parse options
# ~~~~~~~~~~~~~
args=("$@")
emsg=
emsg_warn_regex=
opt_e_flag=$false
opt_E_flag=$false
opt_e_rc_list=
opt_E_regex=
opt_o_flag=$false
opt_O_flag=$false
opt_o_rc_list=
opt_O_regex=
opt_t_arg=,
opt_T_arg=,
opt_w_flag=$false
opt_W_flag=$false
opt_w_rc_list=
opt_W_regex=
ssh_host=
while getopts :e:E:o:O:s:t:T:w:W: opt "$@"
do
case $opt in
e )
opt_e_flag=$true
opt_e_rc_list=$OPTARG
;;
E )
opt_E_flag=$true
opt_E_regex=$OPTARG
;;
o )
opt_o_flag=$true
opt_o_rc_list=$OPTARG
;;
O )
opt_O_flag=$true
opt_O_regex=$OPTARG
;;
s )
ssh_host=$OPTARG
;;
t )
opt_t_arg=$OPTARG
;;
T )
opt_T_arg=$OPTARG
;;
w )
opt_w_flag=$true
opt_w_rc_list=$OPTARG
;;
W )
opt_W_flag=$true
opt_W_regex=$OPTARG
;;
: )
emsg+=$msg_lf"Option $OPTARG must have an argument"
;;
* )
emsg+=$msg_lf"Invalid option '-$OPTARG'"
esac
done
shift $(($OPTIND-1))

# Test for mutually exclusive options
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# There are no mutually exclusive options

# Test for mandatory options not set
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# There are no mandatory options

# Parse option arguments
# ~~~~~~~~~~~~~~~~~~~~~~
oldIFS=$IFS
IFS=,
array=($opt_t_arg)
if ((${#array[*]}<3)); then
if [[ ${array[0]:=10} =~ $duration_OK_regex ]]; then
cmd_timeout=${array[0]}
else
emsg+=$msg_lf"-t option arg ${array[0]} invalid"
emsg+=" (does not match $duration_OK_regex)"
fi
if [[ ${array[1]:=10} =~ $duration_OK_regex ]]; then
connection_timeout=${array[1]}
else
emsg+=$msg_lf"-t option arg ${array[1]} invalid"
emsg+=" (does not match $duration_OK_regex)"
fi
else
emsg+=$msg_lf"-t option: more than two comma-separated values"
emsg+=" ($opt_t_arg)"
fi
array=($opt_T_arg)
if ((${#array[*]}<3)); then
if [[ ${array[0]} =~ $timeout_msg_class_OK_regex ]]; then
cmd_timeout_msg_class=${array[0]}
else
emsg+=$msg_lf"-T option arg ${array[0]} invalid"
emsg+=" (does not match $timeout_msg_class_OK_regex)"
fi
[[ ${array[1]:-} = '' ]] && array[1]= # Default
if [[ ${array[1]} =~ $timeout_msg_class_OK_regex ]]; then
connection_timeout_msg_class=${array[1]}
else
emsg+=$msg_lf"-t option arg ${array[1]} invalid"
emsg+=" (does not match $timeout_msg_class_OK_regex)"
fi
else
emsg+=$msg_lf"-T option: more than two comma-separated values"
emsg+=" ($opt_T_arg)"
fi
IFS=$oldIFS

# Error trap globals
# ~~~~~~~~~~~~~~~~~~
[[ ${cmd:-} = '' ]] && emsg+=$msg_lf'$cmd is required but is unset or empty'
if [[ ${out_fn:-} != '' ]]; then
touch "$out_fn" 2>/dev/null # In case does not exist
buf=$(ck_file "$out_fn" f:rw 2>&1)
[[ $buf != '' ]] && emsg+=$msg_lf$buf
else
emsg+=$msg_lf'$out_fn is required but is unset or empty'
fi
if [[ ${rc_fn:-} != '' ]]; then
touch "$rc_fn" 2>/dev/null # In case does not exist
buf=$(ck_file "$rc_fn" f:rw 2>&1)
[[ $buf != '' ]] && emsg+=$msg_lf$buf
else
emsg+=$msg_lf'$rc_fn is required but is unset or empty'
fi

# Report any invocation errors
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if [[ $emsg != '' ]]; then
msg E "Programming error: ${FUNCNAME[0]} (args ${args[*]})$emsg"
return 1 # Required when being run from finalise function
fi
msg D "opt_o_rc_list: $opt_o_rc_list, opt_w_rc_list: $opt_w_rc_list, opt_e_rc_list: $opt_e_rc_list"
msg D "opt_O_regex: $opt_O_regex, opt_W_regex: $opt_W_regex, opt_E_regex: $opt_E_regex"

# Check ssh connection if requested
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if [[ $ssh_host != '' ]]; then
msg="Testing the $ssh_host ssh connection"
msg I "$msg with timeout $connection_timeout"
echo -n > "$out_fn"; echo -n > "$rc_fn" # Ensure empty
msg_class=
timeout --signal=SIGTERM --kill-after=$connection_timeout \
$connection_timeout ssh "$ssh_host" echo -n OK > "$out_fn" 2>&1
rc=$?
if ((rc==124||rc==137)); then
[[ $connection_timeout_msg_class != '' ]] \
&& msg $connection_timeout_msg_class \
'Timed out'
fct "${FUNCNAME[0]}" 'returning 2'
return 2
fi
msg I OK
fi

# Run the command with timeout
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
msg I "Running command with timeout $cmd_timeout: ${cmd[*]}"
echo -n > "$out_fn"; echo -n > "$rc_fn" # Ensure empty
timeout --signal=SIGTERM --kill-after=$cmd_timeout \
$cmd_timeout "${cmd[@]}" > "$out_fn" 2>&1
rc=$?
if ((rc==124||rc==137)); then
timed_out_flag=$true
msg I 'Timed out'
rc_for_msg='not available (timed out)'
else
timed_out_flag=$false
echo $rc > "$rc_fn"
rc_for_msg=$rc
fi
msg_class=

# Examine any return code if requested
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# The return code is not available after timeout
# -o -w and -e usage notes:
# * When none are specified, return code >0 generates an error message
# * When one or more are specified:
# If e is specified and its rc list matches, generates an error message
# Else if w is specified and its rc list matches, generates a warning message
# Else if o is specified and o's rc list does not match:
# If e is specified, generates a warning message
# Else generates an error message
if [[ ! $timed_out_flag ]]; then
msg D "${FUNCNAME[0]}: Examining return code, $rc"
if [[ $opt_e_rc_list != '' ]]; then
msg D "${FUNCNAME[0]}: -e specified"
run_rc_tests $opt_e_rc_list && msg_class=E
elif [[ $opt_w_rc_list != '' ]]; then
msg D "${FUNCNAME[0]}: -w specified"
run_rc_tests $opt_w_rc_list && msg_class=W
elif [[ $opt_o_rc_list != '' ]] && ! run_rc_tests $opt_o_rc_list; then
msg D "${FUNCNAME[0]}: -o specified and not matched"
[[ $opt_e_rc_list = '' ]] && msg_class=W || msg_class=E
else
msg D "${FUNCNAME[0]}: None of -o -w or -e specified"
(($rc>0)) && msg_class=E
fi
msg D "${FUNCNAME[0]}: After examining return code, msg_class: $msg_class"
fi

# Examine any output if requested
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# -O -W and -E usage notes ("output" is combined stdout and stderr):
# * When none are specified, output is ignored
# * When one or more are specified:
# If E is specified and its regex matches output,
# generates an error message
# Else if W is specified and its regex matches output,
# generates a warning message
# Else if O is specified and O's regex does not match output,
# If E is specified, generates a warning message
# Else generates an error message
out=$(< "$out_fn")
if [[ $out != '' ]]; then
msg D "${FUNCNAME[0]}: Examining output"
if [[ $opt_E_regex != '' && "$out" =~ $opt_E_regex ]]; then
msg D "${FUNCNAME[0]}: -E specified and matched"
msg_class=E
elif [[ $opt_W_regex != '' && "$out" =~ $opt_W_regex ]]; then
msg D "${FUNCNAME[0]}: -W specified and matched"
[[ $msg_class != E ]] && msg_class=W
elif [[ $opt_O_regex != '' ]]; then
if [[ ! "$out" =~ $opt_O_regex ]]; then
msg D "${FUNCNAME[0]}: -O specified ($opt_O_regex) and not matched"
if [[ $opt_E_regex = '' ]]; then
msg D "${FUNCNAME[0]}: -E specified"
[[ $msg_class != E ]] && msg_class=W
else
msg D "${FUNCNAME[0]}: -E not specified"
msg_class=E
fi
fi
fi
msg D "${FUNCNAME[0]}: After examining output, msg_class: $msg_class"
fi

# Generate message if required
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if [[ $msg_class != '' ]]; then
msg="Command: ${cmd[*]}"
msg+=$'\n'"rc: $rc_for_msg"
msg+=$'\n'"Output: $out"
msg $msg_class "$msg" # Does not return when msg_class is E
fi

# Set my return code
# ~~~~~~~~~~~~~~~~~~
# 0 - No problem detected with the command
# 1 - A problem other than timeout was detected
# 2 - The command timed out
if [[ $timed_out_flag ]]; then
my_rc=2
elif [[ $msg_class = W ]]; then
my_rc=1
else
my_rc=0
fi

fct "${FUNCNAME[0]}" "returning $my_rc"
return $my_rc
} # end of function run_cmd_with_timeout

#--------------------------
# Name: run_rc_tests
# Purpose:
# * Runs return code tests (for run_cmd_with_timeout)
# Arguments:
# $1 - comma separated list of arithmentical tests
#
# Global variables read: rc
# Global variables set: none
# Output:
# * stdout and stderr to log or screen, either directly or via msg function
# Returns:
# 0 - $rc matched one of the tests
# 1 - $rc did not match any of the tests
#--------------------------
function run_rc_tests {
local array i matched numcom oldIFS one_char rhs two_char
local -r two_char_numcom_regex='^(<|>|=|!)=$'
local -r one_char_numcom_regex='^(<|>)$'
local -r uint_regex='^[[:digit:]]+$'

# Parse the argument
# ~~~~~~~~~~~~~~~~~~
oldIFS=$IFS
IFS=,
array=($1)
IFS=$oldIFS

# For each test
# ~~~~~~~~~~~~~
for ((i=0;i<${#array[*]};i++))
do
# Parse the test
# ~~~~~~~~~~~~~~
two_char=${array[i]:0:2}
one_char=${array[i]:0:1}
if [[ $two_char =~ $two_char_numcom_regex ]]; then
numcom=$two_char
rhs=${array[i]:2}
elif [[ $one_char =~ $one_char_numcom_regex ]]; then
numcom=$one_char
rhs=${array[i]:1}
else
msg E "Programming error: invalid numerc comparison ${array[i]} in $1"
fi
[[ ! $rhs =~ $uint_regex ]] \
&& msg E "Programming error: invalid numerc comparison ${array[i]} in $1"
msg D "rc: $rc, operator: $numcom, rhs: $rhs"

# Test
# ~~~~
matched=$false
case $numcom in
'<=' ) ((rc<=rhs)) && matched=$true ;;
'>=' ) ((rc>=rhs)) && matched=$true ;;
'==' ) ((rc==rhs)) && matched=$true ;;
'!=' ) ((rc!=rhs)) && matched=$true ;;
'<' ) ((rc<rhs)) && matched=$true ;;
'>' ) ((rc>rhs)) && matched=$true ;;
esac
if [[ $matched ]]; then
msg D 'An rc comparison matched, returning 0'
return 0
fi

done

msg D 'No rc comparisons matched, returning 1'
return 1
} # end of function run_rc_tests
</pre>