Project

General

Profile

Function to run a command

Introduction

When writing bash with full command error trapping including timeouts, very similar error trapping code is written multiple times. Here's a solution.

Globals

Called function

msg: Generalised messaging function

Function run_cmd_with_timeout

#!/bin/bash   <- Does nothing but triggers editor syntax highlighting

# Copyright (C) 2019 Charles Atkinson
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#--------------------------
# Name: run_cmd_with_timeout
# Purpose:
#   * Optionally checks the configured ssh connection
#   * Runs the command in array $cmd
#   * Puts any output of the command in file $out_fn
#   * Puts any return code from the command in file $rc_fn
#   * Sets return code:
#     0 - No warning message was generated by this function
#     1 - A warning message was generated by this function
#     2 - Timed out
#
# Usage:
#   * Before calling this function:
#       * Array ${cmd[*]} must be loaded with a command.
#         It must be a simple command; cannot include | < > &
#       * $out_fn and $rc_fn must contain paths of writeable files.
#         Any existing content in the files will be removed.
# Syntax:
#   run_cmd_with_timeout
#       [-o <rc test list>] [-w <rc test list>] [-e <rc test list>]
#       [-O <regex>] [-W <regex>] [-E <regex>]
#       [-s <ssh host>]
#       [-t <timeout>[,timeout>] [-T <msg class>[,<msg_class>]]
# Options:
#   -o <rc test list> OK return codes.
#      If none of the tests match the return code:
#          If -e is also used, message class warning is set
#          Otherwise (-w is also used), message class error is set
#   -w <rc test list> warning return codes.
#      If any of the tests match the return code, message class warning is set
#   -e <rc test list> error return codes.
#      If any of the tests match the return code, message class error is set
#   -O specifies regex for OK output
#   -W specifies regex for warning output
#   -E specifies regex for error output
#   -s <[<user@>]ssh host> Test the ssh connection to the ssh host before
#      running the command.
#      If <ssh_host> is an entry in the script runner's ~/.ssh/config file, <user@>
#      is not required
#   -t <timeout>[,timeout>] Duration before timing out the command and any
#      remote host connection test
#   -T <msg class>[,<msg_class>] Class of message to generate on timeout
#
# rc_test_list format:
#   One or more of > < >= <= != == followed by an unsigned integer and
#   separated by commas.  Example: >8,==2
#
# -o -w and -e usage notes:
#   * When none are specified, return code >0 generates an error message
#   * When one or more are specified:
#     If e is specified and its rc list matches, generates an error message
#     Else if w is specified and its rc list matches, generates a warning message
#     Else if o is specified and o's rc list does not match:
#         If e is specified, generates a warning message
#         Else generates an error message
#
# -O -W and -E usage notes ("output" is combined stdout and stderr):
#   * When none are specified, output is ignored
#   * When one or more are specified:
#     If E is specified and its regex matches output,
#         generates an error message
#     Else if W is specified and its regex matches output,
#         generates a warning message
#     Else if O is specified and O's regex does not match output,
#         If E is specified, generates a warning message
#         Else generates an error message
#
# Timeout (-t) list format
#   * Comma separated list
#   * First member specifies command timeout
#   * Second member specifies any remote host connection test timeout (-s option)
#   * Members must be empty or an unsigned integer or float with an optional suffix:
#         s for seconds (the default)
#         m for minutes
#         h for hours
#         d for days
#   * Default: 10 for each unspecified member
#
# Timeout message class (-T) list format
#   * Comma separated list
#   * First member specifies message class on command timeout
#   * Second member specifies message class on any remote host connection test
#   * Members must be an empty string, I, W or E
#   * Default (when -t not used): two empty strings meaning no message is
#     generated on timeouts

# After calling this function caller could:
#   * Examine this function's return code
#   * Read the command's output from $out_fn
#   * Read the command's return code from $rc_fn
#
# Example 1
#   cmd=(ssh "$router_FQDN" file print detail)
#   run_cmd_with_timeout
#   case $? in
#       1 | 2 )
#           fct "${FUNCNAME[0]}" 'returning 1'
#           return 1
#   esac
#
# Example 2
#    msg I "Checking the $user@$hostname ssh connection" 
#    cmd=(ssh "$user@$hostname" 'echo OK')
#    run_cmd_with_timeout -t 5
#    case $? in
#        0 )
#            msg I 'ssh connection OK'
#            ;;
#        1 | 2 )
#            msg W 'ssh connection check failed'
#            fct "${FUNCNAME[0]}" 'returning 1'
#            return 1
#    esac
#
# Global variables read:
#   cmd
#   false
#   msg_lf
#   out_fn
#   rc_fn
#   true
# Global variables set: none
# Output:
#   * stdout and stderr to log or screen, either directly or via msg function
# Returns: described above
#--------------------------
function run_cmd_with_timeout {
    fct "${FUNCNAME[0]}" "started with arguments $*" 
    local OPTIND    # Required when getopts is called in a function
    local args opt
    local opt_e_flag opt_E_flag opt_o_flag opt_O_flag opt_w_flag opt_W_flag
    local opt_e_rc_list opt_o_rc_list opt_t_arg opt_w_rc_list
    local opt_E_regex opt_O_regex opt_T_arg opt_W_regex
    local cmd_timeout cmd_timeout_msg_class ssh_host
    local connection_timeout connection_timeout_msg_class
    local -r duration_OK_regex='^\+?[[:digit:]]*\.?[[:digit:]]+(|d|h|m|s)$'
    local -r timeout_msg_class_OK_regex='^(|E|I|W)$'
    local array buf emsg msg msg_class my_rc oldIFS out rc rc_for_msg timed_out_flag

    # Parse options
    # ~~~~~~~~~~~~~
    args=("$@")
    emsg=
    emsg_warn_regex=
    opt_e_flag=$false
    opt_E_flag=$false
    opt_e_rc_list=
    opt_E_regex=
    opt_o_flag=$false
    opt_O_flag=$false
    opt_o_rc_list=
    opt_O_regex=
    opt_t_arg=,
    opt_T_arg=,
    opt_w_flag=$false
    opt_W_flag=$false
    opt_w_rc_list=
    opt_W_regex=
    ssh_host=
    while getopts :e:E:o:O:s:t:T:w:W: opt "$@" 
    do
        case $opt in
            e )
                opt_e_flag=$true
                opt_e_rc_list=$OPTARG
                ;;
            E )
                opt_E_flag=$true
                opt_E_regex=$OPTARG
                ;;
            o )
                opt_o_flag=$true
                opt_o_rc_list=$OPTARG
                ;;
            O )
                opt_O_flag=$true
                opt_O_regex=$OPTARG
                ;;
            s )
                ssh_host=$OPTARG
                ;;
            t )
                opt_t_arg=$OPTARG
                ;;
            T )
                opt_T_arg=$OPTARG
                ;;
            w )
                opt_w_flag=$true
                opt_w_rc_list=$OPTARG
                ;;
            W )
                opt_W_flag=$true
                opt_W_regex=$OPTARG
                ;;
            : )
                emsg+=$msg_lf"Option $OPTARG must have an argument" 
                ;;
            * )
                emsg+=$msg_lf"Invalid option '-$OPTARG'" 
        esac
    done
    shift $(($OPTIND-1))

    # Test for mutually exclusive options
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # There are no mutually exclusive options

    # Test for mandatory options not set
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # There are no mandatory options

    # Parse option arguments
    # ~~~~~~~~~~~~~~~~~~~~~~
    oldIFS=$IFS
    IFS=,
    array=($opt_t_arg)
    if ((${#array[*]}<3)); then
        if [[ ${array[0]:=10} =~ $duration_OK_regex ]]; then
            cmd_timeout=${array[0]}
        else
            emsg+=$msg_lf"-t option arg ${array[0]} invalid" 
            emsg+=" (does not match $duration_OK_regex)" 
        fi
        if [[ ${array[1]:=10} =~ $duration_OK_regex ]]; then
            connection_timeout=${array[1]}
        else
            emsg+=$msg_lf"-t option arg ${array[1]} invalid" 
            emsg+=" (does not match $duration_OK_regex)" 
        fi
    else
        emsg+=$msg_lf"-t option: more than two comma-separated values" 
        emsg+=" ($opt_t_arg)" 
    fi
    array=($opt_T_arg)
    if ((${#array[*]}<3)); then
        if [[ ${array[0]} =~ $timeout_msg_class_OK_regex ]]; then
            cmd_timeout_msg_class=${array[0]}
        else
            emsg+=$msg_lf"-T option arg ${array[0]} invalid" 
            emsg+=" (does not match $timeout_msg_class_OK_regex)" 
        fi
        [[ ${array[1]:-} = '' ]] && array[1]=    # Default
        if [[ ${array[1]} =~ $timeout_msg_class_OK_regex ]]; then
            connection_timeout_msg_class=${array[1]}
        else
            emsg+=$msg_lf"-t option arg ${array[1]} invalid" 
            emsg+=" (does not match $timeout_msg_class_OK_regex)" 
        fi
    else
        emsg+=$msg_lf"-T option: more than two comma-separated values" 
        emsg+=" ($opt_T_arg)" 
    fi
    IFS=$oldIFS

    # Error trap globals
    # ~~~~~~~~~~~~~~~~~~
    [[ ${cmd:-} = '' ]] && emsg+=$msg_lf'$cmd is required but is unset or empty'
    if [[ ${out_fn:-} != '' ]]; then
        touch "$out_fn" 2>/dev/null    # In case does not exist
        buf=$(ck_file "$out_fn" f:rw 2>&1)
        [[ $buf != '' ]] && emsg+=$msg_lf$buf
    else
        emsg+=$msg_lf'$out_fn is required but is unset or empty'
    fi
    if [[ ${rc_fn:-} != '' ]]; then
        touch "$rc_fn" 2>/dev/null    # In case does not exist
        buf=$(ck_file "$rc_fn" f:rw 2>&1)
        [[ $buf != '' ]] && emsg+=$msg_lf$buf
    else
        emsg+=$msg_lf'$rc_fn is required but is unset or empty'
    fi

    # Report any invocation errors
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    if [[ $emsg != '' ]]; then
        msg E "Programming error: ${FUNCNAME[0]} (args ${args[*]})$emsg" 
        return 1    # Required when being run from finalise function
    fi
    msg D "opt_o_rc_list: $opt_o_rc_list, opt_w_rc_list: $opt_w_rc_list, opt_e_rc_list: $opt_e_rc_list" 
    msg D "opt_O_regex: $opt_O_regex, opt_W_regex: $opt_W_regex, opt_E_regex: $opt_E_regex" 

    # Check ssh connection if requested
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    if [[ $ssh_host != '' ]]; then
        msg="Testing the $ssh_host ssh connection" 
        msg I "$msg with timeout $connection_timeout" 
        echo -n > "$out_fn"; echo -n > "$rc_fn"    # Ensure empty
        msg_class=
        timeout --signal=SIGTERM --kill-after=$connection_timeout \
            $connection_timeout ssh "$ssh_host" echo -n OK > "$out_fn" 2>&1
        rc=$?
        if ((rc==124||rc==137)); then
            [[ $connection_timeout_msg_class != '' ]] \
                && msg $connection_timeout_msg_class \
                   'Timed out'
            fct "${FUNCNAME[0]}" 'returning 2'
            return 2
        fi
        msg I OK
    fi

    # Run the command with timeout
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    msg I "Running command with timeout $cmd_timeout: ${cmd[*]}" 
    echo -n > "$out_fn"; echo -n > "$rc_fn"    # Ensure empty
    timeout --signal=SIGTERM --kill-after=$cmd_timeout \
        $cmd_timeout "${cmd[@]}" > "$out_fn" 2>&1
    rc=$?
    if ((rc==124||rc==137)); then
        timed_out_flag=$true
        msg I 'Timed out'
        rc_for_msg='not available (timed out)'
    else
        timed_out_flag=$false
        echo $rc > "$rc_fn" 
        rc_for_msg=$rc
    fi
    msg_class=

    # Examine any return code if requested
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # The return code is not available after timeout
    # -o -w and -e usage notes:
    #   * When none are specified, return code >0 generates an error message
    #   * When one or more are specified:
    #     If e is specified and its rc list matches, generates an error message
    #     Else if w is specified and its rc list matches, generates a warning message
    #     Else if o is specified and o's rc list does not match:
    #         If e is specified, generates a warning message
    #         Else generates an error message
    if [[ ! $timed_out_flag ]]; then
        msg D "${FUNCNAME[0]}: Examining return code, $rc" 
        if [[ $opt_e_rc_list != '' ]]; then
            msg D "${FUNCNAME[0]}: -e specified" 
            run_rc_tests $opt_e_rc_list && msg_class=E
        elif [[ $opt_w_rc_list != '' ]]; then
            msg D "${FUNCNAME[0]}: -w specified" 
            run_rc_tests $opt_w_rc_list && msg_class=W
        elif [[ $opt_o_rc_list != '' ]] && ! run_rc_tests $opt_o_rc_list; then
            msg D "${FUNCNAME[0]}: -o specified and not matched" 
            [[ $opt_e_rc_list = '' ]] && msg_class=W || msg_class=E
        else
            msg D "${FUNCNAME[0]}: None of -o -w or -e specified" 
            (($rc>0)) && msg_class=E
        fi
        msg D "${FUNCNAME[0]}: After examining return code, msg_class: $msg_class" 
    fi

    # Examine any output if requested
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # -O -W and -E usage notes ("output" is combined stdout and stderr):
    #   * When none are specified, output is ignored
    #   * When one or more are specified:
    #     If E is specified and its regex matches output,
    #         generates an error message
    #     Else if W is specified and its regex matches output,
    #         generates a warning message
    #     Else if O is specified and O's regex does not match output,
    #         If E is specified, generates a warning message
    #         Else generates an error message
    out=$(< "$out_fn")
    if [[ $out != '' ]]; then
        msg D "${FUNCNAME[0]}: Examining output" 
        if [[ $opt_E_regex != '' && "$out" =~ $opt_E_regex ]]; then
            msg D "${FUNCNAME[0]}: -E specified and matched" 
            msg_class=E
        elif [[ $opt_W_regex != '' && "$out" =~ $opt_W_regex ]]; then
            msg D "${FUNCNAME[0]}: -W specified and matched" 
            [[ $msg_class != E ]] && msg_class=W
        elif [[ $opt_O_regex != '' ]]; then
            if [[ ! "$out" =~ $opt_O_regex ]]; then
                msg D "${FUNCNAME[0]}: -O specified ($opt_O_regex) and not matched" 
                if [[ $opt_E_regex = '' ]]; then
                    msg D "${FUNCNAME[0]}: -E specified" 
                    [[ $msg_class != E ]] && msg_class=W
                else
                    msg D "${FUNCNAME[0]}: -E not specified" 
                    msg_class=E
                fi
            fi
        fi
        msg D "${FUNCNAME[0]}: After examining output, msg_class: $msg_class" 
    fi

    # Generate message if required
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    if [[ $msg_class != '' ]]; then
        msg="Command: ${cmd[*]}" 
        msg+=$'\n'"rc: $rc_for_msg" 
        msg+=$'\n'"Output: $out" 
        msg $msg_class "$msg"    # Does not return when msg_class is E
    fi

    # Set my return code
    # ~~~~~~~~~~~~~~~~~~
    #   0 - No problem detected with the command
    #   1 - A problem other than timeout was detected
    #   2 - The command timed out
    if [[ $timed_out_flag ]]; then
        my_rc=2
    elif [[ $msg_class = W ]]; then
        my_rc=1
    else
        my_rc=0
    fi

    fct "${FUNCNAME[0]}" "returning $my_rc" 
    return $my_rc
}  #  end of function run_cmd_with_timeout

#--------------------------
# Name: run_rc_tests
# Purpose:
#   * Runs return code tests (for run_cmd_with_timeout)
# Arguments:
#   $1 - comma separated list of arithmentical tests
#
# Global variables read: rc
# Global variables set: none
# Output:
#   * stdout and stderr to log or screen, either directly or via msg function
# Returns:
#   0 - $rc matched one of the tests
#   1 - $rc did not match any of the tests
#--------------------------
function run_rc_tests {
    local array i matched numcom oldIFS one_char rhs two_char
    local -r two_char_numcom_regex='^(<|>|=|!)=$'
    local -r one_char_numcom_regex='^(<|>)$'
    local -r uint_regex='^[[:digit:]]+$'

    # Parse the argument
    # ~~~~~~~~~~~~~~~~~~
    oldIFS=$IFS
    IFS=,
    array=($1)
    IFS=$oldIFS

    # For each test
    # ~~~~~~~~~~~~~
    for ((i=0;i<${#array[*]};i++))
    do
       # Parse the test
       # ~~~~~~~~~~~~~~
       two_char=${array[i]:0:2}
       one_char=${array[i]:0:1}
       if [[ $two_char =~ $two_char_numcom_regex ]]; then
           numcom=$two_char
           rhs=${array[i]:2}
       elif [[ $one_char =~ $one_char_numcom_regex ]]; then
           numcom=$one_char
           rhs=${array[i]:1}
       else
           msg E "Programming error: invalid numerc comparison ${array[i]} in $1" 
       fi
       [[ ! $rhs =~ $uint_regex ]] \
           &&  msg E "Programming error: invalid numerc comparison ${array[i]} in $1" 
       msg D "rc: $rc, operator: $numcom, rhs: $rhs" 

       # Test
       # ~~~~
       matched=$false
       case $numcom in
           '<=' ) ((rc<=rhs)) && matched=$true ;;
           '>=' ) ((rc>=rhs)) && matched=$true ;;
           '==' ) ((rc==rhs)) && matched=$true ;;
           '!=' ) ((rc!=rhs)) && matched=$true ;;
           '<'  ) ((rc<rhs))  && matched=$true ;;
           '>'  ) ((rc>rhs))  && matched=$true ;;
       esac
       if [[ $matched ]]; then
           msg D 'An rc comparison matched, returning 0'
           return 0
       fi

    done

    msg D 'No rc comparisons matched, returning 1'
    return 1
}  #  end of function run_rc_tests