Public pages

h1. Function to run a command

{{toc}}

h1. Introduction

When writing bash with full command error trapping including timeouts, very similar error trapping code is written multiple times.  Here's a solution.

h1. Globals

* [[Globals used in several of these functions]]

* cmd is set by caller

h1. Called function

msg: [[Generalised messaging function]]

h1. Function run_cmd_with_timeout

<pre>

#!/bin/bash   <- Does nothing but triggers editor syntax highlighting

# Copyright (C) 2019 Charles Atkinson

#

# This program is free software; you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation; either version 2 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program; if not, write to the Free Software

# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#--------------------------

# Name: run_cmd_with_timeout

# Purpose:

#   * Optionally checks the configured ssh connection

#   * Runs the command in array $cmd

#   * Puts any output of the command in file $out_fn

#   * Puts any return code from the command in file $rc_fn

#   * Sets return code:

#     0 - No warning message was generated by this function

#     1 - A warning message was generated by this function

#     2 - Timed out

#

# Usage:

#   * Before calling this function:

#       * Array ${cmd[*]} must be loaded with a command.

#         It must be a simple command; cannot include | < > &

#       * $out_fn and $rc_fn must contain paths of writeable files.

#         Any existing content in the files will be removed.

# Syntax:

#   run_cmd_with_timeout

#       [-o <rc test list>] [-w <rc test list>] [-e <rc test list>]

#       [-O <regex>] [-W <regex>] [-E <regex>]

#       [-s <ssh host>]

#       [-t <timeout>[,timeout>] [-T <msg class>[,<msg_class>]]

# Options:

#   -o <rc test list> OK return codes.

#      If none of the tests match the return code:

#          If -e is also used, message class warning is set

#          Otherwise (-w is also used), message class error is set

#   -w <rc test list> warning return codes.

#      If any of the tests match the return code, message class warning is set

#   -e <rc test list> error return codes.

#      If any of the tests match the return code, message class error is set

#   -O specifies regex for OK output

#   -W specifies regex for warning output

#   -E specifies regex for error output

#   -s <[<user@>]ssh host> Test the ssh connection to the ssh host before

#      running the command.

#      If <ssh_host> is an entry in the script runner's ~/.ssh/config file, <user@>

#      is not required

#   -t <timeout>[,timeout>] Duration before timing out the command and any

#      remote host connection test

#   -T <msg class>[,<msg_class>] Class of message to generate on timeout

#

# rc_test_list format:

#   One or more of > < >= <= != == followed by an unsigned integer and

#   separated by commas.  Example: >8,==2

#

# -o -w and -e usage notes:

#   * When none are specified, return code >0 generates an error message

#   * When one or more are specified:

#     If e is specified and its rc list matches, generates an error message

#     Else if w is specified and its rc list matches, generates a warning message

#     Else if o is specified and o's rc list does not match:

#         If e is specified, generates a warning message

#         Else generates an error message

#

# -O -W and -E usage notes ("output" is combined stdout and stderr):

#   * When none are specified, output is ignored

#   * When one or more are specified:

#     If E is specified and its regex matches output,

#         generates an error message

#     Else if W is specified and its regex matches output,

#         generates a warning message

#     Else if O is specified and O's regex does not match output,

#         If E is specified, generates a warning message

#         Else generates an error message

#

# Timeout (-t) list format

#   * Comma separated list

#   * First member specifies command timeout

#   * Second member specifies any remote host connection test timeout (-s option)

#   * Members must be empty or an unsigned integer or float with an optional suffix:

#         s for seconds (the default)

#         m for minutes

#         h for hours

#         d for days

#   * Default: 10 for each unspecified member

#

# Timeout message class (-T) list format

#   * Comma separated list

#   * First member specifies message class on command timeout

#   * Second member specifies message class on any remote host connection test

#   * Members must be an empty string, I, W or E

#   * Default (when -t not used): two empty strings meaning no message is

#     generated on timeouts

# After calling this function caller could:

#   * Examine this function's return code

#   * Read the command's output from $out_fn

#   * Read the command's return code from $rc_fn

#

# Example 1

#   cmd=(ssh "$router_FQDN" file print detail)

#   run_cmd_with_timeout

#   case $? in

#       1 | 2 )

#           fct "${FUNCNAME[0]}" 'returning 1'

#           return 1

#   esac

#

# Example 2

#    msg I "Checking the $user@$hostname ssh connection"

#    cmd=(ssh "$user@$hostname" 'echo OK')

#    run_cmd_with_timeout -t 5

#    case $? in

#        0 )

#            msg I 'ssh connection OK'

#            ;;

#        1 | 2 )

#            msg W 'ssh connection check failed'

#            fct "${FUNCNAME[0]}" 'returning 1'

#            return 1

#    esac

#

# Global variables read:

#   cmd

#   false

#   msg_lf

#   out_fn

#   rc_fn

#   true

# Global variables set: none

# Output:

#   * stdout and stderr to log or screen, either directly or via msg function

# Returns: described above

#--------------------------

function run_cmd_with_timeout {

    fct "${FUNCNAME[0]}" "started with arguments $*"

    local OPTIND    # Required when getopts is called in a function

    local args opt

    local opt_e_flag opt_E_flag opt_o_flag opt_O_flag opt_w_flag opt_W_flag

    local opt_e_rc_list opt_o_rc_list opt_t_arg opt_w_rc_list

    local opt_E_regex opt_O_regex opt_T_arg opt_W_regex

    local cmd_timeout cmd_timeout_msg_class ssh_host

    local connection_timeout connection_timeout_msg_class

    local -r duration_OK_regex='^\+?[[:digit:]]*\.?[[:digit:]]+(|d|h|m|s)$'

    local -r timeout_msg_class_OK_regex='^(|E|I|W)$'

    local array buf emsg msg msg_class my_rc oldIFS out rc rc_for_msg timed_out_flag

    # Parse options

    # ~~~~~~~~~~~~~

    args=("$@")

    emsg=

    emsg_warn_regex=

    opt_e_flag=$false

    opt_E_flag=$false

    opt_e_rc_list=

    opt_E_regex=

    opt_o_flag=$false

    opt_O_flag=$false

    opt_o_rc_list=

    opt_O_regex=

    opt_t_arg=,

    opt_T_arg=,

    opt_w_flag=$false

    opt_W_flag=$false

    opt_w_rc_list=

    opt_W_regex=

    ssh_host=

    while getopts :e:E:o:O:s:t:T:w:W: opt "$@"

    do

        case $opt in

            e )

                opt_e_flag=$true

                opt_e_rc_list=$OPTARG

                ;;

            E )

                opt_E_flag=$true

                opt_E_regex=$OPTARG

                ;;

            o )

                opt_o_flag=$true

                opt_o_rc_list=$OPTARG

                ;;

            O )

                opt_O_flag=$true

                opt_O_regex=$OPTARG

                ;;

            s )

                ssh_host=$OPTARG

                ;;

            t )

                opt_t_arg=$OPTARG

                ;;

            T )

                opt_T_arg=$OPTARG

                ;;

            w )

                opt_w_flag=$true

                opt_w_rc_list=$OPTARG

                ;;

            W )

                opt_W_flag=$true

                opt_W_regex=$OPTARG

                ;;

            : )

                emsg+=$msg_lf"Option $OPTARG must have an argument"

                ;;

            * )

                emsg+=$msg_lf"Invalid option '-$OPTARG'"

        esac

    done

    shift $(($OPTIND-1))

    # Test for mutually exclusive options

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    # There are no mutually exclusive options

    # Test for mandatory options not set

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    # There are no mandatory options

    # Parse option arguments

    # ~~~~~~~~~~~~~~~~~~~~~~

    oldIFS=$IFS

    IFS=,

    array=($opt_t_arg)

    if ((${#array[*]}<3)); then

        if [[ ${array[0]:=10} =~ $duration_OK_regex ]]; then

            cmd_timeout=${array[0]}

        else

            emsg+=$msg_lf"-t option arg ${array[0]} invalid"

            emsg+=" (does not match $duration_OK_regex)"

        fi

        if [[ ${array[1]:=10} =~ $duration_OK_regex ]]; then

            connection_timeout=${array[1]}

        else

            emsg+=$msg_lf"-t option arg ${array[1]} invalid"

            emsg+=" (does not match $duration_OK_regex)"

        fi

    else

        emsg+=$msg_lf"-t option: more than two comma-separated values"

        emsg+=" ($opt_t_arg)"

    fi

    array=($opt_T_arg)

    if ((${#array[*]}<3)); then

        if [[ ${array[0]} =~ $timeout_msg_class_OK_regex ]]; then

            cmd_timeout_msg_class=${array[0]}

        else

            emsg+=$msg_lf"-T option arg ${array[0]} invalid"

            emsg+=" (does not match $timeout_msg_class_OK_regex)"

        fi

        [[ ${array[1]:-} = '' ]] && array[1]=    # Default

        if [[ ${array[1]} =~ $timeout_msg_class_OK_regex ]]; then

            connection_timeout_msg_class=${array[1]}

        else

            emsg+=$msg_lf"-t option arg ${array[1]} invalid"

            emsg+=" (does not match $timeout_msg_class_OK_regex)"

        fi

    else

        emsg+=$msg_lf"-T option: more than two comma-separated values"

        emsg+=" ($opt_T_arg)"

    fi

    IFS=$oldIFS

    # Error trap globals

    # ~~~~~~~~~~~~~~~~~~

    [[ ${cmd:-} = '' ]] && emsg+=$msg_lf'$cmd is required but is unset or empty'

    if [[ ${out_fn:-} != '' ]]; then

        touch "$out_fn" 2>/dev/null    # In case does not exist

        buf=$(ck_file "$out_fn" f:rw 2>&1)

        [[ $buf != '' ]] && emsg+=$msg_lf$buf

    else

        emsg+=$msg_lf'$out_fn is required but is unset or empty'

    fi

    if [[ ${rc_fn:-} != '' ]]; then

        touch "$rc_fn" 2>/dev/null    # In case does not exist

        buf=$(ck_file "$rc_fn" f:rw 2>&1)

        [[ $buf != '' ]] && emsg+=$msg_lf$buf

    else

        emsg+=$msg_lf'$rc_fn is required but is unset or empty'

    fi

    # Report any invocation errors

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    if [[ $emsg != '' ]]; then

        msg E "Programming error: ${FUNCNAME[0]} (args ${args[*]})$emsg"

        return 1    # Required when being run from finalise function

    fi

    msg D "opt_o_rc_list: $opt_o_rc_list, opt_w_rc_list: $opt_w_rc_list, opt_e_rc_list: $opt_e_rc_list"

    msg D "opt_O_regex: $opt_O_regex, opt_W_regex: $opt_W_regex, opt_E_regex: $opt_E_regex"

    # Check ssh connection if requested

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    if [[ $ssh_host != '' ]]; then

        msg="Testing the $ssh_host ssh connection"

        msg I "$msg with timeout $connection_timeout"

        echo -n > "$out_fn"; echo -n > "$rc_fn"    # Ensure empty

        msg_class=

        timeout --signal=SIGTERM --kill-after=$connection_timeout \

            $connection_timeout ssh "$ssh_host" echo -n OK > "$out_fn" 2>&1

        rc=$?

        if ((rc==124||rc==137)); then

            [[ $connection_timeout_msg_class != '' ]] \

                && msg $connection_timeout_msg_class \

                   'Timed out'

            fct "${FUNCNAME[0]}" 'returning 2'

            return 2

        fi

        msg I OK

    fi

    # Run the command with timeout

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    msg I "Running command with timeout $cmd_timeout: ${cmd[*]}"

    echo -n > "$out_fn"; echo -n > "$rc_fn"    # Ensure empty

    timeout --signal=SIGTERM --kill-after=$cmd_timeout \

        $cmd_timeout "${cmd[@]}" > "$out_fn" 2>&1

    rc=$?

    if ((rc==124||rc==137)); then

        timed_out_flag=$true

        msg I 'Timed out'

        rc_for_msg='not available (timed out)'

    else

        timed_out_flag=$false

        echo $rc > "$rc_fn"

        rc_for_msg=$rc

    fi

    msg_class=

    # Examine any return code if requested

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    # The return code is not available after timeout

    # -o -w and -e usage notes:

    #   * When none are specified, return code >0 generates an error message

    #   * When one or more are specified:

    #     If e is specified and its rc list matches, generates an error message

    #     Else if w is specified and its rc list matches, generates a warning message

    #     Else if o is specified and o's rc list does not match:

    #         If e is specified, generates a warning message

    #         Else generates an error message

    if [[ ! $timed_out_flag ]]; then

        msg D "${FUNCNAME[0]}: Examining return code, $rc"

        if [[ $opt_e_rc_list != '' ]]; then

            msg D "${FUNCNAME[0]}: -e specified"

            run_rc_tests $opt_e_rc_list && msg_class=E

        elif [[ $opt_w_rc_list != '' ]]; then

            msg D "${FUNCNAME[0]}: -w specified"

            run_rc_tests $opt_w_rc_list && msg_class=W

        elif [[ $opt_o_rc_list != '' ]] && ! run_rc_tests $opt_o_rc_list; then

            msg D "${FUNCNAME[0]}: -o specified and not matched"

            [[ $opt_e_rc_list = '' ]] && msg_class=W || msg_class=E

        else

            msg D "${FUNCNAME[0]}: None of -o -w or -e specified"

            (($rc>0)) && msg_class=E

        fi

        msg D "${FUNCNAME[0]}: After examining return code, msg_class: $msg_class"

    fi

    # Examine any output if requested

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    # -O -W and -E usage notes ("output" is combined stdout and stderr):

    #   * When none are specified, output is ignored

    #   * When one or more are specified:

    #     If E is specified and its regex matches output,

    #         generates an error message

    #     Else if W is specified and its regex matches output,

    #         generates a warning message

    #     Else if O is specified and O's regex does not match output,

    #         If E is specified, generates a warning message

    #         Else generates an error message

    out=$(< "$out_fn")

    if [[ $out != '' ]]; then

        msg D "${FUNCNAME[0]}: Examining output"

        if [[ $opt_E_regex != '' && "$out" =~ $opt_E_regex ]]; then

            msg D "${FUNCNAME[0]}: -E specified and matched"

            msg_class=E

        elif [[ $opt_W_regex != '' && "$out" =~ $opt_W_regex ]]; then

            msg D "${FUNCNAME[0]}: -W specified and matched"

            [[ $msg_class != E ]] && msg_class=W

        elif [[ $opt_O_regex != '' ]]; then

            if [[ ! "$out" =~ $opt_O_regex ]]; then

                msg D "${FUNCNAME[0]}: -O specified ($opt_O_regex) and not matched"

                if [[ $opt_E_regex = '' ]]; then

                    msg D "${FUNCNAME[0]}: -E specified"

                    [[ $msg_class != E ]] && msg_class=W

                else

                    msg D "${FUNCNAME[0]}: -E not specified"

                    msg_class=E

                fi

            fi

        fi

        msg D "${FUNCNAME[0]}: After examining output, msg_class: $msg_class"

    fi

    # Generate message if required

    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    if [[ $msg_class != '' ]]; then

        msg="Command: ${cmd[*]}"

        msg+=$'\n'"rc: $rc_for_msg"

        msg+=$'\n'"Output: $out"

        msg $msg_class "$msg"    # Does not return when msg_class is E

    fi

    # Set my return code

    # ~~~~~~~~~~~~~~~~~~

    #   0 - No problem detected with the command

    #   1 - A problem other than timeout was detected

    #   2 - The command timed out

    if [[ $timed_out_flag ]]; then

        my_rc=2

    elif [[ $msg_class = W ]]; then

        my_rc=1

    else

        my_rc=0

    fi

    fct "${FUNCNAME[0]}" "returning $my_rc"

    return $my_rc

}  #  end of function run_cmd_with_timeout

#--------------------------

# Name: run_rc_tests

# Purpose:

#   * Runs return code tests (for run_cmd_with_timeout)

# Arguments:

#   $1 - comma separated list of arithmentical tests

#

# Global variables read: rc

# Global variables set: none

# Output:

#   * stdout and stderr to log or screen, either directly or via msg function

# Returns:

#   0 - $rc matched one of the tests

#   1 - $rc did not match any of the tests

#--------------------------

function run_rc_tests {

    local array i matched numcom oldIFS one_char rhs two_char

    local -r two_char_numcom_regex='^(<|>|=|!)=$'

    local -r one_char_numcom_regex='^(<|>)$'

    local -r uint_regex='^[[:digit:]]+$'

    # Parse the argument

    # ~~~~~~~~~~~~~~~~~~

    oldIFS=$IFS

    IFS=,

    array=($1)

    IFS=$oldIFS

    # For each test

    # ~~~~~~~~~~~~~

    for ((i=0;i<${#array[*]};i++))

    do

       # Parse the test

       # ~~~~~~~~~~~~~~

       two_char=${array[i]:0:2}

       one_char=${array[i]:0:1}

       if [[ $two_char =~ $two_char_numcom_regex ]]; then

           numcom=$two_char

           rhs=${array[i]:2}

       elif [[ $one_char =~ $one_char_numcom_regex ]]; then

           numcom=$one_char

           rhs=${array[i]:1}

       else

           msg E "Programming error: invalid numerc comparison ${array[i]} in $1"

       fi

       [[ ! $rhs =~ $uint_regex ]] \

           &&  msg E "Programming error: invalid numerc comparison ${array[i]} in $1"

       msg D "rc: $rc, operator: $numcom, rhs: $rhs"

       # Test

       # ~~~~

       matched=$false

       case $numcom in

           '<=' ) ((rc<=rhs)) && matched=$true ;;

           '>=' ) ((rc>=rhs)) && matched=$true ;;

           '==' ) ((rc==rhs)) && matched=$true ;;

           '!=' ) ((rc!=rhs)) && matched=$true ;;

           '<'  ) ((rc<rhs))  && matched=$true ;;

           '>'  ) ((rc>rhs))  && matched=$true ;;

       esac

       if [[ $matched ]]; then

           msg D 'An rc comparison matched, returning 0'

           return 0

       fi

    done

    msg D 'No rc comparisons matched, returning 1'

    return 1

}  #  end of function run_rc_tests

</pre>