Root

AJ ONeal August 30, 2022

Posix Shell HTTP Requests (curl, wget alternative)

There is no standard Posix utility for making HTTP request, but:

  • BSD curl is available on MacOS
  • GNU wget is almost always the default on Linux
    (and GNU curl is usually available)
  • BSD curl.exe is the default on Windows

In light of that, here are two Posix functions that you can reliably use on just about any Posixish operating system:

  • Safe for command or sourced function use (variables are namespaced with prefixes)
  • Runs in “strict mode” (clean exit on error)
  • shellcheck-linted (Posix-compliant, no bashisms)

Plain http_get

  • Safe for batch mode (silences progress bar)
  • Safe for partial & resumable downloads (prefers wget when available)
  • Safe for sourceing (variables are in subshell, and namespaced)
#!/bin/sh
## Usage: http_get <url> [filepath]
## Copyright (C) 2022 AJ ONeal <aj@therootcompany.com>
## Permission to copy and modify is granted under the CC0-1.0 license
set -e
set -u

http_get() { (
    http_get_url="${1}"
    http_get_filepath="${2:-"$(basename "${http_get_url}")"}"
    if [ -z "${http_get_url}" ]; then
        echo >&2 "Usage: http_get <url> [filepath]"
        return 1
    fi

    if command -v wget > /dev/null; then
        # wget supports resumable downloads
        if ! wget -q -c "${http_get_url}" -O "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    elif command -v curl > /dev/null; then
        if ! curl -fsSL "${http_get_url}" -o "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    fi

    # move downloaded file into file location
    mv "${http_get_filepath}.part" "${http_get_filepath}"
); }

http_get "${1:-}" "${2:-}"

Enhanced http_get

  • Shows progress in interactive shells
#!/bin/sh
## Usage: http_get <url> [filepath]
## Copyright (C) 2022 AJ ONeal <aj@therootcompany.com>
## Permission to copy and modify is granted under the CC0-1.0 license
set -e
set -u

http_get() { (
    http_get_url="${1}"
    http_get_filepath="${2:-"$(basename "${http_get_url}")"}"
    if [ -z "${http_get_url}" ]; then
        echo >&2 "Usage: http_get <url> [filepath]"
        return 1
    fi

    http_get__is_interactive_shell() {
        # $- shows shell flags (error,unset,interactive,etc)
        case $- in
            *i*)
                # true
                return 0
                ;;
            *)
                # false
                return 1
                ;;
        esac
    }

    if command -v wget > /dev/null; then
        http_get_show_progress=""
        if http_get__is_interactive_shell; then
            http_get_show_progress="--show-progress"
        fi

        # wget supports resumable downloads
        if ! wget -q $http_get_show_progress -c "${http_get_url}" -O "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    elif command -v curl > /dev/null; then
        # Silencing by default since even curl's progress bar has an interactive-only percent counter
        # ¯\_(ツ)_/¯
        http_get_show_progress="-s"
        if http_get__is_interactive_shell; then
            # for progress bar (rather than full stats)
            http_get_show_progress="-#"
        fi

        # shellcheck disable=SC2086
        # we want the flags to be split
        if ! curl $http_get_show_progress -fSL "${http_get_url}" -o "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    fi

    # move downloaded file into file location
    mv "${http_get_filepath}.part" "${http_get_filepath}"
); }

http_get "${1:-}" "${2:-}"

License

Public Domain via the Creative Commons Zero license
(attribution appreciated) 

http_get - A Posix-compliant function for HTTP Requests

Written in 2022 by AJ ONeal <aj@therootcompany.com>
To the extent possible under law, the author(s) have dedicated all copyright
and related and neighboring rights to this software to the public domain
worldwide. This software is distributed without any warranty.

You should have received a copy of the CC0 Public Domain Dedication along with
this software. If not, see <https://creativecommons.org/publicdomain/zero/1.0/>.
Categories: