Project

General

Profile

Task #11861

Task #11784: 22 Oct - bung: submit to Debian

bung: make Debian compliant

Added by Charles Atkinson 9 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Start date:
11/03/2021
Due date:
% Done:

0%

Close

Description

  • Move file installations from /opt/bung-<version> and symlinks into it to
    • /etc/cron.weekly/bung
    • /etc/logcheck/ignore.d.server/bung
    • /usr/bin for shell scripts (and remove .sh extensions)
    • /usr/lib/bung for library (*.fun and *.scrippet) files
    • /usr/share/doc/bung for the doc tree
    • /usr/share/man/man5 and man8 for the man pages
  • Incidentally remove the obsolete DG-GS1526E_bu.sh (if anyone is using it, functionality should be replaced by a templated_bu template)
  • Restructure the git tree to suit
  • Directories created during installation
    • /etc/bung
    • /var/log/bung
  • Change tarball creation procedure
    • Use gzip option -9 when creating compressed man pages
    • Introduce root directory bung-<version>
    • Change from .tgz to .tar.gz
  • Update user and programmer docs to suit

History

#1 Updated by Charles Atkinson 9 months ago

  • Description updated (diff)

#2 Updated by Charles Atkinson 9 months ago

  • Description updated (diff)

#3 Updated by Charles Atkinson 9 months ago

  • Status changed from New to In Progress

Plan

  • Create re-organised tree
  • Change version number
  • Update user and programmer docs to suit
  • Create Debian-compliant tarball
  • Test by using tarball in source build
  • Further changes depending on test result, especially lintian output
  • Create and populate new git repo for bung3

Create new tree

Created dir bung3/root then the directory tree in it

c@cw11:~/.../Blue Light/bung/bung3/root$ mkdir -p etc/cron.weekly etc/logcheck/ignore.d.server usr/bin usr/lib/bung usr/share/doc/bung usr/share/man/man{5,8}
c@cw11:~/d/Auroville/Blue Light/bung$ cp -p git/opt/share/cron/cron.weekly root/bung3/etc/cron.weekly/bung
c@cw11:~/d/Auroville/Blue Light/bung$ cp -p git/opt/share/logcheck/bung root/bung3/etc/logcheck/ignore.d.server/bung
c@cw11:~/d/Auroville/Blue Light/bung$ for f in git/opt/*.sh
do
    dest=${f##*/}
    dest=${dest%.sh}
    cp -p $f root/bung3/usr/bin/$dest
done
c@cw11:~/d/Auroville/Blue Light/bung$ rm root/bung3/usr/bin/DG-GS1526E_bu
c@cw11:~/d/Auroville/Blue Light/bung$ for f in git/opt/*.sh
do
    dest=${f##*/}
    dest=${dest%.sh}
    cp -p $f root/bung3/usr/bin/$dest
done
c@cw11:~/d/Auroville/Blue Light/bung$ cp -p git/opt/lib/*.{fun,scrippet} root/bung3/usr/lib/bung
c@cw11:~/d/Auroville/Blue Light/bung$ cp -pr git/opt/share/doc/* bung3/usr/share/doc/bung
c@cw11:~/d/Auroville/Blue Light/bung$ rm root/bung3/usr/share/doc/bung/*.{org,bu[0-9]}
c@cw11:~/d/Auroville/Blue Light/bung$ rm root/bung3/usr/share/doc/bung/examples/rsync.exclude/*.{org,bu[0-9]}
c@cw11:~/d/Auroville/Blue Light/bung$ cp -p git/opt/share/man/man5/*5 root/bung3/usr/share/man/man5/
c@cw11:~/d/Auroville/Blue Light/bung$ cp -p git/opt/share/man/man8/*8 root/bung3/usr/share/man/man8/
c@cw11:~/d/Auroville/Blue Light/bung$ rm root/bung3/usr/bin/DG-GS1526E_bu
c@cw11:~/d/Auroville/Blue Light/bung$ rm root/usr/share/doc/bung/examples/script.conf/DG-GS1526E_bu.sample.conf

Change version number

c@cw11:~/.../Blue Light/bung/bung3$ echo script_ver=3.0.0 > usr/lib/bung/version.scrippet

Update user and programmer docs to suit (initialise)

Created nextcloud/bung/3.0 containing "Backup scripts next generation 3.0.0 Programmer's Guide.odt" and "Backup scripts next generation 3.0.0 User Guide.odt"

Create Debian-compliant tarball

Edited "Backup scripts next generation 3.0.0 Programmer's Guide.odt" while changing and testing the procedure
  • Updated "6.3.3 Update the man pages and create compressed versions"
  • Added "6.5 Clean the git tree"
  • Updated "6.6 Create the tarball"

Tarball creation command

tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX) \
    && version=$(<usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a ./ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' \) -delete \
   && tar cvz -f ~/bung-$version.tar.gz .

Test by using tarball in source build

Copied the previous source build tree and updated

c@cw11:~/.../Blue Light/bung/bung3 source build$ tree
.
├── bung-3.0.0
│   └── debian
│       ├── bung.substvars
│       ├── changelog
│       ├── compat
│       ├── control
│       ├── copyright
│       ├── debhelper-build-stamp
│       ├── files
│       ├── install
│       ├── patches
│       │   └── series
│       ├── README.Debian
│       ├── rules
│       ├── source
│       │   ├── format
│       │   └── local-options
│       └── watch
└── bung-3.0.0.tar.gz
c@cw11:~/.../Blue Light/bung/bung3 source build$ tar -xzf bung-3.0.0.tar.gz
@cw11:~/.../Blue Light/bung/bung3 source build$ ln -s bung-3.0.0.tar.gz bung_3.0.0.orig.tar.gz

#4 Updated by Charles Atkinson 9 months ago

  • Status changed from In Progress to Closed

Test by using tarball in source build (continued)

After fixing debian/install

c@cw11:~/.../bung/bung3 source build/bung-3.0.0$ cat debian/install 
etc/* etc/
usr/* usr/

c@cw11:~/.../bung/bung3 source build/bung-3.0.0$ debuild
...
Now running lintian bung_3.0.0-1_amd64.changes ...
E: bung changes: changed-by-address-malformed <>
E: bung changes: changed-by-name-missing <>
E: bung changes: changed-by-address-missing <>
W: bung source: extra-whitespace-around-name-in-changelog-trailer
W: bung source: changelog-should-mention-nmu
W: bung source: source-nmu-has-incorrect-version-number 3.0.0-1
W: bung: wrong-bug-number-in-closes l3:#nnnn
W: bung: new-package-should-close-itp-bug
E: bung: changelog-is-dh_make-template
W: bung: binary-without-manpage usr/bin/check_hotplug_usage
[snip similar]
W: bung: script-not-executable etc/cron.weekly/bung
W: bung: script-not-executable usr/lib/bung/ck_file.fun
[snip similar]
Finished running lintian.

Fix etc/cron.weekly/bung not executable

c@cw11:~/.../bung/bung3/root$ chmod 755 etc/cron.weekly/bung
Updated tarball using command in #11861#note-3 above
c@cw11:~/.../Blue Light/bung/bung3 source build$ cp -p ~/bung-3.0.0.tar.gz bung-3.0.0.tar.gz
c@cw11:~/.../Blue Light/bung/bung3 source build$ tar -xzf bung-3.0.0.tar.gz
c@cw11:~/.../bung/bung3 source build/bung-3.0.0$ debuild
["etc/cron.weekly/bung not executable" not shown]

Test + debug complete

#5 Updated by Charles Atkinson 9 months ago

  • Status changed from Closed to In Progress

Re-opened because, as noted in #11784#note-19, bung man page names do not match bung "binary" (actually "executable") names because the man pages' have a "bung." prefix.

#6 Updated by Charles Atkinson 9 months ago

  • Status changed from In Progress to Closed
Would the man page names be unique when the "bung." prefix is removed? The bung prefixes were added to make the man page file names unique in the man pages namespace. Now the scripts are in the /usr/bin/* namespace, are they unique? Searched for each at https://packages.debian.org/search?searchon=contents&mode=path&suite=stable&arch=any&keywords=cp&lt;script name>
  • check_hotplug_usage
  • hotplug_bu
  • hotplug_bu_launcher
  • mysql_bu
  • openldap_bu
  • postgres_bu
  • remote_agent
  • rsync_bu
  • rsync_restore
  • super_bu
  • sysinfo_bu
  • templated_bu
  • validate_ssh_cmd

None were found by the search so all names were available.

Removed the man pages' file name prefix "bung."

Found the existing tarball build command did not need changing

Issue closed pending further changes required

#7 Updated by Charles Atkinson 9 months ago

  • Status changed from Closed to In Progress
Re-opened as noted in #11784#note-20 to
  • Create a man page for remote_agent
  • *.fun files: remove shebang line, add last line # vim: filetype=bash:

#8 Updated by Charles Atkinson 9 months ago

Library files: remove shebang line, add last line "# vim: filetype=bash::

Except for version.scrippet (no shebang or copyright). Decided to keep mtimes because the changes are trivial
for f in *.fun *.scrippet
do
    [[ $f = version.scrippet ]] && continue
    cp -p $f $f.bak
    sed -i '1,2d' $f
    echo '# vim: filetype=bash:' >> $f
    touch --reference=$f.bak $f
done
rm *.bak
  • err_trap_rsync_files_and_dirs.fun. No copyright + licence
  • initialise_1.scrippet. No copyright + licence
  • msg_on_screen.fun. No copyright + licence

Next

  • Fix the "No copyright + licence" files
  • Create a man page for remote_agent

#9 Updated by Charles Atkinson 9 months ago

Fix the "No copyright + licence" files

Done. Preserved mtime except for initialise_1.scrippet which was updated as below

Fix bin and etc directory envars

Updated initialise_1.scrippet and remote_agent

    export BUNG_BIN_DIR=${BUNG_BIN_DIR:-/usr/bin}
    export BUNG_ETC_DIR=${BUNG_ETC_DIR:-/etc/bung}

Create a man page for remote_agent

Done

Next

  • Environment variable BUNG_LIB_DIR: update
  • man page name common.5 and 8 Already used?
  • man pages content
    • Update paths
    • Remove "bung." prefix

#10 Updated by Charles Atkinson 9 months ago

Environment variable BUNG_LIB_DIR: update

Done. Details lost in system crash

man page names common.5 and 8 already used?

No but found several man pages <name>-common. Changed bung accordingly

man pages content

Update paths

Done. Details lost in system crash. Key command were (also for section 5)

sed -i 's|/etc/opt/bung|/etc/bung|g' *8
sed -i 's|/opt/bung/share|/usr/share/bung|g' *8
sed -i 's|/opt/bung/lib|/usr/lib/bung|g' *8
sed -i 's|/opt/bung/doc|/usr/share/doc/bung|g' *8
sed -i 's|/opt/bung|/usr/bin|g' *8

Remove "bung." prefix

Done. Details lost in system crash. Key command were (also for section 5)

sed -i 's/bung\.common/bung-common/g' *8
sed -i 's/bung\.\(.\)/\1/' *8
sed -i 's|Copyright (C) .* Charles|Copyright (C) 2021 Charles|' *8

Test

Tested by creating tarball (command in #11861#note-3 above) and using in source build

c@cw11:~/.../Blue Light/bung/bung3 source build$ cp -p ~/bung-3.0.0.tar.gz bung-3.0.0.tar.gz
c@cw11:~/.../Blue Light/bung/bung3 source build$ rm -r bung-3.0.0/{etc,usr}
c@cw11:~/.../Blue Light/bung/bung3 source build$ tar -xzf bung-3.0.0.tar.gz
c@cw11:~/.../bung/bung3 source build/bung-3.0.0$ debuild
...
Now running lintian bung_3.0.0-1_amd64.changes ...
E: bung changes: changed-by-address-malformed <>
E: bung changes: changed-by-name-missing <>
E: bung changes: changed-by-address-missing <>
W: bung source: extra-whitespace-around-name-in-changelog-trailer
W: bung source: changelog-should-mention-nmu
W: bung source: source-nmu-has-incorrect-version-number 3.0.0-1
W: bung: wrong-bug-number-in-closes l3:#nnnn
W: bung: new-package-should-close-itp-bug
E: bung: changelog-is-dh_make-template
Finished running lintian.
The errors are all expected so OK

Next

  • .odt files and their .pdf derivatives
    • Update
    • Add to cw11.iciti.av:/home/c/d/Auroville/Blue Light/bung/bung3 > tarball > source build
  • Install resulting .deb and test by usage
  • Create git repo for bung3
  • Put bung3 and its source build in git

#11 Updated by Charles Atkinson 9 months ago

.odt files

They are "/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/Backup scripts next generation 3.0.0 Programmer's Guide.odt" and "Backup scripts next generation 3.0.0 User Guide.odt"

Similar to the man pages update, the required changes (with additions found interactively) in order are
  • /etc/opt/bung > /etc/bung
  • /opt/bung/share > /usr/share/bung
  • /opt/bung/lib > /usr/lib/bung
  • /opt/bung/doc > /usr/share/doc/bung
  • /opt/bung/sample > /usr/share/doc/bung/examples
  • /opt/bung > /usr/bin
  • Remove bung. prefix from man page names
  • Other changes identified by inspection
    • Remove DG-GS1526E references
For the Programmer's Guide only, update
  • Procedure for installing from tarball
  • *.sh: remove .sh suffix
  • "Multiple versions on the same system"
  • "Create the .deb"
  • "Searching the bash files"

Next

  • Continue updating .odt files
  • Explore creating .html files from the .odt files (Debian policy to use HTML docs?)
  • What is Debian policy re .pdf docs?
  • Add .odt and derivative files to cw11.iciti.av:/home/c/d/Auroville/Blue Light/bung/bung3 > tarball > source build
  • Install resulting .deb and test by usage
  • Create git repo for bung3
  • Put bung3 and its source build in git

#12 Updated by Charles Atkinson 9 months ago

.odt files (continued)

Completed updating the User Guide and Programmer's Guide. New procedures copied from above. Docs need testing by usage

HTML and .pdf

From https://www.debian.org/doc/debian-policy/ch-docs.html#preferred-documentation-formats

12.4. Preferred documentation formats
The unification of Debian documentation is being carried out via HTML.
If the package comes with extensive documentation in a markup format that can be converted to various other formats you should if possible ship HTML versions in a binary package. ... The documentation must be installed as specified in Additional documentation.
Other formats such as PostScript may be provided at the package maintainer’s discretion.

Tried saving the .odt files as HTML. The original indentations and vertical space below lines were lost but when opened with Firefox and with Opera the HTML was usable

Added a note to the .odt files "The .odt version is better formatted than the .html and .pdf derivatives"

In the Programmer's Guide, in the Release procedure, added creation of .html files

Add .odt and derivative files to tarball

tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX) \
    && version=$(<usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a ./ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung-$version.tar.gz .

Test

Test + debug the Programmer's Guide "Create the .deb" section

c@cw11:~/.../bung/bung3/source build$ version=$(sed 's/script_ver=//' ../root/usr/lib/bung/version.scrippet)
c@cw11:~/.../bung/bung3/source build$ cp -p ~/bung-$version.tar.gz .
c@cw11:~/.../bung/bung3/source build$ rm -r bung-$version/{etc,usr}
c@cw11:~/.../bung/bung3/source build$ tar -xzf bung-$version.tar.gz
c@cw11:~/.../bung/bung3/source build$ ln -s bung-$version.tar.gz bung_$version.orig.tar.gz
ln: failed to create symbolic link 'bung_3.0.0.orig.tar.gz': File exists
c@cw11:~/.../bung/bung3/source build$ cd bung-$version && debuild
...
Test the resulting .deb
root@cw11.iciti.av:~# dpkg -i '/home/c/d/Auroville/Blue Light/bung/bung3/source build/bung_3.0.0-1_all.deb'
(Reading database ... 228382 files and directories currently installed.)
Preparing to unpack .../bung_3.0.0-1_all.deb ...
Unpacking bung (3.0.0-1) over (2.1.0-99) ...
Setting up bung (3.0.0-1) ...
Processing triggers for man-db (2.8.5-2) ...
Prepare for wider testing
c@cw11:~/.../bung/bung3/source build$ cp -p bung_3.0.0-1_all.deb '/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0

Next

  • Install resulting .deb on further computers and test by usage
  • Why any amd64 files created by the source build? The architecture is "all"
    c@cw11:~/.../bung/bung3/source build$ ll *64*
    -rw-r--r-- 1 c c 1.8K Mar 19 08:26 bung_3.0.0-1_amd64.build
    -rw-r--r-- 1 c c 4.6K Mar 19 08:26 bung_3.0.0-1_amd64.buildinfo
    -rw-r--r-- 1 c c 1.7K Mar 19 08:26 bung_3.0.0-1_amd64.changes
    
  • Update guides and derivatives as required
  • Create git repo for bung3
  • Put bung3 (now including the source build tree) in git. Need to exclude several files from the source build tree? Better build in a temporary directory?

#13 Updated by Charles Atkinson 9 months ago

Bugs + fixes

Found many files with content still having .sh extensions and /opt/

grep --files-with-matches -r '\.sh' | grep -Ev '\.(org|bu[0-9])$' | grep -v 'NEWS$'
...
grep --files-with-matches -r '/opt/' | grep -Ev '\.(org|bu[0-9])' | grep -v 'NEWS'
Fixed those under cw11.iciti.av:"/home/c/d/Auroville/Blue Light/bung/bung3/root" and built again (as above create tarball > update source build > run debuild)

Streamlined updating the source build to

version=$(sed 's/script_ver=//' ../root/usr/lib/bung/version.scrippet)
orig=bung_$version.orig.tar.gz
tar=bung-$version.tar.gz
cp -p ~/bung-$version.tar.gz . \
    && rm -r bung-$version/{etc,usr} \
    && tar -xzf $tar \
    && [[ -e $orig ]] && true || ln -s $tar $orig \
    && cd bung-$version \
    && debuild
...
c@cw11:~/.../bung/bung3/source build$ cp -p bung_3.0.0-1_all.deb '/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/bung_3.0.0-1_all.deb'

Test

Had already created /etc/bung by copying /etc/opt/bung and updating

root@cw11.iciti.av:~# dpkg -i '/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/bung_3.0.0-1_all.deb'
(Reading database ... 228387 files and directories currently installed.)
Preparing to unpack .../bung/3.0/bung_3.0.0-1_all.deb ...
Unpacking bung (3.0.0-1) over (2.1.0-1) ...
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc/examples/hook scripts': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc/examples/script.conf': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc/examples/rsync.exclude': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc/examples/cron': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc/examples': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share/doc': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0/share': Directory not empty
dpkg: warning: unable to delete old directory '/opt/bung-2.1.0': Directory not empty
Setting up bung (3.0.0-1) ...
Processing triggers for man-db (2.8.5-2) ...
root@cw11.iciti.av:~# /usr/bin/sysinfo_bu -c /etc/opt/bung/sysinfo -l /dev/tty
[OK]
Plugged in a USB backup disk. The backup ran OK

Next

  • Install resulting .deb on further computers and test by usage
  • Why any amd64 files created by the source build? The architecture is "all"
    c@cw11:~/.../bung/bung3/source build$ ll *64*
    -rw-r--r-- 1 c c 1.8K Mar 19 08:26 bung_3.0.0-1_amd64.build
    -rw-r--r-- 1 c c 4.6K Mar 19 08:26 bung_3.0.0-1_amd64.buildinfo
    -rw-r--r-- 1 c c 1.7K Mar 19 08:26 bung_3.0.0-1_amd64.changes
    
  • Update guides and derivatives as required
  • Create git repo for bung3
  • Put bung3 (now including the source build tree) in git. Need to exclude several files from the source build tree? Better build in a temporary directory?

#14 Updated by Charles Atkinson 9 months ago

Bug + fix

Test by usage of bung 3.0.0 on cw11.iciti.av gave, from net1.iciti.av
15:03:43 rsync_bu.sh: ERROR: Command: scp -p /run/bung/rsync_bu.sh+bind_to_cw11.kdRTBt/bung.32578c61.conf root@dns-master_to_cw11:/tmp/bung.32578c61.conf
rc: 1
Output: bash: /opt/bung/validate_ssh_cmd.sh: No such file or directory
lost connection
Fix
root@cw11.iciti.av:/opt# rm bung && mkdir bung && ln -s /usr/bin/validate_ssh_cmd bung/validate_ssh_cmd.sh && ln -s /usr/bin/remote_agent bung/remote_agent.sh
Test
root@cw11.iciti.av:~# /etc/cron.daily/sync_from_dns_and_nas &
Bugs
  • On net1.iciti.av (.sh in log file name)
    21:45:29 rsync_bu.sh: ERROR: Command: ssh root@dns-master_to_cw11 /opt/bung/remote_agent.sh -c /tmp/bung.43baee2c.conf -D
    rc: 1
    Output: validate_ssh_cmd.sh: Command did not pass validation (/opt/bung/remote_agent.sh -c /tmp/bung.43baee2c.conf -D)
    
  • On cw11.iciti.av
    root@cw11.iciti.av:~# lrt /var/log/bung | tail -1
    -rw-r--r-- 1 root root  931 Mar 21 21:42 validate_ssh_cmd.sh-2021-03-21.log
    

Next

  • Fix .sh in logfile name
  • Implement Task #11880 "validate_ssh_cmd: update to use conffile" and provide a conffile for both 2.x.x and 3.x.x clients
  • Continue with the "Next" list from the previous note

#15 Updated by Charles Atkinson 9 months ago

Fix .sh in logfile name

Found the .sh in logfile name is correct because /root/.ssh/authorized_keys has command="/opt/bung/validate_ssh_cmd.sh". Changed to command="/usr/bin/validate_ssh_cmd"

Test

root@cw11.iciti.av:~# /etc/cron.daily/sync_from_dns_and_nas &
OK
root@cw11.iciti.av:/var/log/bung# cd
root@cw11.iciti.av:~# lrt /var/log/bung | tail -1
-rw-r--r-- 1 root root 1.5K Mar 22 08:40 validate_ssh_cmd-2021-03-22.log

#16 Updated by Charles Atkinson 9 months ago

Migrate from cw11.iciti.av local storage to Aurinoco Nextcloud

To make more widely available and backed up pending putting in git

Migration itself

c@cw11:~$ rsync -a '/home/c/d/Auroville/Blue Light/bung/bung3/' /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git

Task #11880 "validate_ssh_cmd: update to use conffile"

Tested on cw11.iciti.av. Updated the tree

c@cw11:~$ bu /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git/root/usr/bin/validate_ssh_cmd
c@cw11:~$ cp -p /usr/bin/validate_ssh_cmd /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git/root/usr/bin
c@cw11:~/.../pre-git/root/etc$ mkdir bung
c@cw11:~/.../pre-git/root/etc$ cp -p /etc/bung/validate_ssh_cmd.conf bung

Task #11911 etc/logcheck/ignore.d.server/bung: make specific to bung scripts

Fixed in the pre-git tree

Bug #11897 rsync_bu: conffile without rsync keyword: "src_dir: unbound variable"

Fixed in the pre-git tree

Evolve build procedure to suit git

Many files in the build tree (was cw11.iciti.av:"/home/c/d/Auroville/Blue Light/bung/bung3/source build" is now cw11.iciti.av:"/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git/source build") should not be in git
  • All files in the "source build" directory
    • The tarball, bung_<version>.orig.tar.gz
    • The remaining files, created during the build
  • Files created by unpacking the tarball into "source build/bung-<version>", specifically the subdir etc and usr trees

So the only files which should be in git are the "source build/bung-<version>/debian" tree

So the (pre-)git tree has two subdirs
  • source/ for the bung source files
  • "source build/bung-<version>/debian" for the build files
So, extending the tarball creation procedure from #11861#note-12 above, the full build procedure could be
  • cd to the git root directory (currently "/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git")
  • Create a temporary directory with sub-directory bung-$version
  • Generate tarball "source build/bung_<version>.orig.tar.gz" from
    • Files under source. Uncompressed man pages to be compressed
    • .html, .odt and .pdf files
  • Unpack the tarball. That creates $tmpdir/bung-<version>
  • Copy the "source build/bung-<version>/debian" tree to $tmpdir/bung-<version>/debian
  • cd to $tmpdir/bung-<version>
  • Run debuild
  • Copy the .deb and the three "source package" files to long term storage

tmpdir tree before running debuild

├── bung-<version>
│   ├── debian
│   │   ├── bung.substvars
│   │   ├── changelog
│   │   ├── compat
│   │   ├── control
│   │   ├── copyright
│   │   ├── debhelper-build-stamp
│   │   ├── files
│   │   ├── install
│   │   ├── manpages.disabled
│   │   ├── patches
│   │   │   └── series
│   │   ├── README.Debian
│   │   ├── rules
│   │   ├── source
│   │   │   ├── format
│   │   │   └── local-options
│   │   └── watch
│   ├── etc/
│   └── usr/
├── bung_3.0.0.orig.tar.gz

docdir='/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/'
gitdir=/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git
tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX)
cd "$gitdir" \
    && version=$(<source/usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a source/ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p $docdir/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung_$version.orig.tar.gz . \
   && mv ~/bung_$version.orig.tar.gz . \
   && cd $tmpdir/bung-$version \
   && rsync -a "$gitdir/source build/bung-$version/debian/" debian \
   && debuild \
   && cd .. \
   && cp -p *.{deb,debian.tar.xz,dsc,orig.tar.gz} ~

Test

Ran the above command. As wanted except "W: bung: manpage-has-bad-whatis-entry usr/share/man/man5/check_hotplug_usage.5.gz"

Debug section 5 check_hotplug_usage man page

manpage-has-bad-whatis-entry not listed at https://lintian.debian.org/tags/ but bad-whatis-entry was. From https://lintian.debian.org/tags/bad-whatis-entry.html

A manual page should start with a NAME section, which lists the program name and a brief description. The NAME section is used to generate a database that can be queried by commands like apropos and whatis. You are seeing this tag because lexgrog was unable to parse the NAME section.

c@cw11:~/.../share/man/man5$ diff check_hotplug_usage.5{.bu1,}
23c23
< check_hotplug_usage
---
> check_hotplug_usage - configuration for bung's check_hotplug_usage script

Tested by repeating the build. OK

Distribute

c@cw11:~$ cp -p bung_3.0.0* /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/

Test resulting .deb on cw11.iciti.av

root@cw11.iciti.av:~# apt remove bung
...
root@cw11.iciti.av:~# dpkg -i ~c/bung_3.0.0-1_all.deb
...

Plugged in a USB backup disk. The log and the notification mail looked OK

Pilot on cw10.iciti.av

For test by usage and to evolve the migration procedure

Pre-install conf

root@CW10:~# rsync -a /etc/opt/bung/ /etc/bung
root@CW10:/etc/bung# mv *.org *.bu[0-9] Archive/
root@CW10:/etc/bung# grep -E '\.sh|/opt' * 2>/dev/null | grep -v /usr/local/bin/ | grep -v ':#'
hotplug:Subsidiary script = sysinfo_bu.sh sysinfo nice=19
hotplug:Subsidiary script = rsync_bu.sh whole_tree nice=19
hotplug:Subsidiary script = rsync_bu.sh vbox_vms nice=19 schedule=".*/.*/.*/(2|5)/.*/.*/." 
hotplug:Subsidiary script = rsync_bu.sh cs6_to_cw10 nice=19
hotplug:Subsidiary script = rsync_bu.sh cs7_to_cw10 nice=19
root@CW10:/etc/bung# sed -i 's/_bu\.sh/_bu/' hotplug
root@CW10:~# f=/etc/udev/rules.d/99-bung.rules; bu $f && sed -i -e 's/\.sh / /g' -e 's|/opt/bung|/usr/bin|g' $f && udevadm control --reload

Install

root@CW10:~# apt remove bung
...
Removing bung (2.1.0-1) ...
root@CW10:~# dpkg -i '/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/bung_3.0.0-1_all.deb' 
Selecting previously unselected package bung.
(Reading database ... 216536 files and directories currently installed.)
Preparing to unpack .../bung/3.0/bung_3.0.0-1_all.deb ...
Unpacking bung (3.0.0-1) ...
Setting up bung (3.0.0-1) ...
Processing triggers for man-db (2.8.5-2) ...

Post-install conf

Required for remote clients running pre-3.0.0 bung

root@CW10:~# mkdir -p /opt/bung && ln -s /usr/bin/validate_ssh_cmd /opt/bung/validate_ssh_cmd.sh && ln -s /usr/bin/remote_agent /opt/bung/remote_agent.sh
Required to get unsurprising log file name
fn=~/.ssh/authorized_keys2; bu "$fn" && sed -i 's|/opt/bung/validate_ssh_cmd.sh|/usr/bin/validate_ssh_cmd|' ~/.ssh/authorized_keys2

Test

Plugged a USB backup disk. The log and the notification mail looked OK

#17 Updated by Charles Atkinson 9 months ago

Bugfix and create version 3.0.1

As logged in #11922#note-7, /etc/logcheck/ignore.d.server was defective. Debugged on backup5.iciti.av. Updated "nextcloud/bung/3.0/pre-git/source/etc/logcheck/ignore.d.server/bung". Bumped version to 3.0.1 ...

c@CW10:~/.../3.0/pre-git/source$ cat usr/lib/bung/version.scrippet
script_ver=3.0.1
c@CW10:~/.../3.0/pre-git/source build$ mkdir -p bung-3.0.1 && rsync -a bung-3.0.0/debian/ bung-3.0.1/debian
c@CW10:~/.../3.0/pre-git/source build$ rm -r bung-3.0.1/debian/bung
c@CW10:~/.../source build/bung-3.0.1/debian$ sed --in-place 's/3\.0\.0/3.0.1/g' $(grep --files-with-matches '3\.0\.0' * 2>/dev/null)
... and re-built
docdir='/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/'
gitdir=/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git
tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX)
cd "$gitdir" \
    && version=$(<source/usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a source/ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p $docdir/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung_$version.orig.tar.gz . \
   && mv ~/bung_$version.orig.tar.gz . \
   && cd $tmpdir/bung-$version \
   && rsync -a "$gitdir/source build/bung-$version/debian/" debian \
   && debuild \
   && cd .. \
   && cp -p *.{deb,debian.tar.xz,dsc,orig.tar.gz} ~
...
WARNING generated by debuild:
Making debian/rules executable!
c@CW10:~$ cp -p bung_3.0.1* /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/

#18 Updated by Charles Atkinson 9 months ago

Bugfix and create version 3.0.2

As logged in #11922#note-13, /etc/logcheck/ignore.d.server was defective. Debugged on redmine2.iciti.av. Updated "nextcloud/bung/3.0/pre-git/source/etc/logcheck/ignore.d.server/bung"

c@CW10:~$ scp -p root@redmine2.iciti.av:/etc/logcheck/ignore.d.server/bung "/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git/source/etc/logcheck/ignore.d.server/bung" 
Bumped version to 3.0.2 ...
c@CW10:~/.../3.0/pre-git/source$ cat usr/lib/bung/version.scrippet
script_ver=3.0.2
c@CW10:~/.../3.0/pre-git/source build$ mkdir -p bung-3.0.2 && rsync -a bung-3.0.1/debian/ bung-3.0.2/debian
c@CW10:~/.../source build/bung-3.0.2/debian$ sed --in-place 's/3\.0\.1/3.0.2/g' $(grep --files-with-matches '3\.0\.1' * 2>/dev/null)
... and re-built
docdir='/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/'
gitdir=/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git
tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX)
cd "$gitdir" \
    && version=$(<source/usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a source/ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p $docdir/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung_$version.orig.tar.gz . \
   && mv ~/bung_$version.orig.tar.gz . \
   && cd $tmpdir/bung-$version \
   && rsync -a "$gitdir/source build/bung-$version/debian/" debian \
   && debuild \
   && cd .. \
   && cp -p *.{deb,debian.tar.xz,dsc,orig.tar.gz} ~
...
WARNING generated by debuild:
Making debian/rules executable!
c@CW10:~$ cp -p bung_3.0.2* /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/

#19 Updated by Charles Atkinson 9 months ago

Create version 3.0.3

For updated docs and pedantic lintian (logged in #11784#note-21)

Bumped version to 3.0.3

c@CW10:~/.../3.0/pre-git/source$ cat usr/lib/bung/version.scrippet
script_ver=3.0.3
c@CW10:~/.../3.0/pre-git/source build$ mkdir -p bung-3.0.3 && rsync -a bung-3.0.2/debian/ bung-3.0.3/debian
c@CW10:~/.../source build/bung-3.0.3/debian$ sed --in-place 's/3\.0\.2/3.0.3/g' $(grep --files-with-matches '3\.0\.2' * 2>/dev/null)
Fixed perms bug (should not need doing again)
c@CW10:~/.../source build/bung-3.0.3/debian$ chmod 755 rules
Built
docdir='/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/'
gitdir=/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/pre-git
tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX)
cd "$gitdir" \
    && version=$(<source/usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a source/ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p $docdir/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung_$version.orig.tar.gz . \
   && mv ~/bung_$version.orig.tar.gz . \
   && cd $tmpdir/bung-$version \
   && rsync -a "$gitdir/source build/bung-$version/debian/" debian \
   && debuild \
   && cd .. \
   && cp -p *.{deb,debian.tar.xz,dsc,orig.tar.gz} ~
...
Now running lintian bung_3.0.3-1_amd64.changes ...
N: Using profile debian/main.
N: Starting on group bung/3.0.3-1
N: Unpacking packages in group bung/3.0.3-1
N: ----
N: Processing changes file bung (version 3.0.3-1, arch source all) ...
E: bung changes: changed-by-address-malformed <>
E: bung changes: changed-by-name-missing <>
E: bung changes: changed-by-address-missing <>
N: ----
N: Processing source package bung (version 3.0.3-1, arch source) ...
P: bung source: file-contains-trailing-whitespace debian/rules (line 10)
W: bung source: extra-whitespace-around-name-in-changelog-trailer
W: bung source: changelog-should-mention-nmu
W: bung source: source-nmu-has-incorrect-version-number 3.0.3-1
N: ----
N: Processing buildinfo package bung (version 3.0.3-1, arch all source) ...
N: ----
N: Processing binary package bung (version 3.0.3-1, arch all) ...
W: bung: wrong-bug-number-in-closes l3:#nnnn
W: bung: new-package-should-close-itp-bug
E: bung: changelog-is-dh_make-template
N: Finished processing group bung/3.0.3-1
Finished running lintian.
Fixed debian/rules line 10 "trailing whitespace" (removed empty last line) and re-built. OK

Copied built files to nextcloud storage

c@CW10:~$ cp -p bung_3.0.3* /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/

#20 Updated by Charles Atkinson 9 months ago

  • Status changed from In Progress to Closed

Issue closed because bung is now Debian compliant pending Debian saying it is not

#21 Updated by Charles Atkinson 8 months ago

Create version 3.0.4

For at least Bug #11940 ".deb postinst and postrm scripts: create and remove /var/log/bung" and Feature #11952 "sysinfo_bu: remove BLLIS conf logging'

Bumped version to 3.0.4

c@CW10:~$ cd '/home/c/d/Auroville/Blue Light/bung/bung3/source'
master c@CW10:~/.../bung/bung3/source$ cat usr/lib/bung/version.scrippet
script_ver=3.0.4
master c@CW10:~/.../bung/bung3/source build$ mkdir -p bung-3.0.4 && rsync -a bung-3.0.3/debian/ bung-3.0.4/debian
master c@CW10:~/.../bung/bung3/source build$ cd bung-3.0.4/debian/
master c@CW10:~/.../source build/bung-3.0.4/debian$ sed --in-place 's/3\.0\.3/3.0.4/g' $(grep --files-with-matches '3\.0\.3' * 2>/dev/null)

#22 Updated by Charles Atkinson 8 months ago

debian/changelog

Doc https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog

bung (3.0.4-1) UNRELEASED; urgency=low

  * Initial release. Closes: #985918

 -- Charles Atkinson <bung@charlesmatkinson.org>  Fri, 02 Apr 2021 11:45:00 +0530

Build version 3.0.4

docdir='/home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/'
gitdir='/home/c/d/Auroville/Blue Light/bung/bung3'
tmpdir=$(mktemp -d /tmp/bung.XXXXXXXXXX)
cd "$gitdir" \
    && version=$(<source/usr/lib/bung/version.scrippet) \
    && version=${version#*=} \
    && mkdir $tmpdir/bung-$version \
    && rsync -a source/ $tmpdir/bung-$version \
    && cd $tmpdir \
    && date=$(date '+%-d %b %Y') \
    && for section in 5 8
    do
        line_match='.TH bung\.[^ ]\+ '$section' "[^"]\+" "Auroville" "Version '
        subs_match='\(.TH bung\.[^ ]\+ '$section' "\)'
        subs_match+='[^"]\+'
        subs_match+='\(" "Auroville" "Version \)' 
        subs_match+='[^"]\+'
        subs_match+='\(.*$\)'
        subs_repl="\1$date\2$version\3" 
        for f in bung-$version/usr/share/man/man$section/*.$section
        do   
            echo === $f ===
            sed -i \
                "/$line_match/s/$subs_match/$subs_repl/" \
                "$f" 
            gzip -9 --to-stdout $f > $f.gz
            rm "$f" 
        done 
   done \
   && find \( -name '*.org' -o -name '*.bu[0-9]' -o -name '*~' \) -delete \
   && cp -p $docdir/*Guide.{html,odt,pdf} $tmpdir/bung-$version/usr/share/doc/bung \
   && tar cvz -f ~/bung_$version.orig.tar.gz . \
   && mv ~/bung_$version.orig.tar.gz . \
   && cd $tmpdir/bung-$version \
   && rsync -a "$gitdir/source build/bung-$version/debian/" debian \
   && debuild \
   && cd .. \
   && cp -p *.{deb,debian.tar.xz,dsc,orig.tar.gz} ~
...
Now running lintian bung_3.0.4-1_amd64.changes ...
N: Using profile debian/main.
N: Starting on group bung/3.0.4-1
N: Unpacking packages in group bung/3.0.4-1
N: ----
N: Processing changes file bung (version 3.0.4-1, arch source all) ...
N: ----
N: Processing source package bung (version 3.0.4-1, arch source) ...
N: ----
N: Processing buildinfo package bung (version 3.0.4-1, arch all source) ...
N: ----
N: Processing binary package bung (version 3.0.4-1, arch all) ...
N: Finished processing group bung/3.0.4-1
Finished running lintian.
Copied built files to nextcloud storage
c@CW10:/tmp/bung.I2i0IMnf21$ cp -p bung_3.0.4* /home/c/d/Auroville/Aurinoco/nextcloud/bung/3.0/

Also available in: Atom PDF