Project

General

Profile

Support #12895

Migrate backups to Bung

Added by Philippe May about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
Start date:
16/09/2021
Due date:
% Done:

0%

Close

Description

It's long time overdue to move backups of the gisdb VM to Bung.

History

#1 Updated by Charles Atkinson about 1 month ago

Very good :)

#2 Updated by Philippe May about 1 month ago

Installed the latest (3.1.1) Bung .deb from https://redmine.auroville.org.in/projects/bung/files .

Installed packages at and lsscsi which are needed (maybe shouldn't they be optional?)

Playing with bung, filing tickets for Charles ;)

#3 Updated by Philippe May about 1 month ago

Copied the provided sample super_bu sample config to /etc/bung/super_bu.conf

Then added for postgres:

Subsidiary script = postgres_bu pg.conf

with a minimal /etc/bung/pg.conf:

root@gisdb:/var/log/bung# cat /etc/bung/pg.conf 
PostgreSQL =

Test run with super_bu -c /etc/bung/super_bu.conf

The log file reports a missing directory, fixing with:

mkdir -p /var/backup/postgres

#4 Updated by Philippe May about 1 month ago

After a run with super_bu -c /etc/bung/super_bu.conf, found something happened:

root@gisdb:/var/log/bung# ll /var/backup/postgres/avgis/avgis-2021-09-16@15\:53\:00.sql 
-rw-r--r-- 1 root root 97635587 Sep 16 15:53 '/var/backup/postgres/avgis/avgis-2021-09-16@15:53:00.sql'

By the way, that file is not a sql script: the file extension is misleading (@Charles, if you read this... innocent)

Now, i suppose that i need to add a command to push via rsync, after the postgres_bu Subsidiary script (I didn't find it explicit in the documentation).

So:

root@gisdb:/var/log/bung# tail -n 2 /etc/bung/super_bu.conf 
Subsidiary script = postgres_bu pg.conf
Subsidiary script = rsync_bu to_remote.conf nice=19

root@gisdb:/var/log/bung# cat /etc/bung/to_remote.conf
rsync = /var/backup/ backup-rsync.iciti.av:/srv/remote_backup/gisdb.csr.av

Note that the rsync destination was found in the config of the current backup tool:

root@gisdb:/var/log/bung# tail -n 1 /etc/default/autopostgresqlbackup 
POSTBACKUP="rsync -a /var/lib/autopostgresqlbackup/daily/avgis backup-rsync.iciti.av:/srv/remote_backup/gisdb.csr.av" 

#5 Updated by Philippe May about 1 month ago

  • Assignee changed from Philippe May to Charles Atkinson

Ran the command super_bu -c /etc/bung/super_bu.conf.

From the log file, it seems that bung deleted on the remote server all backups done previously. That should free a lot of space there, but let's hope they won't be needed.

Eg:

...
deleting avgis/avgis_2021-06-18_22h35m.Friday.sql.gz
deleting avgis/avgis_2021-06-17_22h35m.Thursday.sql.gz
deleting avgis/avgis_2021-06-16_22h35m.Wednesday.sql.gz
deleting avgis/
...

I guess our postgres DB has been backed up and pushed to the backup server.

I'd feel more comfortable if:

@Charles, could you check if the backup has reached the remote server, before i schedule the bung job?

Note: i haven't decided yet if i'll keep autopostgresqlbackup altogether. Maybe i'll keep it and remove the rsync post-hook.

#6 Updated by Charles Atkinson about 1 month ago

  • Assignee changed from Charles Atkinson to Philippe May

Charles, could you check if the backup has reached the remote server, before i schedule the bung job?

root@backup5.iciti.av:~# tree /srv/backup/gisdb.csr.av
/srv/backup/gisdb.csr.av
├── avgis
│   ├── avgis_2021-09-10_22h35m.Friday.sql.gz
│   ├── avgis_2021-09-12_22h35m.Sunday.sql.gz
│   ├── avgis_2021-09-13_22h35m.Monday.sql.gz
│   ├── avgis_2021-09-14_22h35m.Tuesday.sql.gz
│   ├── avgis_2021-09-15_22h35m.Wednesday.sql.gz
│   └── avgis_2021-09-16_22h35m.Thursday.sql.gz
├── _Changed and deleted files
│   └── 2021
│       └── Sep
│           └── 16@16:27
│               └── avgis
│                   ├── avgis_2021-06-16_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-06-17_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-06-18_22h35m.Friday.sql.gz
│                   ├── avgis_2021-06-20_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-06-21_22h35m.Monday.sql.gz
│                   ├── avgis_2021-06-22_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-06-23_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-06-24_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-06-27_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-06-28_10h05m.Monday.sql.gz
│                   ├── avgis_2021-06-28_22h35m.Monday.sql.gz
│                   ├── avgis_2021-06-29_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-06-30_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-07-01_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-07-02_22h35m.Friday.sql.gz
│                   ├── avgis_2021-07-04_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-07-05_22h35m.Monday.sql.gz
│                   ├── avgis_2021-07-06_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-07-07_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-07-08_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-07-09_22h35m.Friday.sql.gz
│                   ├── avgis_2021-07-11_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-07-12_22h35m.Monday.sql.gz
│                   ├── avgis_2021-07-13_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-07-14_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-07-15_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-07-16_15h33m.Friday.sql.gz
│                   ├── avgis_2021-07-16_22h35m.Friday.sql.gz
│                   ├── avgis_2021-07-18_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-07-19_22h35m.Monday.sql.gz
│                   ├── avgis_2021-07-20_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-07-21_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-07-22_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-07-23_22h35m.Friday.sql.gz
│                   ├── avgis_2021-07-25_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-07-26_22h35m.Monday.sql.gz
│                   ├── avgis_2021-07-27_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-07-28_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-07-29_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-07-30_22h35m.Friday.sql.gz
│                   ├── avgis_2021-08-01_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-08-02_22h35m.Monday.sql.gz
│                   ├── avgis_2021-08-03_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-08-04_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-08-05_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-08-06_22h35m.Friday.sql.gz
│                   ├── avgis_2021-08-08_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-08-09_22h35m.Monday.sql.gz
│                   ├── avgis_2021-08-10_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-08-11_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-08-12_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-08-13_22h35m.Friday.sql.gz
│                   ├── avgis_2021-08-15_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-08-16_22h35m.Monday.sql.gz
│                   ├── avgis_2021-08-17_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-08-18_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-08-19_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-08-20_22h35m.Friday.sql.gz
│                   ├── avgis_2021-08-22_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-08-23_22h35m.Monday.sql.gz
│                   ├── avgis_2021-08-24_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-08-25_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-08-26_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-08-27_22h35m.Friday.sql.gz
│                   ├── avgis_2021-08-29_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-08-30_22h35m.Monday.sql.gz
│                   ├── avgis_2021-08-31_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-09-01_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-09-02_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-09-03_22h35m.Friday.sql.gz
│                   ├── avgis_2021-09-05_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-09-06_22h35m.Monday.sql.gz
│                   ├── avgis_2021-09-07_11h35m.Tuesday.sql.gz
│                   ├── avgis_2021-09-07_22h35m.Tuesday.sql.gz
│                   ├── avgis_2021-09-08_22h35m.Wednesday.sql.gz
│                   ├── avgis_2021-09-09_22h35m.Thursday.sql.gz
│                   ├── avgis_2021-09-10_22h35m.Friday.sql.gz
│                   ├── avgis_2021-09-12_22h35m.Sunday.sql.gz
│                   ├── avgis_2021-09-13_22h35m.Monday.sql.gz
│                   ├── avgis_2021-09-14_22h35m.Tuesday.sql.gz
│                   └── avgis_2021-09-15_22h35m.Wednesday.sql.gz
└── postgres
    ├── avgis
    │   ├── avgis-2021-09-16@15:53:00.sql
    │   └── avgis-2021-09-16@16:26:39.sql
    ├── global-2021-09-16@15:53:00.sql
    ├── global-2021-09-16@16:26:38.sql
    ├── phil
    │   ├── phil-2021-09-16@15:53:00.sql
    │   └── phil-2021-09-16@16:26:39.sql
    └── postgres
        ├── postgres-2021-09-16@15:53:00.sql
        └── postgres-2021-09-16@16:26:39.sql

10 directories, 95 files

#7 Updated by Philippe May about 1 month ago

Thanks, looks OK to me.

Configure systemd service:

root@gisdb:~# cat /etc/systemd/system/bung.service
[Unit]
Description=Backup Postgres and push to Aurinoco's server

[Service]
ExecStart=/usr/bin/super_bu -c /etc/bung/super_bu.conf

Add timer:

systemctl enable --now bung.timer

Deactivate uploads from autopostgresqlbackup (commented out rsync):

root@gisdb:~# tail -n 1 /etc/default/autopostgresqlbackup
#POSTBACKUP="rsync -a /var/lib/autopostgresqlbackup/daily/avgis backup-rsync.iciti.av:/srv/remote_backup/gisdb.csr.av" 

Also available in: Atom PDF