Project

General

Profile

Support #7161

Setup remote backup

Added by Philippe May over 3 years ago. Updated over 3 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Start date:
30/11/2018
Due date:
% Done:

0%

Close

Description

Setup an off-site backup for the server on Aurinoco's backup (Bacula?).

TODO: define exactly what to backup: database for sure, whole domUs, whole LVM?

History

#1 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

Charles, it's a long term topic that i'm finally taking up.

How do you want to proceed? Is there a documentation? Discussion for our weekly meeting?

#2 Updated by Charles Atkinson over 3 years ago

  • Status changed from New to In Progress
  • Assignee changed from Charles Atkinson to Philippe May

Let's discuss face to face some convenient time TBD

#3 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

As discussed, can you please send me OpenVPN certs and your ssh key?

#4 Updated by Charles Atkinson over 3 years ago

  • Assignee changed from Charles Atkinson to Philippe May

For the OpenVPN certs, what is the client's FQDN? My ssh key emailed to you.

#5 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

Starting with the DB server: gisdb.csr.av (csr.av being handled by a local bind server)

#6 Updated by Philippe May over 3 years ago

OK, your ssh key was already there: server was ansiblized with the BL server role (equivalent of Bliss) including BL's standard set of keys sunglasses

#7 Updated by Charles Atkinson over 3 years ago

Generating OpenVPN certs

Doc: [[Aurinoco Systems:OpenVPN_24_operations#On-the-OpenVPN-server]]

  • Created openvpn2.iciti.av:/etc/bind/pri.csr.av with A record for gisdb and address 172.16.9.1.
  • Extended openvpn2.iciti.av:/etc/bind/named.conf.local with pri.csr.av
  • Created OpenVPN certs file:
    root@openvpn2.iciti:~# /root/scripts/openvpn/setup_client_on_server.sh -f gisdb.csr.av -i 172.16.9.1
    Creating a temporary directory
    Checking for existing client certifficate and key files
    Creating certificate and key
    Certificate and key successully created:
    -rw------- 1 root root 4446 Dec 11 16:02 /etc/openvpn/easy-rsa/pki/issued/gisdb.csr.av.crt
    -rw------- 1 root root 1704 Dec 11 16:02 /etc/openvpn/easy-rsa/pki/private/gisdb.csr.av.key
    Creating /etc/openvpn/ccd/gisdb.csr.av
    Creating the client configuration inline file
    The next step is to copy /etc/openvpn/client_inline_files/gisdb.csr.av.ICITI.ovpn to the client
    
  • Fixed the above typos and pushed to git
  • Mailed the file privately to Phil

#8 Updated by Charles Atkinson over 3 years ago

  • Assignee changed from Charles Atkinson to Philippe May

Certs file mailed to you, Phil. Installation procedure: [[Aurinoco Systems:OpenVPN_24_operations#Debian]]

#9 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

Looks OK: 172.16.9.1 @ tun0.

Just a small note on openvpn management by systemd: i prefer to have the config in /etc/openvpn/client, and the service at openvpn-client@gisdb.csr.av.ICITI.service. It makes it more explicit to differentiate between server and client. Ref: https://unix.stackexchange.com/questions/409665/starting-openvpn-client-as-daemon-in-debian

Back to you to log in and proceed, the database dumps are in:

root@gisdb:/var/log# ll /var/lib/autopostgresqlbackup/daily/avgis
total 162512
drwxr-xr-x 2 root postgres     4096 Dec 11 06:25 ./
drwxr-xr-x 7 root postgres     4096 Nov 21 17:58 ../
-rw------- 1 root root     27573587 Nov 25 06:25 avgis_2018-11-25_06h25m.Sunday.sql.gz
-rw------- 1 root root     27715338 Dec  3 06:25 avgis_2018-12-03_06h25m.Monday.sql.gz
-rw------- 1 root root     27749682 Dec  5 06:25 avgis_2018-12-05_06h25m.Wednesday.sql.gz
-rw------- 1 root root     27761994 Dec  6 06:25 avgis_2018-12-06_06h25m.Thursday.sql.gz
-rw------- 1 root root     27773790 Dec  7 06:25 avgis_2018-12-07_06h25m.Friday.sql.gz
-rw------- 1 root root     27822839 Dec 11 06:25 avgis_2018-12-11_06h25m.Tuesday.sql.gz

We'll see how it goes with this directory first.

#10 Updated by Charles Atkinson over 3 years ago

  • Assignee changed from Charles Atkinson to Philippe May

Regards having the config in /etc/openvpn/client, from the linked page (dated 8 Dec 2017):

Note that newer versions of OpenVPN have split the configuration files directory into /etc/openvpn/client and /etc/openvpn/server. This has not (yet) percolated down into a stable version of Debian

When designing the current implementation I considered introducing /etc/openvpn/{client,server} but they would break the Stretch systemd OpenVPN generator which only works with /etc/openvpn/*.conf files.

Sorry -- somehow I have disabled ssh access:

c@CW10:~$ ssh -A root@172.16.9.1
root@172.16.9.1's password: 
It was working until I fumbled copying some files into /root for my personal convenience like .bashrc_scrippet_for_charles and .bashrc.d and its contents. cold_sweat

#11 Updated by Philippe May over 3 years ago

Oh oh... open_mouth

Halt the domU, mount the file system on dom0.

Found that /root was owned by unknown user 10012. Reset uid to 0.

Umount the file system from dom0, reboot: OK.

Using a production system as a first machine to back up wasn't such a great idea.

And, finally, i might prefer to set up Bung myself... Let's put it on hold for a while.

#12 Updated by Charles Atkinson over 3 years ago

And, finally, i might prefer to set up Bung myself... Let's put it on hold for a while

As you wish (I don't normally screw up as above).

#13 Updated by Philippe May over 3 years ago

Just another validation of Murphy's law :)

We might also take this opportunity to validate the installation process by someone who is quite a used to test the above-mentioned law (me).

#14 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

I installed bung.

I think that we now need to set up the backup target. I checked https://redmine.auroville.org.in/projects/backup-service , but it doesn't have information, maybe a Wiki page would be nice there.

How would you like to proceed now? You create the target destination, give permissions, and give me the URL for rsync?

The required size of backup space would be less than 1GB (less than 30 MB per day, 1 month retention).

#15 Updated by Charles Atkinson over 3 years ago

  • Assignee changed from Charles Atkinson to Philippe May

The remote server should be backup-rsync.iciti.av (currently resolves to backup3.iciti.av). Conventionally the ssh host used to access it and the comment in the ssh keys would be gisdb.csr_to_backup-rsync.iciti

The remote path should be /srv/remote_backup/gisdb.csr.av.

If you send me the public key, I will install it in backup3.iciti.av:/root/.ssh/authorized_keys2 with the command= restriction normally used with bung.

#16 Updated by Philippe May over 3 years ago

  • Assignee changed from Philippe May to Charles Atkinson

Here's the private public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Yv7i/yXDFy1gZzW4tagHWzWkUDEeTqinKncmhOiPpfTDpQG9Ug4RIZRVBPq9yirDXhXWfSWzxfgsthwLToaiIL0mSj8qyPJuBFS/apOHrMok2jkAqzqsqB/7CeGMLN28RvM0AC1/aj8emsuNHmhD0iU5scObgjqxMuXwNezyXMmXVUcwmNnM//ariY53MbepybxhxLa0ft43uzmnZ5wodtZHGgYdRj+ncK9saz1tLoB2qNdn/zmU4E/RrpPHsSj0SH3V3nFLoLu57loyGjZ92yq06Iln3VLNZe8TGylBt3EMmSxlVX5zbLc4uOmc74EfVwSYF66Pu1Dyev5Cz1FT root@gisdb

#17 Updated by Charles Atkinson over 3 years ago

  • Assignee changed from Charles Atkinson to Philippe May

Installed it in backup3.iciti.av:/root/.ssh/authorized_keys2 with the command= restriction normally used with bung.

#18 Updated by Philippe May over 3 years ago

Seems to be working, thanks Charles :)

Schedule the job:

root@gisdb:/etc/systemd/system# cat bung.service 
[Unit]
Description=Remote backup to Aurinoco

[Service]
ExecStart=/opt/bung/rsync_bu.sh -c /etc/opt/bung/rsync_bu.conf
root@gisdb:/etc/systemd/system# cat bung.timer 
[Unit]
Description=Runs the remote backup periodically

[Timer]
OnBootSec=15min
OnCalendar=*-*-* 22:43:00
Persistent=true

[Install]
WantedBy=timers.target

TODO: cross check the logs tomorrow.

#19 Updated by Philippe May over 3 years ago

I didn't set the remote host on the config file, so was rsync-ing on localhost.

Looks OK now.

Here's the config:

root@gisdb:/etc/opt/bung# cat rsync_bu.conf 
Organisation name = csr.av
rsync = /var/lib/autopostgresqlbackup/daily/avgis backup-rsync.iciti.av:/srv/remote_backup/gisdb.csr.av options="--archive --verbose" 

Next to backup:

  • import baskets

Also available in: Atom PDF