Project

General

Profile

Support #7156

Make server boot again

Added by Philippe May over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Start date:
30/11/2018
Due date:
% Done:

0%

Close

Description

Yesterday the server could not reboot normally.

History

#1 Updated by Philippe May over 3 years ago

To make a long story short:

- Yesterday morning, found the filesystem of gisdb remounted as readonly.

- A kernel update was pending on dom0

- Rebooting the server: could not find a way to reboot it properly

#2 Updated by Philippe May over 3 years ago

Decided to install a fresh Stretch on sdc2: could boot without issue.

After fresh boot:

root@dream:~# lsblk -f
NAME                                 FSTYPE            LABEL   UUID                                   MOUNTPOINT
sda                                                                                                   
├─sda1                               vfat                      8E56-E147                              /boot/efi
├─sda2                               btrfs                     35e94cd2-7d9e-4b45-88ba-f6ea79756bb8   
├─sda3                               btrfs                     39b40adb-70a8-4271-93c1-065bb53d7fed   
├─sda4                               swap                      a62cdc04-4846-4649-9bdd-72c3d701ec74   
├─sda5                               btrfs                     594e4dcc-7202-4398-95d2-afb37b4f9df2   
└─sda6                               linux_raid_member dream:0 8f064ad3-8046-7ac4-8748-7e98b372ee9b   
  └─md0                              LVM2_member               X4Jifu-HeCl-fhzw-fYbM-rdm2-dYLY-kbqB8K 
    ├─dream.csr-gisaf.csr.av--swap   swap                      d54dae72-6d79-4c74-a9d9-03dfdce55000   
    ├─dream.csr-gisaf.csr.av--disk   ext4                      92db39ef-d7f0-4227-97f6-2992196a09bb   
    ├─dream.csr-infra.csr.av--swap   swap                      c9745d54-c279-4a33-85d9-ef512ae76dfd   
    ├─dream.csr-infra.csr.av--disk   ext4                      097afe53-8590-4bee-a979-595658f34cf4   
    ├─dream.csr-samba.csr.av--swap   swap                      826e3a7c-7fa1-4c9c-892b-bcdb64abdc47   
    ├─dream.csr-samba.csr.av--disk   ext4                      8a7004b1-d0ae-4b8e-aa3f-15b26f2d96d2   
    ├─dream.csr-gisaf2.csr.av--swap  swap                      581499fd-3d3a-4ccf-b097-dce8d6eace24   
    ├─dream.csr-gisaf2.csr.av--disk  ext4                      fb9221ae-d10b-4c70-bc60-b48ec001729d   
    ├─dream.csr-freedombox--swap     swap                      1d818299-0433-4d8f-823d-2f12704be732   
    ├─dream.csr-freedombox--disk     ext4                      0ae31051-aaf7-4589-b00d-e1c103de63cb   
    ├─dream.csr-gisdb.csr.av--swap   swap                      b8fad7c6-1b86-4509-adbc-3806702e6b97   
    ├─dream.csr-gisdb.csr.av--disk   ext4                      dea7e1e3-197d-40ac-8892-2358c7a38619   
    ├─dream.csr-jupyter.csr.av--swap swap                      71450732-86a4-4496-9073-82287961925c   
    └─dream.csr-jupyter.csr.av--disk ext4                      51b56adf-b971-43b1-882f-9addb86244f6   
sdb                                                                                                   
├─sdb1                               vfat                      06D4-5B7C                              
├─sdb2                                                                                                
├─sdb3                                                                                                
├─sdb4                               swap                      a55edd67-196f-4405-bbcc-d6abb7114ac4   
└─sdb5                                                                                                
sdc                                                                                                   
├─sdc1                               ext4              backup  6b8e0ed6-4b99-4d41-b6a2-4ea37c023726   
├─sdc2                               ext4                      06195c52-a6c4-4cf1-90ae-53827b09bea2   /
└─sdc3                               swap                      b29a8022-07ea-49cf-bb52-31820d308e2c   [SWAP]
sdd                                                                                                   
├─sdd1                               vfat                      D540-3601                              
├─sdd2                               btrfs                     35e94cd2-7d9e-4b45-88ba-f6ea79756bb8   
├─sdd3                               btrfs                     39b40adb-70a8-4271-93c1-065bb53d7fed   
├─sdd4                               swap                      d0ac7c97-5e83-47c4-90ac-c03061d93b73   
├─sdd5                               btrfs                     594e4dcc-7202-4398-95d2-afb37b4f9df2   
└─sdd6                               linux_raid_member dream:0 8f064ad3-8046-7ac4-8748-7e98b372ee9b   
  └─md0                              LVM2_member               X4Jifu-HeCl-fhzw-fYbM-rdm2-dYLY-kbqB8K 
    ├─dream.csr-gisaf.csr.av--swap   swap                      d54dae72-6d79-4c74-a9d9-03dfdce55000   
    ├─dream.csr-gisaf.csr.av--disk   ext4                      92db39ef-d7f0-4227-97f6-2992196a09bb   
    ├─dream.csr-infra.csr.av--swap   swap                      c9745d54-c279-4a33-85d9-ef512ae76dfd   
    ├─dream.csr-infra.csr.av--disk   ext4                      097afe53-8590-4bee-a979-595658f34cf4   
    ├─dream.csr-samba.csr.av--swap   swap                      826e3a7c-7fa1-4c9c-892b-bcdb64abdc47   
    ├─dream.csr-samba.csr.av--disk   ext4                      8a7004b1-d0ae-4b8e-aa3f-15b26f2d96d2   
    ├─dream.csr-gisaf2.csr.av--swap  swap                      581499fd-3d3a-4ccf-b097-dce8d6eace24   
    ├─dream.csr-gisaf2.csr.av--disk  ext4                      fb9221ae-d10b-4c70-bc60-b48ec001729d   
    ├─dream.csr-freedombox--swap     swap                      1d818299-0433-4d8f-823d-2f12704be732   
    ├─dream.csr-freedombox--disk     ext4                      0ae31051-aaf7-4589-b00d-e1c103de63cb   
    ├─dream.csr-gisdb.csr.av--swap   swap                      b8fad7c6-1b86-4509-adbc-3806702e6b97   
    ├─dream.csr-gisdb.csr.av--disk   ext4                      dea7e1e3-197d-40ac-8892-2358c7a38619   
    ├─dream.csr-jupyter.csr.av--swap swap                      71450732-86a4-4496-9073-82287961925c   
    └─dream.csr-jupyter.csr.av--disk ext4                      51b56adf-b971-43b1-882f-9addb86244f6   

#3 Updated by Philippe May over 3 years ago

root@dream:~# uname -a
Linux dream 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
apt install xen-system-amd64 libvirt-daemon libvirt-daemon-system

Reboot: OK. Systemctl: all OK.

#4 Updated by Philippe May over 3 years ago

Allow root login by ssh (/etc/ssh/sshd_config).

Connect with virt-manager (using xen+ssh://root@192.168.1.107/): OK.

Copy the libvirt config:

mkdir /mnt/sda2
mount /dev/sda2 /mnt/sda2
cp -a /mnt/sda2/etc/libvirt/libxl /etc/libvirt/
cp /mnt/sda2/etc/libvirt/libxl.conf /etc/libvirt/libxl.conf
systemctl restart libvirtd.service

The virtual machines show up in virt-manager.

Update the boot kernel options (importantly: dom0_mem=512M:max=512M dom0_max_vcpus=1 dom0_vcpus_pin):

cp /mnt/sda2/etc/default/grub.d/xen.cfg /etc/default/grub.d/xen.cfg
update-grub

Reboot, OK.

Try to start a domU, got an error. More after lunch.

#5 Updated by Philippe May over 3 years ago

From /var/log/libvirt/libxl/libxl-driver.log:

2018-11-30 09:39:09.449+0000: xc: panic: xc_dom_core.c:208: failed to open file '/boot/vmlinuz-4.18.0-0.bpo.1-amd64': No such file or directory: Internal error

So:

root@dream:/etc/apt/sources.list.d# cat > ftp_de_debian_org_debian.list << EOF
> deb http://ftp.de.debian.org/debian/ stretch-backports main contrib non-free
EOF
root@dream:/var/log/libvirt# cat > /etc/apt/apt.conf.d/01norecommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF
apt install linux-image-4.18.0-0.bpo.1-amd64

#6 Updated by Philippe May over 3 years ago

Reboot, still OK.

root@dream:/var/log/libvirt/libxl# uname -a
Linux dream 4.18.0-0.bpo.1-amd64 #1 SMP Debian 4.18.6-1~bpo9+1 (2018-09-13) x86_64 GNU/Linux

By the way, boots directly into grub, skipping the refind bootloader step: one less animal in the zoo.

Still doesn't boot domU:

root@dream:/var/log/libvirt/libxl# tail /var/log/libvirt/libxl/libxl-driver.log
2018-11-30 10:11:11.627+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [1660] exited with error status 1
2018-11-30 10:11:11.627+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0
2018-11-30 10:11:11.627+0000: libxl: libxl_create.c:1461:domcreate_attach_devices: unable to add nic devices
2018-11-30 10:11:11.879+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge offline [1707] exited with error status 1
2018-11-30 10:11:11.879+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0
2018-11-30 10:11:16.542+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [1923] exited with error status 1
2018-11-30 10:11:16.542+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0
2018-11-30 10:11:16.542+0000: libxl: libxl_create.c:1461:domcreate_attach_devices: unable to add nic devices
2018-11-30 10:11:16.807+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge offline [1975] exited with error status 1
2018-11-30 10:11:16.807+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0

cp /mnt/sda2/etc/systemd/network/* /etc/systemd/network/
systemctl enable --now systemd-networkd.service

OK, the infra.csr.av domU starts.

#7 Updated by Philippe May over 3 years ago

Make sure grub takes the Xen image by default:

Change /etc/default/grub.d/xen.cfg and set:

XEN_OVERRIDE_GRUB_DEFAULT = 1

#8 Updated by Philippe May over 3 years ago

Installed the server on its 192.168.0.0 network and all is well.

TODO: backups.

#9 Updated by Philippe May over 3 years ago

For information: in this process, the BIOS of the PowerEdge T30 server was updated from 1.0.12 to 1.0.15.

#10 Updated by Philippe May over 3 years ago

  • Status changed from New to In Progress

#11 Updated by Philippe May over 3 years ago

Ansiblize (roles: avcan):

ansible-playbook site.yml -i inventory.ini -l dream.csr.av

Backupninja:

apt install backupninja
cp /mnt/sda2/etc/backup.d/* /etc/backup.d/

Add in /etc/fstab:

LABEL=backup /var/backups ext4 auto 0 0

Mount it:

mount /var/backups

#12 Updated by Philippe May over 3 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF