Support #7156
Make server boot again
Start date:
30/11/2018
Due date:
% Done:
0%
Description
Yesterday the server could not reboot normally.
History
#1 Updated by Philippe May about 6 years ago
To make a long story short:
- Yesterday morning, found the filesystem of gisdb remounted as readonly.
- A kernel update was pending on dom0
- Rebooting the server: could not find a way to reboot it properly
#2 Updated by Philippe May about 6 years ago
Decided to install a fresh Stretch on sdc2: could boot without issue.
After fresh boot:
root@dream:~# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 vfat 8E56-E147 /boot/efi
├─sda2 btrfs 35e94cd2-7d9e-4b45-88ba-f6ea79756bb8
├─sda3 btrfs 39b40adb-70a8-4271-93c1-065bb53d7fed
├─sda4 swap a62cdc04-4846-4649-9bdd-72c3d701ec74
├─sda5 btrfs 594e4dcc-7202-4398-95d2-afb37b4f9df2
└─sda6 linux_raid_member dream:0 8f064ad3-8046-7ac4-8748-7e98b372ee9b
└─md0 LVM2_member X4Jifu-HeCl-fhzw-fYbM-rdm2-dYLY-kbqB8K
├─dream.csr-gisaf.csr.av--swap swap d54dae72-6d79-4c74-a9d9-03dfdce55000
├─dream.csr-gisaf.csr.av--disk ext4 92db39ef-d7f0-4227-97f6-2992196a09bb
├─dream.csr-infra.csr.av--swap swap c9745d54-c279-4a33-85d9-ef512ae76dfd
├─dream.csr-infra.csr.av--disk ext4 097afe53-8590-4bee-a979-595658f34cf4
├─dream.csr-samba.csr.av--swap swap 826e3a7c-7fa1-4c9c-892b-bcdb64abdc47
├─dream.csr-samba.csr.av--disk ext4 8a7004b1-d0ae-4b8e-aa3f-15b26f2d96d2
├─dream.csr-gisaf2.csr.av--swap swap 581499fd-3d3a-4ccf-b097-dce8d6eace24
├─dream.csr-gisaf2.csr.av--disk ext4 fb9221ae-d10b-4c70-bc60-b48ec001729d
├─dream.csr-freedombox--swap swap 1d818299-0433-4d8f-823d-2f12704be732
├─dream.csr-freedombox--disk ext4 0ae31051-aaf7-4589-b00d-e1c103de63cb
├─dream.csr-gisdb.csr.av--swap swap b8fad7c6-1b86-4509-adbc-3806702e6b97
├─dream.csr-gisdb.csr.av--disk ext4 dea7e1e3-197d-40ac-8892-2358c7a38619
├─dream.csr-jupyter.csr.av--swap swap 71450732-86a4-4496-9073-82287961925c
└─dream.csr-jupyter.csr.av--disk ext4 51b56adf-b971-43b1-882f-9addb86244f6
sdb
├─sdb1 vfat 06D4-5B7C
├─sdb2
├─sdb3
├─sdb4 swap a55edd67-196f-4405-bbcc-d6abb7114ac4
└─sdb5
sdc
├─sdc1 ext4 backup 6b8e0ed6-4b99-4d41-b6a2-4ea37c023726
├─sdc2 ext4 06195c52-a6c4-4cf1-90ae-53827b09bea2 /
└─sdc3 swap b29a8022-07ea-49cf-bb52-31820d308e2c [SWAP]
sdd
├─sdd1 vfat D540-3601
├─sdd2 btrfs 35e94cd2-7d9e-4b45-88ba-f6ea79756bb8
├─sdd3 btrfs 39b40adb-70a8-4271-93c1-065bb53d7fed
├─sdd4 swap d0ac7c97-5e83-47c4-90ac-c03061d93b73
├─sdd5 btrfs 594e4dcc-7202-4398-95d2-afb37b4f9df2
└─sdd6 linux_raid_member dream:0 8f064ad3-8046-7ac4-8748-7e98b372ee9b
└─md0 LVM2_member X4Jifu-HeCl-fhzw-fYbM-rdm2-dYLY-kbqB8K
├─dream.csr-gisaf.csr.av--swap swap d54dae72-6d79-4c74-a9d9-03dfdce55000
├─dream.csr-gisaf.csr.av--disk ext4 92db39ef-d7f0-4227-97f6-2992196a09bb
├─dream.csr-infra.csr.av--swap swap c9745d54-c279-4a33-85d9-ef512ae76dfd
├─dream.csr-infra.csr.av--disk ext4 097afe53-8590-4bee-a979-595658f34cf4
├─dream.csr-samba.csr.av--swap swap 826e3a7c-7fa1-4c9c-892b-bcdb64abdc47
├─dream.csr-samba.csr.av--disk ext4 8a7004b1-d0ae-4b8e-aa3f-15b26f2d96d2
├─dream.csr-gisaf2.csr.av--swap swap 581499fd-3d3a-4ccf-b097-dce8d6eace24
├─dream.csr-gisaf2.csr.av--disk ext4 fb9221ae-d10b-4c70-bc60-b48ec001729d
├─dream.csr-freedombox--swap swap 1d818299-0433-4d8f-823d-2f12704be732
├─dream.csr-freedombox--disk ext4 0ae31051-aaf7-4589-b00d-e1c103de63cb
├─dream.csr-gisdb.csr.av--swap swap b8fad7c6-1b86-4509-adbc-3806702e6b97
├─dream.csr-gisdb.csr.av--disk ext4 dea7e1e3-197d-40ac-8892-2358c7a38619
├─dream.csr-jupyter.csr.av--swap swap 71450732-86a4-4496-9073-82287961925c
└─dream.csr-jupyter.csr.av--disk ext4 51b56adf-b971-43b1-882f-9addb86244f6
#3 Updated by Philippe May about 6 years ago
root@dream:~# uname -a Linux dream 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
apt install xen-system-amd64 libvirt-daemon libvirt-daemon-system
Reboot: OK. Systemctl: all OK.
#4 Updated by Philippe May about 6 years ago
Allow root login by ssh (/etc/ssh/sshd_config).
Connect with virt-manager (using xen+ssh://root@192.168.1.107/): OK.
Copy the libvirt config:
mkdir /mnt/sda2 mount /dev/sda2 /mnt/sda2 cp -a /mnt/sda2/etc/libvirt/libxl /etc/libvirt/ cp /mnt/sda2/etc/libvirt/libxl.conf /etc/libvirt/libxl.conf systemctl restart libvirtd.service
The virtual machines show up in virt-manager.
Update the boot kernel options (importantly: dom0_mem=512M:max=512M dom0_max_vcpus=1 dom0_vcpus_pin):
cp /mnt/sda2/etc/default/grub.d/xen.cfg /etc/default/grub.d/xen.cfg update-grub
Reboot, OK.
Try to start a domU, got an error. More after lunch.
#5 Updated by Philippe May about 6 years ago
From /var/log/libvirt/libxl/libxl-driver.log:
2018-11-30 09:39:09.449+0000: xc: panic: xc_dom_core.c:208: failed to open file '/boot/vmlinuz-4.18.0-0.bpo.1-amd64': No such file or directory: Internal error
So:
root@dream:/etc/apt/sources.list.d# cat > ftp_de_debian_org_debian.list << EOF > deb http://ftp.de.debian.org/debian/ stretch-backports main contrib non-free EOF root@dream:/var/log/libvirt# cat > /etc/apt/apt.conf.d/01norecommend << EOF APT::Install-Recommends "0"; APT::Install-Suggests "0"; EOF
apt install linux-image-4.18.0-0.bpo.1-amd64
#6 Updated by Philippe May about 6 years ago
Reboot, still OK.
root@dream:/var/log/libvirt/libxl# uname -a Linux dream 4.18.0-0.bpo.1-amd64 #1 SMP Debian 4.18.6-1~bpo9+1 (2018-09-13) x86_64 GNU/Linux
By the way, boots directly into grub, skipping the refind bootloader step: one less animal in the zoo.
Still doesn't boot domU:
root@dream:/var/log/libvirt/libxl# tail /var/log/libvirt/libxl/libxl-driver.log 2018-11-30 10:11:11.627+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [1660] exited with error status 1 2018-11-30 10:11:11.627+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0 2018-11-30 10:11:11.627+0000: libxl: libxl_create.c:1461:domcreate_attach_devices: unable to add nic devices 2018-11-30 10:11:11.879+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge offline [1707] exited with error status 1 2018-11-30 10:11:11.879+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0 2018-11-30 10:11:16.542+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [1923] exited with error status 1 2018-11-30 10:11:16.542+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0 2018-11-30 10:11:16.542+0000: libxl: libxl_create.c:1461:domcreate_attach_devices: unable to add nic devices 2018-11-30 10:11:16.807+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge offline [1975] exited with error status 1 2018-11-30 10:11:16.807+0000: libxl: libxl_device.c:1237:device_hotplug_child_death_cb: script: Could not find bridge device br0
cp /mnt/sda2/etc/systemd/network/* /etc/systemd/network/ systemctl enable --now systemd-networkd.service
OK, the infra.csr.av domU starts.
#7 Updated by Philippe May about 6 years ago
Make sure grub takes the Xen image by default:
Change /etc/default/grub.d/xen.cfg and set:
XEN_OVERRIDE_GRUB_DEFAULT = 1
#8 Updated by Philippe May about 6 years ago
Installed the server on its 192.168.0.0 network and all is well.
TODO: backups.
#9 Updated by Philippe May about 6 years ago
For information: in this process, the BIOS of the PowerEdge T30 server was updated from 1.0.12 to 1.0.15.
#10 Updated by Philippe May about 6 years ago
- Status changed from New to In Progress
#11 Updated by Philippe May about 6 years ago
Ansiblize (roles: avcan):
ansible-playbook site.yml -i inventory.ini -l dream.csr.av
Backupninja:
apt install backupninja cp /mnt/sda2/etc/backup.d/* /etc/backup.d/
Add in /etc/fstab:
LABEL=backup /var/backups ext4 auto 0 0
Mount it:
mount /var/backups
#12 Updated by Philippe May almost 6 years ago
- Status changed from In Progress to Resolved