Home » Posts tagged "debian"

Fixing Broken Kernel Packages in Debian/Ubuntu

Sometimes you just hit really bad luck, or you’ve done something without due caution. Having too small a /boot partition, or uninstalling the wrong package can cause a system to be non-upgradable, or even non-bootable.

Filled up boot partition

If the boot partition filled up, then kernel upgrades will continually fail until space is cleared. Unfortunately, this also means that attempts to uninstall kernels through APT will fail too, because the package manager must try to finish the last failed install operation before it can proceed to further work.

You can use this script to forcefully remove excess kernel images, and then run apt-get -f install :

https://github.com/taikedz/handy-scripts/blob/master/bin/rmkernel.sh

Example:

wget https://raw.githubusercontent.com/taikedz/handy-scripts/master/bin/rmkernel.sh -O rmkernel.sh

# Keep 2 most recent kernels
bash rmkernel.sh 2 | sudo bash

# Fix broken installation process
apt-get -f install

Removing kernels in this way forcibly removes them, then runs the dependency fix, hopefully completing the incomplete kernel build operation that normally fails.

In future, during regular maintenance, remember to run the sudo apt-get autoclean && sudo apt-get autoremove command. You can automate this by placing the appropriate script in /etc/cron.daily/

Debian/Ubuntu keeps booting to memory test / cannot find kernel

If no kernel can be found, the system cannot boot. You need to rescue the system at this point.

These steps describe the process when using a Ubuntu Server DVD, but a similar workflow is applicable to pretty much any standard GNU/Linux system

1. Boot from the Ubuntu Server installation DVD

To boot from DVD in a hypervisor, poweroff the VM, edit its configuration and choose to mount a CD/DVD from filesystem (or datastore in a hosted environment).

You may find you need to force entering BIOS configuration to ensure that the CD drive is booted from before the First Hard Disk

2. Once booted in to Ubuntu, choose “Rescue a broken system” from the first menu.

You will be asked a few questions, of which network setup etc. Answer as appropriate

3. You will eventually be asked to choose a root partition – choose the appropriate partition (usually the largest one on /dev/sda)

If prompted to mount the separate /boot partition, do so

4. Get a shell “in the installer” ; you will be informed that the target (your main system you are rescuing) is mounted to /target

You will need to move/copy over the installer environment’s /etc/resolv.conf to /target/etc/resolv.conf (unlink the existing /target/etc/resolv.conf first though)

5. Switch to your target system by running chroot /target

You will now be in the same context as your original server. Run bash to get back to bash shell (by default you start in sh)

6. Run the following – note the dpkg section is one line of pipes, do not forget those “|” characters ! This is a modified version of what exists in the rmkernel.sh script from above, which essentially purges all existing kernel installation data to start anew.

dpkg --list 'linux-image*' |
 grep ii |
 awk '{print $2}' | while read; do
     dpkg --force-all --remove "$REPLY"; 
 done

apt-get update && apt-get install linux-image-generic

7. Power off the machine. Ensure there is no CD in the disk drive anymore, and bring the machine back up, this should be fixed now

 

If the above still does not add at least one bootable kernel, you may need to install a different/new kernel. Look for “linux-image” and install the latest

# find a suitable kernel image
apt-cache search linux-image

# In this example, the package chosen from the above step is linux-image-4.4.0-109-generic
apt-get install linux-image-4.4.0-109-generic

You will have a particular version of a kernel now, which may or may not continue to receive updates ; you need to consider moving to a new server or further fixing the existing one.