Restoring a broken Ubuntu upgrade

Wed, 20 Nov 2024 15:17:00 +0530

A couple of weeks ago, Shantanu and I were discussing Ubuntu upgrades – his upgrade was borked and he ended up re-installing his system. I was celebrating the fact that I never really had to throw away my OS installation entirely since I switched to Ubuntu after getting tired of Arch Linux in 2011-12.

And on cue, I had a really broken update on my dad’s laptop! It was really broken – no WiFi, no Ethernet connection, no GUI packages, etc.

The crash

I started the update but forgot about it, promptly. My dad was semi-supervising it, since he was using the laptop on-and-off. I came back to it a couple of days later, and I ran an autoremove, assuming everything else in the install went ok. Everything continued to work alright until some time later, we had to restart the X server because it had hung up. And it never came back. And on a full restart, no network – both Ethernet and Wifi.

`chroot` to the rescue!

Thankfully these are not the days when I had access to a single computer, and crashing it meant running to a friend and using their computer to download stuff. I quickly made a bootable USB on my laptop.

I boot my dad’s laptop using the live distro and chroot to the installed OS using the live distro. Live Ubuntu has Ethernet working, thankfully.

For apt-get install and dist-upgrade to work correctly with the chroot, I had to mount a bunch of volumes.

Due to some network manager changes in the update, the chroot system still wasn’t connecting to the internet. I probably should’ve looked at the resolv.conf file first, but ended up going the offline package install route…

Offline updates

I generate a list of missing packages to be downloaded using the --print-uris flag to apt-get on the chroot system; And use wget to download the packages on the “live” system; Then copying these packages to /var/cache/apt/archives/ gets the update to finish.

But more missing packages!

I rebooted into the installed OS, once apt-get dist-upgrade seems to have everything resolved. But, I still didn’t have a network! I had to chroot again into the system, and this time I figured out that there was an issue with resolv.conf being a broken soft-link which seemed to cause the broken network (with Ethernet).

For WiFi, I had to install proprietary drivers for the Broadcom wireless card (BCM43142 802.11b/g/n).

I had to also install the ubuntu-desktop and related packages that seem to have gone missing, and on reboot everything was up and running!

Outro

When I told Shantanu, I didn’t have to reinstall my Ubuntu from scratch doesn’t mean that all my updates were flawless, and things never broke. I have had such broken updates a few times – network updates leaving the system in a broken state after network connections died, Ubuntu upgrade being in a borked state after the xz vulnerability was discovered, etc.

Updates do fail. But, more often than not I’ve been able to restore things thanks to dpkg ’s robustness and awesomness! And in some extreme cases, reaching for chroot to save myself.

There’s probably nothing new to learn or discover in this post, but just some help forum links that might come in handy, in future. In this age of AI answers, it seemed nice to look through some old forum QAs and work things out using them.

Thanks to Shantanu and Kamal for reading drafts of this post.

Restoring GRUB with Grub2

Tue, 10 Aug 2010 00:00:00 +0530

It’s a one liner with grub2.

$ sudo grub-install --root-directory=/mount/point/of/root/ /dev/sdX

where sdX is sda,sdb,.. depending on which hard-drive it is on.

This is the first instance when I found grub2 to be better than grub.

Telugu Rendering in Ubuntu...

Wed, 11 Jul 2007 00:00:00 +0530

I had problems with this and tried quite a few things to get it working… [uninstalling and re-installing packages… downloading fonts.. etc.. ] but couldn’t until I found help here…

[Apparently there’s a bug in the Pothana font (in Feisty)]

[ Don’t worry if you can’t figure out what’s the big deal in linking to an already existing post.. :P]

Ubuntu on Noetic Nought