Home » Computing » Archive by category "Linux"

Fixing Broken Kernel Packages in Debian/Ubuntu

Sometimes you just hit really bad luck, or you’ve done something without due caution. Having too small a /boot partition, or uninstalling the wrong package can cause a system to be non-upgradable, or even non-bootable.

Filled up boot partition

If the boot partition filled up, then kernel upgrades will continually fail until space is cleared. Unfortunately, this also means that attempts to uninstall kernels through APT will fail too, because the package manager must try to finish the last failed install operation before it can proceed to further work.

You can use this script to forcefully remove excess kernel images, and then run apt-get -f install :

https://github.com/taikedz/handy-scripts/blob/master/bin/rmkernel.sh

Example:

wget https://raw.githubusercontent.com/taikedz/handy-scripts/master/bin/rmkernel.sh -O rmkernel.sh

# Keep 2 most recent kernels
bash rmkernel.sh 2 | sudo bash

# Fix broken installation process
apt-get -f install

Removing kernels in this way forcibly removes them, then runs the dependency fix, hopefully completing the incomplete kernel build operation that normally fails.

In future, during regular maintenance, remember to run the sudo apt-get autoclean && sudo apt-get autoremove command. You can automate this by placing the appropriate script in /etc/cron.daily/

Debian/Ubuntu keeps booting to memory test / cannot find kernel

If no kernel can be found, the system cannot boot. You need to rescue the system at this point.

These steps describe the process when using a Ubuntu Server DVD, but a similar workflow is applicable to pretty much any standard GNU/Linux system

1. Boot from the Ubuntu Server installation DVD

To boot from DVD in a hypervisor, poweroff the VM, edit its configuration and choose to mount a CD/DVD from filesystem (or datastore in a hosted environment).

You may find you need to force entering BIOS configuration to ensure that the CD drive is booted from before the First Hard Disk

2. Once booted in to Ubuntu, choose “Rescue a broken system” from the first menu.

You will be asked a few questions, of which network setup etc. Answer as appropriate

3. You will eventually be asked to choose a root partition – choose the appropriate partition (usually the largest one on /dev/sda)

If prompted to mount the separate /boot partition, do so

4. Get a shell “in the installer” ; you will be informed that the target (your main system you are rescuing) is mounted to /target

You will need to move/copy over the installer environment’s /etc/resolv.conf to /target/etc/resolv.conf (unlink the existing /target/etc/resolv.conf first though)

5. Switch to your target system by running chroot /target

You will now be in the same context as your original server. Run bash to get back to bash shell (by default you start in sh)

6. Run the following – note the dpkg section is one line of pipes, do not forget those “|” characters ! This is a modified version of what exists in the rmkernel.sh script from above, which essentially purges all existing kernel installation data to start anew.

dpkg --list 'linux-image*' |
 grep ii |
 awk '{print $2}' | while read; do
     dpkg --force-all --remove "$REPLY"; 
 done

apt-get update && apt-get install linux-image-generic

7. Power off the machine. Ensure there is no CD in the disk drive anymore, and bring the machine back up, this should be fixed now

If the above still does not add at least one bootable kernel, you may need to install a different/new kernel. Look for “linux-image” and install the latest

# find a suitable kernel image
apt-cache search linux-image

# In this example, the package chosen from the above step is linux-image-4.4.0-109-generic
apt-get install linux-image-4.4.0-109-generic

You will have a particular version of a kernel now, which may or may not continue to receive updates ; you need to consider moving to a new server or further fixing the existing one.

The Power of #!/bin/bash

The pictured snippet is probably one of the dirtier pieces of code I have had the misfortune of needing – and not being able to refactor to anything cleaner.

I am trying to make bash my main programming language, and to this goal I have created a number of tools to aid me on my way: a library of code snippets that anybody can re-use, a packing tool to create executables, a pre-compilation tool to add compiler directives to bash, a build tool to pull all these together, and a make/release tool to manage versions. And I’m not yet done.

Of the main efforts here is the library of scriptlets.

Once of the main attractions of one language over the other is the availability of a large amount of re-usable code released as libraries and which are, pretty much, taken for granted.

Python can get modules from pip, there are Java libraries in JAR files around the web and even JavaScript has such libraries as jQuery to help to write terser, more manageable code more easily.

bash has none of this. And quite a few quirks. You have to contend with the very string-y way of passing data around (arrays are a bit of a nightmare until you get used to the arcane notations), and nearly every non-trivial operation you want to do is a command, an external process.

The inset image depicts some code I wrote so that assigning variables

But that is also exactly what makes so appealing – any language, and its associated libraries, can become your library!

For starters, python, perl and PHP can all be leveraged for their respective strengths in specific situations, and it is possible to write and store additional scripts such as long MySQL tasks in their own files. Use a perl script to write change on the fly, and pipe to MySQL.

Consider the following silly example:

controlapp -getusers |
    perl alter_sql.pl "$SCENARIO/mysqltemplate.sql" |
    mysql -u "$sqluser" -p"$sqlpass" "$mydb" |
    php to_xml.php |
    controlapp -dostuff

It looks quite awful at first glance, but consider the power of perl to perform text processing, keeping your mysql files ordered in a directory-based hierarchy, and passing the result to PHP which will handle outputting XML much more ergonomically than PHP.

So whilst bash does have its odd and at times even infuriating quirks, I am learning to love it.

It is now after all the de facto language of DevOps. It seems even Windows can no longer do without it 🙂

Posted on April 26, 2016 by Tai Kedzierski, Reply

Let It B…SD – and how to record songs with Audacity (Open Source Free Software)

A few week-ends ago I had a go at recording “Let It BSD,” a pastiche of “Let It Be,” focused on the BSD operating system. It was the first time for me in years that I had had a go at recording music.

What I used to do when I was in high school was to record myself playing on a cassette tape, then play that back through my parents’ dual-tape hi-fi system whilst recording the vocal track onto the second tape. If I was happy with that I would be able to record a third time by re-recording in similar fashion over the first tape. Onerous, time-consuming (especially when I made mistakes), and with very limited mixing opportunities (read: none), it was a rather challenging (vexing) experience.

That was 2002. 14 years on, the technology available for casual hobbyist recording has come leaps and bounds; and no, I did not need any particularly powerful equipment for this at all. A modern laptop (from within the last couple of years) and a couple of small accessories are enough.

The song, the subject

Play Let It BSD (new tab)

BSD stands for “Berkley Software Distribution,” and generally refers to a variety of related operating systems based off of the original 386BSD from the 80’s, itself derived from the original portable version of UNIX.
Let It Be is a fairly cheesy, albeit popular, song by the Beatles, which I am not sure is appropriate to sing in all times of conflict (I was never sure of what we should “let be.”)
“Let It BSD“‘s lyrics were written by Jacqueline Kory Westlund, as a result of having heard one too many episodes of her husband’s favorite tech security and systems podcasts, TechSnap and BSD Now.
JKW released her parody lyrics under a Creative Commons with Attribution license (CC/Attr), which is a license for content creators that allows everyone to share and modify material, so long as the original author/s is/are given credit in appropriate and visible form. Which is fortunate for me, because I was not able to place a comment on her blog to ask/thank her.
My recorded track is, as such, also released under Creative Commons/Attr 4.0 license, for anyone to do what they would like to with.

My Setup – hardware and software

On the hardware side, I used

a Lenovo Flex laptop and simply its built-in mic
a set of headphones (really good Sony ones, cost me about £40 a few months ago)
and a USB stick.

I used headphones (not earbuds) to get the best pitch range on playback.

I used the USB stick to record the temp files to – in Audacity preferences under the Directories section, you can specify what space to use for the temp directory. Since my laptop has a HDD and I did not want the fans or disk kicking in, a USB was a suitable workaround. It wouldn’t have been necessary if I’d had an SSD.

4GB might have been just about sufficient with no other apps present and a lightweight desktop; and as I type, I wonder, if I’d had less RAM, if using a file on the USB stick for a swap file, would have helped…

For the software, recording and “mixing” was done in Audacity, running on Ubuntu MATE (a Linux system — yes, ironic isn’t it). I had two windows with the lyrics open so that I could have everything in front of me fully annotated, so no scrolling would ever be needed. Paper would have been an acceptable substitute.

Instruments involved

a mandolin (two actually for different sound qualities and ease of handling)
a steel string acoustic guitar
a Spanish nylon-strung guitar
a tambourine
and a metronome (because I had not yet found out about the click track feature built-in to Audacity!)

You can tune manually, but having a guitar tuner makes it all the easier. Pity my cheap penny whistles are all out of tune.

Method

The most frequently used keyboard shortcuts used during recording will likely be these:

(R) record
(P) pause
(Space) stop/play
(J) jump to start of track
(K) jump to end of track

Starting a recording writes to a new track, always. You can use the ←→ dual arrow tool to move track pieces around, and split tracks on the cursor in the Edit menu: Labeled Audio.

First take: I recorded myself strumming and singing against a metronome, to lay down the reference track.
Second take: playing the first take back in the headphones, I did a new take solely recording the guitar being strummed.
Third take: this was supposed to be the vocal track, but since I had to turn off the first recording I lost the metronome ticks, so third take was …. clapping in time to the first track to create a poor man’s click track.
Third take (bis): I recorded a first take of the vocals. This was not so much to be a final take, but rather to serve as a guide as to where I was in the song on subsequent takes.
Fourth take: tambourine. It turns out playing a tambourine so that it blends in to a song decently is not quite as easy as just shaking and beating it. You want to shake it fairly deftly to avoid jangling at odd moments, which requires constant concentration…!
Fifth take: Spanish guitar arpeggiations, nothing too fancy. The bright timbre of the steel string folk guitar was much more preferable for strumming the background, so arpeggiation was left to the mellower nylon-strung Spanish guitar. I could have used an electric guitar to get a different timbre, but I wanted to keep it all as “acoustic” as possible.
Sixth take: Here’s where I cheated a little – I tuned the Spanish guitar to a drop-D, and played my bass track on it. In Audacity I then used the Change Pitch effect to drop the track by an octave, amplified it a little to bring the sound back and voilà – I have no bass guitar, but still have a bass track 🙂 The downside is that the low D does not translate well to digital re-tuning down by an octave, so it sounds a little funky. Not sure how to resolve this.
Seventh & Eighth takes: by now with my old cassette tape method I’d probably have been tearing my hair out and weeping in despair. At this point I was recording two backing vocals at the end of the track; harmonies to accompany the final slew of choruses. I actually reduced the volume on the reference vocal track to minimize distraction. Singing a harmony and keeping to it is not easy when a more familiar tune is being sung into your head. Even trickier to get two harmony voices in and keep to them. I sang in my normal voice, and in falsetto, to be sure to get different timbres.
Ninth track: solo time. Grabbed the ash mandolin to do this, it has a slightly higher action which suits me better for melody playing. I didn’t write anything for this, in fact I replayed the entire track from the beginning and practised scales and mini-licks until I got to the solo area, paused a bit, and improvised along through the solo area. I cut the rest from the take, it took me about 2-3 takes to get something I liked, then another one when I realized I had deleted it during an ill-advised bout of undo-redo. Bleh. No two takes were the same.
Tenth track: easy one – redburst mandolin with a lower action, which I just tried to strum as fast as possible. I was originally going to have it all through the choruses, but sustaining that proved too much for me, so only kept it in the final flourish.
Eleventh track: final vocals. I can sing in different registers. You don’t want to hear what happened when I went up an octave. For this one, rather than use my normal baritone voice, I tried to keep the timbre higher. The original take (3bis) made me sound a bit like an opera singer trying to do folk. Yeuch.
Final track: the Allan track. My intention was to grab clips of Allan Jude saying the names of each of the BSD flavours and substituting them accordingly, but I didn’t have the courage to actually go through a whole heap of shows to identify where he might have said each line, if at all. So we’re stuck with my cheap imitation-Canadian accent.

How affordable does this make hobbyist music recording?

(The rig I did all this with is a little more powerful than regular laptops; I have a small Gigabyte Brix with 4GB RAM and a HDD, 2 processors. I will need to use an external webcam as it has no built-in mic, but could still make sure to keep it near the unit to simulate an internal mic. I’ll do some tests there too to find out whether recording with that setup would be viable.)

System

If you do not have access to high-end PCs or modern laptops with top-specs, you’re probably finding yourself limited in choice. Getting a second-hand laptop from the last 3 years would probably work fine for the task, preferably with at least 4GB RAM

Your base minimum would probably be dual-core at 1.4GHz, 2-4GB RAM, and 2-4 USB ports. Any PC/laptop produced within the last 4 years should be able to manage that.

The laptop I used has 8GB RAM which probably helps in keeping recording + playback in memory responsive, reducing recourse to the fans I expect.

If you do not have this amount of RAM, and can’t get/afford any modules to expand, there still might be a way – create a swap file on a second USB stick, which would burn through the stick faster (and you probably would only ever want to use it as swap thereafter), but would prevent the HDD from kicking in when memory needs to be offloaded to disk. I’ve heard that the stick burns out faster used in this way, but no idea what timeframes – hours, days, weeks or months of usage.

I don’t think you can explicitly and dynamically configure swap location in Windows or OS X (even though the latter is lightly related to BSD), but in strict BSD and Linux it’s a doozy – as root just do the following

swapoff # turn off normal swap, wherever it is  
SWAPFILE=/path/to/swap/file/swapfile
dd if=/dev/zero of="$SWAPFILE" bs=1024 count=$(( 4 * 1024 * 1024 )) # 4GB swap file
mkswap "$SWAPFILE"
swapon "$SWAPFILE" # since we turned off all other swap, we only swap on this file

And then remove it from swap and turn on the normal swap

swapoff
mount -a # assuming swap is normally set up in fstab

Recording Software

You would be hard pressed to get such a complete solution for basic audio recording and mixing as Audacity for free. Apple’s Garage Band costs to acquire now if you don’t have it preinstalled (I don’t, I had to reinstall my Mac some time in the past) or want the latest version, and requires off the bat that you buy a Mac. Microsoft does not make or bundle any similar-grade software out of the box, and Adobe’s solution is aimed at professionals, with a price tag to match.

Audacity works on BSD, Linux, Windows and Mac OS X and whilst not as feature-complete and pretty as its commercial counterparts is still very flexible and powerful. Also it doesn’t chew memory just to launch – it’s a lean mean recording machine.

It is Open Source Free Software, and supports recording to OGG (for lossy compressed files) and FLAC (non-lossy compressed files), both open standards that any software can read (if it wants to). There’s no patent tax on the software makers to pass down to their users.

If you want to record video at the same time as the audio, you can try Open Broadcaster Software which is released for Linux, Mac, and Windows which can record desktop, multiple webcams, and audio as required; also open source free.

To mix the video portions you could check out Blender which is also multi-platform, including BSD, open source free as well.

If you are into electronic music and trackers, there’s LMMS, which I have not tried but may eventually come to explore if I get back into recording with trackers more frequently.

Conclusion

I don’t think I could have done this easily on commodity hardware without Linux and Open Source Free Software in general. To do even simple home recording, I expect without FOSS, I would have had to shell out for a proper system – or stuck to tapes and hi-fis.

And Allan’s wish for this to be sung at a BSD con may yet come true; what I hope to have done is laid down a reference version to build upon. It would be fun to see an instrumental team get together to record a version, or do a live take at a meetup, with Allan doing his part too…

Or at least a Jupiter Broadcasting version, recorded on Noah and Chris’s pro-grade equipment – if they can put aside their other allegiance for the sake of a song 😉

Posted on April 10, 2016 by Tai Kedzierski, Reply

About That: Linux Market Share

“Linux is free, nobody sells it – so there’s no market share!” Often I see this rebuttal in comment threads when someone calls for more market share for desktop Linux (yes, I read comment threads, I know it’s not healthy for me).

I want to briefly address here what is meant by “market share,” because Linux does have a market share in the world IT market, and it’s very significant; even servers running gratis Linux deployments are an integral part of that market share.

Rather than looking at market share as “how much Linux can we sell,” we need to look at it in terms of “how many Linux users are there on servers and desktops, and how can we sell software and services to them?”

“Market Share” does not mean “how much is sold” but “how much could be sold to.” Read more

Posted on March 12, 2016 by Tai Kedzierski, Reply

About That: Linux Mint’s site hack

The Linux Mint site hack a few weeks ago has brought to the fore how lackadaisical the security efforts behind some projects’ hosting and distribution sites may be. The truth of the matter is though, without a dedicated resource to look after this aspect, any effort can quickly grow stale and obsolete.

The tools and workflows required to keep sites and software packages secure are moving targets and a full-time effort; and the demand for latest-and-greatest software does not help one bit as a culture of blind trust has washed in on the back of the false mantra “Linux is inherently more secure.”

No it is not, and its growing popularity is demonstrating this. Linux is set up so that you can look more easily into your security and manage it, but security does not come without at least some planning and consideration. Jumping to BSD will not save us either. Improving our tooling and workflows is the only viable, forward-looking strategy we have at the moment – and it’s lacking.

Matt Hartley’s synopsis of the event is worth a read; the following are a copy of my initial reactions on his article. Read more

Posted on March 9, 2016 by Tai Kedzierski, Reply

Panic Button – EFI woes

Thank you to all at Linux Unplugged who chimed in to give their advice, as well as Rod Smith on SuperUser who provided an extensive, spot-on answer even given my mostly incorrect description and assumptions.

I’ll admit that I panicked when I sent the piece in, as I might have just turned a friend’s PC into a brick in the very first hour we bought it! Luckily he himself is already comfortable using Linux, just he hadn’t installed it in a very long time, so had the patience and the Windows-aversion to bear with me.

It turns out, this had nothing to do with Secure Boot, and after this saga, I have a slightly better understanding of what Secure Boot is, what scenarios it’s meant to apply in, and what went wrong…

Posted on January 25, 2016 by Tai Kedzierski, Reply

SSH fingerprint

Warnings about changed SSH host identities should be taken seriously – Man-In-The-Middle attacks are whre an impoersonator gets between you and your destination. They can sniff your traffic and even gain access codes and passwords, and even control of your computers.

When you ssh to a server, SSH checks the fingerprint returned by the server against what you have in your .ssh/known_hosts file

If when you SSH you get a big warning the identity of the server has changed, you may want to investigate…

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
5e:6f:20:b5:06:c1:3e:a7:7b:55:a6:9c:be:dc:79:24.
Please contact your system administrator.

If you have no other means of getting to the server, get a different computer, on a different network (for example, different laptop on your tethered dataplan, just get onto a different network!), and try again. It’s important to note that the new laptop/computer needs to also have previously contacted the remote server, otherwise it will simply prompt you to merrily add the remote server. Check the fingerprint to see how it matches up. If it’s the same that you are getting a warning about DO NOT ADD IT.

If you have direct console access to the affected server (VNC console given to you via your provider, or a physical terminal), you can run

ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key

This will print the fingerprint for the key.

If this is the same as you are being given a warning about, it’s possibly safe to assume it’s the real server, and your known_hosts file is wrong. Try turning ssh off and using ssh to connect – if you get a timeout or connection refused, instead of the big mean error, you might — MIGHT — be safe.

If however the fingerprint returned is different from what you see, you may want to check the other identity files:

for x in $(ls /etc/ssh/*key*); do echo -n "$x : " ; ssh-keygen -lf "$x" ; done

This will list the fingerprints of all the key files in that directory. If none of them match the fingerprint you are warned about, you are likely not connecting to the server you intended to – attack or mis-routing, you should consider the route to host unsafe – and that SSH is blocking you for your protection..

Posted on November 11, 2015 by Tai Kedzierski, Reply

Terminal escape characters (‘^]’ , etc.)

If you are trying to troubleshoot a connection issue, you have probably used the telnet tool. Telnet is an old socket protocol which, for the intents of our explanation, simply opens a network socket to the other server and passes data through plain.

For example, if you want to see if a SMTP server is running on a remote server 1.2.3.4 you would telnet to it on port 25:

telnet 1.2.3.4 25

If you wanted to check if an FTP server was running, you would instead run it against port 21, and so on – lookup different protocols online and find out their “default port” for “plain text” traffic (note that sending passwords via plaintext is a bad idea in general – but some services still allow you to do it. Tsk tsk.)

On Mac OS X, Linux and BSD, when you launch telnet, your session will probably open with the following statement

tai@demoserver:~/$ telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
220 demoserver.local ESMTP Postfix (Ubuntu)

The topic of this article is: what’s that escape character??

You can in fact come out of the telnet stream. In the above example, if you simply typed the normal interrupt ( Ctrl+D ) it would just send the interrupt byte along the wire – and not get caught by the application.

To do that we need to first invoke the escape sequence. So how to type the escape character?

There are two sequences needed to type it:

( Ctrl + “v” ) this causes the input to wait for a special character

( Ctrl + “]” ) this provides the special character

This generates a single character, denoted as “^]”. Send it by pressing return. This returns you to the local telnet prompt. You can now issue a ( Ctrl + D ) command to exit.

This technique also works elsewhere.

For example, you can display text in colour:

$> echo “^[[1;31mhello^[[0m”

Where the ” ^[ ” sequence is actually a special sequence as described above. Note this is using “[” and not “]”

^[ — special character for output stream control

[ — formatting code follows

1 — bold true (can be “0” to turn off bold)

;31 — red ; 32 is green, 33 yello, 34 blue. Try other combinations.

m — end modification

Text entry in this form also works when editing text in vi for example – when the resulting file is output via `cat` or `less -R`, you get colours and bolds!

Note that unless you include the code to turn off custom colours (^[[0m = “terminal default”) then the rest of your command line will keep the last selected colour mode.

Posted on November 10, 2015 by Tai Kedzierski, Reply

Solving “Permission denied” when using ‘locate’

On some Linux machines you might encounter a permissions error when trying to use locate as a regular user:

locate: can not open `/var/lib/mlocate/mlocate.db': Permission denied

I’m not entirely sure when this comes about, but it is the case on a number of AWS CentOS 6 machines.

The reason is multi-fold, and the following commands, run as root, enable the ability to use locate as a regular user. You need the database’s directory to b readable and executable globally, and you need the locate command to be executable with SGID.

chown root:slocate /usr/bin/locate
chmod g+x /usr/bin/locate
chmod g+s /usr/bin/locate
chmod a+rx /var/lib/mlocate

Posted on July 20, 2015 by Tai Kedzierski, Reply

Install a secure web server on Linux

Setting up secure connection on your Apache web server is very much straightforward on Linux — all the tools are at your disposal, and in just a few commands, you can be fully set up.

The following instructions are for Ubuntu and CentOS, and covers generating a self-signed certificate.

For an overview of free and cheap SSL certificates, see http://webdesign.about.com/od/ssl/tp/cheapest-ssl-certificates.htm. These certificates from Certificate Authorities only certify that the certificate was issued to the same person controlling the domain. They are fine for internal sites and personal home pages, but not for eCommerce sites..

For an overview of Enhanced Validation certificates (more expensive but more globally trusted), see http://webdesign.about.com/od/ssl/tp/cheapest-ev-ssl-certificates.htm. These certificates are issued against a real-world check of your identity, carrying thus a higher cost and higher trust. They are suitable for high-traffic sites that want to be properly identified and commercial sites; they are overkill for small project sites and testing. Read more

Posted on June 21, 2015 by Tai Kedzierski, Reply

Scrambling for notepads