Get faster in bash

August 01, 2014

Bourne-again Shell is definitely the most widely used and de-facto standard shell in many UNIX-like systems. Even though there are much more feature-packed alternatives (like zsh) many people use shell only casually or just think it's too much hassle to install and learn other shells which are not usually available on other hosts anyway.

So, let's not write off bash(1) completely and let's see how we can improve its user experience at least a bit.

I'm not going to talk about basic stuff like aliases, functions or globbing. Instead I want to sum the biggest slowdowns that even every-day users could encounter in bash(1) and how I went about to overcome them.

Note: These tips are only about plain bash(1) and readline(3). For my tips on improving your efficiency with external tools, read my post Improving your command line workflow (to be done).

History repeating

This first notable slowdown I'll address is handling the history. You must feel that pressing the up-arrow until you find previous command (only to change some argument or typo) isn't the fastest way you can take.

bash(1) lets you manipulate and even edit the history in ways which can help you speed-up your workflow a great deal if you learn to use it right.

Now let's say you want to run again that horribly long command from yesterday. There are several ways. First is the "classical" up-arrow approach. What are the others?

Let's see the history first:

$ history 30    # print last 30 commands
   ...
  48 ffmpeg -ss 80 -vcodec mpeg4 -sameq -acodec copy -i orig.avi -i orig.mp3 output.avi
  49 history 30
  ...
$

You can re-run the ffmpeg command multiple ways:

$ !48        # run the history line 48 (you can put history number in your prompt: export PS1="\! $PS1"
$ !ffmpeg    # run the last command that starts with 'ffmpeg'
$ !?vcodec   # run the last command that contains 'vcodec'

These are called event designators and they are pointers to certain lines of history. On top of this there are also word designators, which narrow the selection to specific argument (or range of arguments) of given line and finally there are modifiers, which can modify the selected objects.

I don't want get into too much details (see man bash) so let's just demonstrate:

$ !ffmpeg
ffmpeg -ss 80 -vcodec mpeg4 -sameq -acodec copy -i orig.avi -i orig.mp3 output.avi
ffmpeg: cannot access orig.avi: No such file or directory
$ rm !$                           # '!$' means last argument of last command
rm output.avi
$ !ff:s/mpeg4/copy/:s/avi/mp4/    # modifiers can be combined
ffmpeg -ss 80 -vcodec copy -sameq -acodec copy -i orig.mp4 -i orig.mp3 output.avi
$ !ff:3-$                         # word designator can specify range of arguments
ffmpeg -vcodec copy -sameq -acodec copy -i orig.mp4 -i orig.mp3 output.avi
$

Correcting a typo from previous command is so frequent that the shortcut was defined:

$ cp /usr/share/doc/vrey/long/name/file-name.txt /tmp
cp: cannot stat '/usr/share/doc/vrey/long/name/file-name.txt': No such file or directory
$ ^vrey^very^       # substitute for !!:s/vrey/very/
cp /usr/share/doc/very/long/name/file-name.txt /tmp
$

Another example shows event designator '!#' which points to current line. Word designator '1' narrows the selection to the first argument and modifier 'h' removes trailing file name component.

$ cp /usr/share/doc/very/long/name/file-name.txt !#:1:h/renamed.txt
cp /usr/share/doc/very/long/name/file-name.txt /usr/share/doc/very/long/name/renamed.txt


$ cd !$:h
cd /usr/share/doc/very/long/name/
$

Grepping the history can be simplified with alias:

$ alias gh="nl $HISTFILE | grep"

Another way to re-use & modify history is to use fc built-in (means fix command).

$ fc -l -16

works the same way as

$ history 16

but you can also do

$ fc -5

which will put last 5 lines of history to your editor (set with $EDITOR variable). The lines will be executed after you exit the editor.

You can also use

$ fc start_line_number [end_line_number]

Some useful settings:

Command	Effect
`HISTCONTROL=ignoredups`	Don't save consecutive duplicate lines to history.
`HISTIGNORE='ls:history'`	List of lines to be ignored. Note that only 'ls' itself will be removed.
	You could use 'ls*' to ignore any command starting with ls.
`shopt -s histappend`	By default, the history is overwritten at the end each session. Therefore only the last standing wins.
`PROMPT_COMMAND="history -a"	With this settings it will be updated after each command.
`shopt -s histreedit`	If the history substitution failed it is automatically re-inserted to your current command line buffer.
`shopt -s histverify`	Don't run the substituted command right away, only put it to your current command line buffer.
`bind space:magic-space`	Perform history expansion on the current line and inserts space (very useful, try it!). Although it's a `readline(3)` variable, this only works in `bash(1)`.
`shopt -s cmdhist`	Save multiple-line commands as one line. This is useful for re-editing functions or longer commands.
`shopt -s autocd`	Type only the name of the directory (without `cd`) to enter it. This command was inspired by zsh. bash(1)>=v4.x
`shopt -s cdspell`	Fix minor spelling mistakes when changing to directory.

Readline history

GNU Readline is a library that provides line-editing and history features not only to bash(1). Any program that makes use of this library can take advantage of features that are mentioned here. Even the programs that doesn't use readline(3) library can be wrapped with rlwrap. This awesome application saved me lots of times and nerves.

Let's take a look at the history features first to seamlessly append on previous chapter.

Another way of recalling history is interactive search. Most of your probably know that you can press C-r (reverse-i-search) and type a part of command you want to re-run. Pressing C-r multiple times iterates through all matching from newest to oldest, C-s goes the other direction.

Note: To get C-s working you need to turn of STOP/START terminal control with stty -ixon -ixoff. This ancient option is still enabled by default on all major distributions for some unknown reason. Now you can also map i.e. C-s in vim to save!

Another useful shortcut is M-. (action yank-last-arg) which iterates through last arguments of previous commands:

 $ ls -l some_file.txt
 $ ls -l other_file.txt
 $ rm [`M-.`]
 $ rm other_file.txt[`M-.`]
 $ rm some_file.txt
 $

This is much more snappy than using event designators.

Moving faster

Okay, maybe this should be the slowdown number one, but I wanted to introduce readline(3) first. Using only left and right arrows to navigate around current line is ineffective and tediously boring.

readline(3) provides lots of shortcuts for jumping over words, deleting words, discarding the beginning/end of the line/word, etc.

Here are the most useful shortcuts which can do wonders to your speed:

Shortcut	Readline action	Description
`C-a`	`beginning-of-line`	Jump to the beginning of line (just as Home)
`C-e`	`end-of-line`	Jump to the end of line
`M-f`	`forward-word`	Move forward a word
`M-b`	`backward-word`	Move backward a word
`M-.`	`yank-last-arg`	Insert last argument of last command

There are also commands for cutting & pasting:

Shortcut	Readline action	Description
`C-w`	`unix-word-rubout`	Cut the word (and save to yank-ring)
`C-k`	`kill-line`	Cut the end of the line from cursor position
`C-u`	`unix-line-discard`	Cut the beginning of the line before cursor position
`C-y`	`yank`	Paste last-cut string
`M-y`	`yank-pop`	Cycle through yank-ring (after `C-y` was pressed)
unbound	`copy-backward-word`	Copy the word before cursor
unbound	`copy-forward-word`	Copy the word after cursor

These are great, but I find the shortcuts kinda outdated. Sure, it's great to have everything near your home row, but all GUI programs are pretty much consistent in movements over words - holding Control key + arrows and Control backspace for delete. Having different shortcut in shell only confuses me and slows me down, so I usually rebind it.

Mapping shortcuts with Control key can differ in systems and even in terminals. You might need some troubleshooting to get it working on your system. See /etc/inputrc (or include it from yours) to get you started. To map Control + Backspace I use following binding which works on xterm(1) and rxvt(1):

"\b": backward-kill-word

Note: All this applies to Emacs mode, which is default input mode of readline(3), but there's also vim mode with normal and insert mode which some people consider better.

This article is mainly about bash(1), so I will cut this here, but definitely take a time to learn (or define) at least some shortcuts if you want to be faster in shells. You can also bind a macro - just put the string (in quotes) after the semicolon:

"\C-xg": "\!!:gs///\C-b\C-b"

See this nifty cheatsheet for available shortcuts in Emacs mode. You can also find list of actions and default bindings in readline(3) manual page.

Completion

The insufficient completion features are the main reason why people migrate to zsh. But in my opinion it can get quite powerful as well when you configure it to match your needs.

Everyone knows, that you can complete command names, file names and environment variables. With programmable completions, you can do much more - completing options, hostnames on network commands, processes on kill and killall commands, git commands & refnames, packages on apt commands, extension-aware completion and much, much more.

New completion functions are still being added. You can find them at bash-completion.alioth.debian.org. Mostly all you need to is installing bash-completion package from your distribution's repository. If completion for your favorite command is missing, write it!

But let's face it, once you start to want more, it is time you switched to zsh(1) ;)

Lazy tab completion

Another minor annoyance is having to press tabulator twice if the completion is ambiguous. First tab rings the bell and second lists the options. I never understood how's that useful, but luckily there's a way to change it: Put this in your ~/.inputrc:

set show-all-if-ambiguous on

When there's lot of multiple choices (more than 100 by default), readline(3) presents you with a choice to display all possibilities. If you're working with big directories frequently, you can increase the number:

set completion-query-items 300

Don't set the number too high, though! When there are several hundreds of matches the shell can do annoying freezes.

Another useful option is case-insensitive completion:

set completion-ignore-case on

See readline(3) manual page all options.

`ls` after `cd`

This seems like it's not a big deal, but it is.

Analyzing my .bash_history confirmed, that ls follows almost every cd command. Try for yourself:

$ echo $(( $(grep -Pzo 'cd.*\nls$' .bash_history | wc -l) / 2 ))

I weren't even thinking about it, it was just automatic. Then why not list the directory automatically when you change into it?

zsh(1) provides hook for this. It executes chpwd command (if it exists) everytime the working directory changes.

Sadly, bash(1) doesn't have anything like that, but we can redefine the cd command by creating function;

function cd() {
    builtin cd "$@" && ls;
}

You can do the similar for pushd & popd commands, if you're using them frequently.

Bonus: Brace expansion

Brace expansion is used to generate arbitrary strings. How can that be useful? Let's see some examples. Remember the renaming of very long path from first chapter? We can do it with brace expansion as well:

$ cp /usr/share/doc/very/long/name/{file-name,renamed}.txt
$ mkdir /usr/share/doc/very/long/name/{create,several,directories,at,once}
$

But what makes brace expansion so powerful are sequences:

$ echo "Odd numbers: {1..100..2}
Odd numbers: 1 3 5 7 9 11 13 15 17 19 21    # etc.
$ for chess_field in {A..H}{1..8}; do ...; done
$

One downside is that the variables are expanded after the brace expansion. So you can't use variables to define ranges:

$ A="hello"; B="world"
$ echo test-{$A,$B}
test-hello test-world
$ # works, but
$ range=1..5
$ echo test{$range}
test{1..5}

There's a workaround for that, though! (thanks to reddit user Vaphell for pointing it out! )

$ eval echo test{$range}
test1 test2 test3 test4 test5

Conclusion

Whenever there's some discussion about tweaking the bash(1) you'll see several guys telling you you should use zsh(1) instead.

Yeah, zsh(1) is awesome and I use it too, but we shouldn't underestimate the good old bash(1) because it can do much more than it seems on the first glance.

If you're interested in my dotfiles, you can see them on github.com.