Today I Learned

hashrocket A Hashrocket project

301 posts about #command-line surprise

Time terminal commands

If you enter time before any terminal command, it will output some information about how long the task took to run.

For example, let's run it on these two, crude, simple ruby scripts to see the time difference

# ./million.rb
1_000_000.times do
  2 + 2
end
> time ruby million.rb
ruby million.rb  0.05s user 0.02s system 47% cpu 0.143 total

​

# ./billion.rb
1_000_000_000.times do
  2 + 2
end
> time ruby billion.rb
ruby billion.rb  18.41s user 0.13s system 99% cpu 18.618 total

H/T Matt Polito

Delete to beginning of current word

If you are typing something in your shell and you need to delete the entire word you're on, instead of pressing backspace repeatedly, you can press ESC + Backspace and this will delete back to the start of the current word you are on.

Example; Say you wanted to delete this whole filepath, or a majority of it. Simply place your cursor at the end (or wherever you want to delete up to) and press ESC + Backspace to delete all the way back to the word vim

vim /i/accidentally/typed/something/big/that/was/incorrect

and it might leave you with something like this

vim /that/was/incorrect

Use comm to Compare Files

Today I learned about comm, which is used to select the common lines in two files. It's pretty neat, but has a strange output format.

Say we have two text files:

# first.txt
one
two
three

# second.txt
one
three
four

We can run the below command to find the common lines. In the output, the first column is what's only in first.txt, the second column is what's in second.txt, and the third column is what's common.

$ comm first.txt second.txt                                                                                         
                one
        three
        four
two
three

Hmm, that doesn't look right - one and three are common, not just one. The caveat with comm is that the files need to be sorted lexically. You can sort easily in bash with bird beak notation for process substitution.

$ comm <(sort first.txt) <(sort second.txt)
        four
                one
                three
two

If we only want the common lines, we can apply the flags -12 to hide the first and second columns:

$ comm -12 <(sort first.txt) <(sort second.txt)
one
three

Silence output of command in Makefile

Have you ever wanted to keep your makefile output a bit tidier? The @ symbol is your secret weapon.

Think of it like a silencer for your makefile. Every time you run a command, the makefile helpfully echos it back to you. But that echo gets silenced with the @ symbol at the beginning of a line.

This can be handy for keeping things clean, especially when you have a long list of commands in your makefile. Imagine a recipe with a million ingredients – you only care about the final dish, not every single step along the way, right?

Here's an example:

server: # Runs rails server
  @RAILS_LOG_LEVEL=debug bin/rails server

See how that works? The command executed for make server runs silently in the background.

Now, remember, this doesn't mean errors magically disappear. If something goes wrong, the error message will still show up. But for everything else, it's like a behind-the-scenes operation, keeping your makefile output focused on the important stuff.

So next time you want to streamline your makefile output, grab the @ symbol and hit the mute button on those noisy commands!

Set Env Vars with Shell Scripts

Sometimes I have limited screen real estate in my terminal, and my normal prompt of current/working/directory (git_branch) % takes up too much space. I wanted a bash script that could change my prompt to something short like & in one quick command. So I wrote a shell script:

#!/bin/bash

PS1="\[\e[32m\]& \[\e[m\]"

Nice and simple, right? 

~/src/dotfiles (main) % ./shorter.sh
~/src/dotfiles (main) %

Except, it doesn't change anything 😱. That's because changing PS1 isn't executing a command, it's setting an environment variable. So, just executing this shell script isn't enough, we need to source it to source the new PS1 in this terminal.* 

~/src/dotfiles (main) % . ./shorter.sh
&

* This is also why changing the prompt like this only affects the current terminal, and not any others that you may have open at the same time.

List Installed Fonts via the Command Line

Today I learned there's a command line utility called fc-list. It lists the fonts installed on your system.

Running fc-list will print out a lot of information - font families, the different styles available, where they're installed.

I find it's much more useful to run fc-list : family, which will print out all the font family names installed:

$ fc-list : family

Fira Code,Fira Code SemiBold
0xProto Nerd Font
Iosevka Term,Iosevka Term Extrabold
Menlo
.SF NS Mono
...

This is a lot easier to read and grep through!

My main use case for this is if I want to use a font in my alacritty config, but don't know the font's exact name. For example, if I want to use Apple's new(-ish) SF Mono font, the font family is not SF Mono as you might expect - its clearly .SF NS Mono 🤷.

Autofill last argument from previous commands

Have you ever found yourself typing out long commands in the terminal, only to need a part of that command in a subsequent command? You can save time by reusing the last argument of the previous command using the ESC key followed by a period .

Here's an example:

First, you use cat to display the contents of the file example.txt:

cat path/to/your/files/example.txt

Now, if you want to perform an operation on the same file, you don't need to type out the entire path again. Instead, use the ESC key followed by the period .

vim ESC. 

# After pressing ESC. this becomes: 
vim path/to/your/files/example.txt

In this example, ESC. automatically inserts the last argument from the previous command, which is the file path path/to/your/files/example.txt.

Note: You can use ESC . repeatedly to cycle through previous arguments in your command history.

Command Line Wildcards

The * character can be used as a wildcard to match sequences of unknown characters in the command line.

For example, lets say my elixir project has a few tests that I want to run in a directory: MyProject/tests. The folder is filled with a bunch of random files, but the ones that i want to run have a similar name, tests/user_views_home and tests/user_views_show. We could use a wild card to match on both of these file names and run the tests (assuming there are no other files that match) like this:

mix test MyProject/tests/user_views*

The Word Count Command

By using the wc command, you can print out word count information to the terminal. By default, if you use the wc command along with a file path, the command will return 3 values: the line count, word count, and character count of that file.

You can also pipe the wc command into any other terminal command to receive the word count information about the previous command. For instance, you could use the command ls | wc to see the word count info for the current directory's file list.

If you want the wc command to only output either the line count, word count, or char count, you can pass it the following flags: wc -l for line count, wc -w for word count, and wc -c for the char count.

Import Github User Public SSH Keys

You can use this command to add keys to the current user authorized_keys file. This command works for public keyservers, but in my case, I used it for Github. Handy when setting up a new machine or adding a new user's keys to a system. 

# Example - ssh-import-id gh:GITHUB_USERNAME

ssh-import-id gh:avogel3

If the PROTOCOL (gh above) portion is absent, it defaults to lp. Acceptable values are lp = launchpad and gh = github. You can also configure this command to handle a specific URL when not specifying a protocol.

https://manpages.ubuntu.com/manpages/xenial/man1/ssh-import-id.1.html

Find meta information about Github's services

If you need info about github like what their current SSH fingerprints are or IP address ranges of their services... it turns out that much of that is readily available via the meta information api endpoint

Going through the docs makes it seem like you need to do things like pesky authentication but you can just go right to https://api.github.com/meta and you'll find most of the info you're looking for.

image

Remove SSH keys from known host file by IP address

Github had a situation which caused the need to potentially remove Github's keys from your known_hosts file.

This can be done easily with the SSH command:

ssh-keygen -R github.com

This is great... except when it isn't. I specifically ran into an issue where my connection was trying to utilize specific github IP addresses. Have no fear... it turns out that the same command can be utilized.

ssh-keygen -R 140.82.114.4

I actually had to do a few of them

ssh-keygen -R 140.82.113.3

etc...

You could potentially use github's meta information endpoint to find address ranges, but that's a problem for a more clever person.

Parsing nested string json

Today I learned how to parse a nested json with jq, but the nested json is a string. It's just easier to show an example so here we are:

{
  "a": "{\"b\": \"c\"}"
}

This is not a common situation but I found that out today on a codebase and my first thought was to call jq get the content of the node a and then pipe it into another jq command. It would look like this:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq '.a' | jq
# => "{\"b\": \"c\"}"

As we can see the result is not a json, but a string so we cannot access inner nodes just yet.

And the solution to this problem is to use the -r flag on the first jq call to output the result in a raw format, so the " surounding double quotes will disappear. And with that in place we can easily parse the nested/nasty json:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq
# => {
# =>   "b": "c"
# => }

Then finally:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq '.b'
# => "c"

How to Change Password of SSH key

It's possible to change the password of your current ssh key if you have the current password or it is not currently password protected. You can use the command: 

ssh-keygen -p

From the man pages -

Requests changing the passphrase of a private key file instead of
creating a new private key.  The program will prompt for the file
containing the private key, for the old passphrase, and twice for
the new passphrase.

https://man.openbsd.org/ssh-keygen.1#p

Group Json data by a key

Today I learned how to group by json data by a key using jq. In ruby that's very trivial, it's just about using the group_by method like that:

[
  {name: "John", age: 35},
  {name: "Bob", age: 40},
  {name: "Wally", age: 35}
].group_by{|u| u[:age]}

# {
#   35=>[{:name=>"John", :age=>35}, {:name=>"Wally", :age=>35}],
#   40=>[{:name=>"Bob", :age=>40}]
# }

But using jq I had to break it down to a few steps. So let's say that I have this json:

[
  {"name": "John", "age": 35},
  {"name": "Bob", "age": 40},
  {"name": "Wally", "age": 35}
]

The idea is that we'll call the group_by(.age)[] function to return multiple groups, then I pipe it to create a map with the age as the key. Finally we'll have these bunch of nodes not surounded by an array yet, so I am pipeing to a new jq command to add with slurp:

cat data.json |
  jq 'group_by(.age)[] | {(.[0].age | tostring): [.[] | .]}' |
  jq -s add;

# {
#   "35": [{"name": "John", "age": 35},{"name": "Wally", "age": 35}],
#   "40": [{"name": "Bob", "age": 40}]
# }

Upgrade Heroku PostgreSQL

We just did a PostgreSQL bump in Heroku from 13.8 => 14.5 (the latest Heroku supports at this day). The process was very smooth and kind of quick for a 1GB database. Here's the script we end up running:

# Change the following `basic` to the right plan for you
heroku addons:create heroku-postgresql:basic -r heroku-staging
heroku pg:wait -r heroku-staging
heroku pg:info -r heroku-staging
# Now grab the NEW and OLD URLS to change the following commands:

heroku maintenance:on -r heroku-staging
# It took less than 2 mins for a 1GB database
heroku pg:copy DATABASE_URL CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
# It's usually fast, it depends on how long the app takes to reboot
heroku pg:promote CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
heroku maintenance:off -r heroku-staging

heroku addons:destroy CHANGE_HERE_OLDCOLOR_URL -r heroku-staging

curl with a progress bar

You can download files with a nice progress bar using curl's -# flag:

curl -# -O https://files.example.com/large/long_video.mp4
#################                               38.6%

This might be preferable to the verbose output:

curl --no-progress-meter -O https://files.example.com/large/long_video.mp4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 17  433M   17 75.6M    0     0  28.7M      0  0:00:15  0:00:02  0:00:13 28.8M

Preview ffmpeg video filters without re-encoding

I was playing around with some ffmpeg filters, like cropping, scaling and overlays and I was tired of waiting for the video to be fully re-encoded in order to see the changes.

ffmpeg -i video.mp4 -vf "crop=in_w:in_h/2:0:0" -c:a copy output.mp4

I'm glad that this is not a problem because you can use ffplay to preview the changes instantly without having to wait:

ffplay -i video.mp4 -vf "crop=in_w:in_h/2:0:0"

What directory is the parent of root 👨‍👦📁

I learned that in unix, root (e.g. /) is the only directory that is the parent directory of itself.

$ ls -lai / | grep '\./'
                  2 drwxr-xr-x   20 root  wheel   640 Jan  1  2020 ./
                  2 drwxr-xr-x   20 root  wheel   640 Jan  1  2020 ../

In the above example, the files . and .. both have the same i-node: 2

Source: Brian W. Kernighan, & Rob Pike (1984) The UNIX Programming Environment. Prentice-Hall, Inc

Kill a Program with pkill

I have a cronjob to open macOS's Photo Booth every weekday so I can take a picture of my work life. Unfortunately, it opens the program every weekday; I'd rather it quickly closes itself if I'm not there or otherwise occupied. Today I used pkill in the cronjob to terminate the program five minutes after opening:

20 9 * * 1-5  pkill "Photo Booth"

pkill kills a process by name. You can figure out how to make pkill effective using pgrep, a companion program that searches for running processes by name. Using it, I learned that the string "Photo Booth" was specific enough to find and kill Photo Booth:

$ pgrep -l "Photo"
292 Photo Booth

"Photo Booth", PID 292 (today), is the process I programmatically kill every weekday at 9:20 AM.

Tmux Clear Server Pane

Here's a situation: you're watching a server log in Tmux, about to trigger an action that will produce log data you care about. You hit return a bunch of times to create a visual break in the server log. Then you can scroll up and see the beginning of your revelant history.

What actually happens? Sometimes, the server logs tons and tons of information, and your visual break gets buried way above the fold. It's hard to find the break, and you're searching through all that information, plus anything that happened before.

There's a better way! Tmux's clear-history command "removes and frees the history of the specified pane." In the Hashrocket Dotmatrix, we combine that with send keys -R, which "causes the terminal state to be reset." Here's the mapping:

 bind-key C-k send-keys -R \; clear-history

Type the Tmux leader, then C-k, and your terminal pane will be visually cleared and cleared of its history, making reading and reverse searching much easier.

h/t Gabe Reis

How to convert JSON to CSV with jq

I had this json file that was an array of objects and some of those objects had different keys. I wanted to visualize that data in a spreadsheet (data science, AI, machine learning stuff) so I thought about having a CSV file where each JSON key would become a CSV column.

// file.json
[
  {
    "type": "Event1",
    "time": 20
  },
  {
    "type": "Event2",
    "distance": 100
  }
]

jq to the rescue:


cat file.json | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' > file.csv

Here is the output:

"distance","time","type"
,20,"Event1"
100,,"Event2"
Screen Shot 2020-08-07 at 4 39 16 PM

Tmux Send Keys to Pane

I wrote a script the other day designed to help me download and edit files faster. In part of the script, I wanted to open Vim in an existing Tmux pane, and in the process I learned about the tmux send-keys command. Here's how it works:

$ tmux send-keys -t 3 "vim" Enter

send-keys, aliased send, sends your string of commands to your pane (t) of choice. Running the above opens Vim in pane #3.

The -t flag accepts negative numbers, too, like an array index. In my version of the above command, I send the keys to pane -1, the last pane on the screen, which is where I keep Vim in my Tmux session.

zsh comes with help (set the $HELPDIR)

zsh helpfully comes installed with help files for all the builtins and a run-help command to help you access those help files. There is a trick though, before setting any environment variables here's what happens:

$ run-help
There is no list of special help topics available at this time.

This is because the HELPDIR isn't set. You have to find the install location for zsh's help files and set the env var to that dir. On my system that looks like this:

export HELPDIR='/usr/share/zsh/help'

Then when you run run-help you should see a list of builtins for which there is help documentation. This is the same documentation that you can get via man builtins but much more readable and discoverable. run-help will call man as well if it can't find you're arg in the help files.

For me run-help is cumbersome to type so I alias it. Here's what goes into my .zshrc:

export HELPDIR='/usr/share/zsh/help'
alias help=run-help

zsh is now much more helpful!

zsh comes with Tetris

zsh comes with it's very own tetris game. No plugins needed!

You do need to autoload the tetriscurses function:

autoload -Uz tetriscurses

And then run tetriscurses.

While in the game, you can press H to learn which keys do what, and that looks like this:

left: h, j, left
right: right, n, l
rotate: up, c, i
soft drop: down, t, k
hard drop: space
quit: q
press space to return

also, maybe you want an alias for this?

autoload -Uz tetriscurses
alias tetris=tetriscurses

I'm putting the above straight into my .zshrc! Happy Sunday!

Follow the link in linux with `readlink -e`

Sometimes linux can be a maze of symbolic links. On my system, the java command exists at /usr/bin/java which is a link that points to /etc/alternatives/java which is a link that points to /usr/lib/jvm/java-8-oracle/jre/bin/java.

Instead of looking up each of the links of these files with ls -l, readlink -e will the links all the way through to the eventual file. In my case that would look like this:

$ readlink -e `which java`
# returns /usr/lib/jvm/java-8-oracle/jre/bin/java

You can learn more with man readlink.

On Mac, there is a readlink command, but there is no -e flag and it is not recursive.

Global Alias in zsh

What makes an alias global? Well, the -g flag of course. And what does this globality give you? Well, the ability to invoke an alias anywhere in the command line.

If I like the word 'Potateos' but I don't ever have the energy to type the whole thing then I can create a global alias for that word:

> alias -g PO="Potatoes"
> echo PO
Potatoes

That's convenient and cool. What is it actually for? Maybe redirecting errors to /dev/null:

> alias -g NO='2> /dev/null'
> echo foo >> /dev/stderr
foo
> (echo foo >> /dev/stderr) NO
# no output, it got swallowed!

Looks weird and maybe not useful, but maybe you can creatively find a useful way to use it:

I learned about this zsh functionality and other functionality here.

Shortcuts with hash -d in zsh

I stumbled across this zsh tricks post yesterday and am blown away by the hash command, which allows you to see and manipulate the hash table for either commands or for directory shortcuts.

hash by itself in zsh will output the location for all the commands.

hash -d shows you all of the named directories, and check this out you can navigate to one of those directories with ~shortcutname, like this:

$ hash -d | grep bin
bin=/bin
daemon=/usr/sbin
proxy=/bin
sync=/bin
$ cd ~daemon
$ pwd
/usr/sbin

You can create your own directory shortcuts like this:

$ hash -d mydir=/home/me/very/long/path

And then cd to it:

$ cd ~mydir
$ pwd
/home/me/very/long/path

Crazy! Read more in the zsh docs.

List Files by Updated

I'm currently working on an app that forwards logging around to various locations on the Linux server. It's a bit tricky for me to figure out where any action I take in the browser is being logged. I need those logs!

A nice way to figure out where the logging is happening is to narrow it down to one directory (say, /var/log/) then ls that directory, ordering by most recently updated. The items at the top of the list have been recently updated, and thus probably contain valuable loggings!

$ ls -lt

I'm throwing on the -l flag for more detail. If there are lot of logs, filter it down with head:

$ ls -lt | head

Thanks for the idea, Kori!

The three amigos of the current directory

I always have trouble remembering how to get the name of the current directory. So strange pneumonics is the way to go.

The first amigo is a shell variable:

echo $PWD
# returns '/home/chris/tils'

There is also a pwd command that returns the same thing.

The second amigo is basename which gives you the current directory name without its path:

basename $PWD
# returns 'tils'

The third amigo is dirname which gives you the path without the current directory name:

dirname $PWD
# returns '/home/chris'

So now I can do things like

alias tnew=tmux new -s $(basename $PWD)

because I always, always, name my tmux session after the name of the current directory.

DNS Lookup with host

Today while doing some sleuthing, I learned about the host command. host "is a simple utility for performing DNS lookups." It helped me connect a series of domains to their respective AWS EC2 servers, without a visit to the domain registrar.

Example:

$ host jakeworth.com
jakeworth.com has address 184.168.131.241
jakeworth.com mail is handled by 10 mailstore1.secureserver.net.
jakeworth.com mail is handled by 0 smtp.secureserver.net.

More info: man host

Get actual file size with du on Linux

You can use du, to report on the size of directories or files, but when my file is smaller than the block size I don't see the output I expect.

With a small file, this should be the size of the number of characters.

$ echo 'Every Good Boy Deserves Fudge' > staff.txt
$ cat staff.txt | wc -c
30

But when I use du to examine file, I don't see 30.

$ du -h staff.txt
4.0K    staff.txt

du measures in block sizes because in general if any part of a block is used, then for the purposes of the operating system the entire block is used.

You can tell du to care only about the size of the file with --apparent-size which is only apparent because between the beginning and end of the file the OS can't tell which bytes are in use or are not in use.

$ du --apparent-size staff.txt
1       staff.txt

When reporting apparent size it rounds up to kilobytes, or --block-size=1k

To get the actual size of the file, you can use -b which is the same as --apparent-size --block-size=1

$ du -b staff.txt
30      staff.txt

Object construction with jq

jq is a powerful command-line tool to help you parse, analyze and script json output.

My current problem in jq is to turn this:

{
  "modules": [
  {
    "name": "x",
    "size": 10
  },
  {
    "name": "y",
    "size": 20
  }
  ]
}

into this:

{x: 10}
{y: 20}

This is possible using object construction:

jq '.modules[] | {(.name): .size}'

You pipe the result of the initial attribute as an array syntax .modules[] to an object {}. To use an attribute as a key you put parens around the attribute (.name) and declare that the value should be a different attribute .size.

Read more in the jq docs