wc is well known for giving you word, line and character counts of a file or standard input.
Today I Learned wc can also tell you the length of the longest line in a file with the -L flag!
$ cat file.txt
one
two
three
$ wc -L file.txt
5 file.txt
Man pages are full of useful information and trivia. Did you know wc has been around since Unix Version 1?
I stumbled upon this earlier today, if you're in a terminal and have typed some things in, pressing Esc, then Backspace will delete the word on or before your cursor. Ctrl + W will do the same thing.
So if I start typing a word and want to
$ one two three four
^ cursor here
Then Esc, Backspace will produce
$ one two three
^ cursor hereIt deletes through the beginning of the word, but starts where the cursor is, so you can delete partial if your cursor is in the middle of a word
$ longer command here
^ cursor here
Then Esc, Backspace will produce
$ longer and here
^ cursor here
Confirmed both work in recent versions of bash and zsh.
If you pass the -p option to mkdir, it will create any necessary parent directories that do not exist yet. In Elixir the File.mkdir_p/1 function will do the same thing!
If you enter time before any terminal command, it will output some information about how long the task took to run.
For example, let's run it on these two, crude, simple ruby scripts to see the time difference
# ./million.rb
1_000_000.times do
2 + 2
end> time ruby million.rb
ruby million.rb 0.05s user 0.02s system 47% cpu 0.143 total
# ./billion.rb
1_000_000_000.times do
2 + 2
end> time ruby billion.rb
ruby billion.rb 18.41s user 0.13s system 99% cpu 18.618 totalH/T Matt Polito
If you are typing something in your shell and you need to delete the entire word you're on, instead of pressing backspace repeatedly, you can press ESC + Backspace and this will delete back to the start of the current word you are on.
Example; Say you wanted to delete this whole filepath, or a majority of it. Simply place your cursor at the end (or wherever you want to delete up to) and press ESC + Backspace to delete all the way back to the word vim
vim /i/accidentally/typed/something/big/that/was/incorrectand it might leave you with something like this
vim /that/was/incorrect
Today I learned about comm, which is used to select the common lines in two files. It's pretty neat, but has a strange output format.
Say we have two text files:
# first.txt
one
two
three
# second.txt
one
three
four
We can run the below command to find the common lines. In the output, the first column is what's only in first.txt, the second column is what's in second.txt, and the third column is what's common.
$ comm first.txt second.txt
one
three
four
two
three
Hmm, that doesn't look right - one and three are common, not just one. The caveat with comm is that the files need to be sorted lexically. You can sort easily in bash with bird beak notation for process substitution.
$ comm <(sort first.txt) <(sort second.txt)
four
one
three
two
If we only want the common lines, we can apply the flags -12 to hide the first and second columns:
$ comm -12 <(sort first.txt) <(sort second.txt)
one
three
Have you ever wanted to keep your makefile output a bit tidier? The @ symbol is your secret weapon.
Think of it like a silencer for your makefile. Every time you run a command, the makefile helpfully echos it back to you. But that echo gets silenced with the @ symbol at the beginning of a line.
This can be handy for keeping things clean, especially when you have a long list of commands in your makefile. Imagine a recipe with a million ingredients – you only care about the final dish, not every single step along the way, right?
Here's an example:
server: # Runs rails server
@RAILS_LOG_LEVEL=debug bin/rails server
See how that works? The command executed for make server runs silently in the background.
Now, remember, this doesn't mean errors magically disappear. If something goes wrong, the error message will still show up. But for everything else, it's like a behind-the-scenes operation, keeping your makefile output focused on the important stuff.
So next time you want to streamline your makefile output, grab the @ symbol and hit the mute button on those noisy commands!
I found out that I can add the - symbol in front of the include call in a Makefile, and that will check if the file exists before including, so it won't show any error if the file is not there. Here's an example:
-include .env
.PHONY: console
.env:
cp .env.example .env
console: ## Opens the App console.
iex -S mix
Sometimes I have limited screen real estate in my terminal, and my normal prompt of current/working/directory (git_branch) % takes up too much space. I wanted a bash script that could change my prompt to something short like & in one quick command. So I wrote a shell script:
#!/bin/bash
PS1="\[\e[32m\]& \[\e[m\]"Nice and simple, right?
~/src/dotfiles (main) % ./shorter.sh
~/src/dotfiles (main) %
Except, it doesn't change anything 😱. That's because changing PS1 isn't executing a command, it's setting an environment variable. So, just executing this shell script isn't enough, we need to source it to source the new PS1 in this terminal.*
~/src/dotfiles (main) % . ./shorter.sh
& * This is also why changing the prompt like this only affects the current terminal, and not any others that you may have open at the same time.
Today I learned there's a command line utility called fc-list. It lists the
fonts installed on your system.
Running fc-list will print out a lot of
information - font families, the different styles available, where they're
installed.
I find it's much more useful to run fc-list : family, which will print out all
the font family names installed:
$ fc-list : family
Fira Code,Fira Code SemiBold
0xProto Nerd Font
Iosevka Term,Iosevka Term Extrabold
Menlo
.SF NS Mono
...This is a lot easier to read and grep through!
My main use case for this is if I want to use a font in my
alacritty config, but don't know the font's exact name. For
example, if I want to use Apple's new(-ish) SF Mono font, the font family is not
SF Mono as you might expect - its clearly .SF NS Mono 🤷.
Pressing Ctrl+R on the terminal will bring up bck-i-search:
It will then search in reverse order for whatever commands you used based on what you type in.
To keep going back through commands based on the same search term, just press Ctrl+R again and it will go to the next one in the history.
To display an alphabetically-ordered list of all aliases currently set, simply type alias
> alias
gap='git add -p'
gc='git commit'
gco='git checkout'
# continued...
If you want to gracefully kill all of your tmux sessions/windows/panes in one fell swoop, run tmux kill-server
Use lsof -i :port_number to find details about a process listening on whichever port you pass.
For example, if you were running a local server on port 4000 and wanted to find the server's PID you could run lsof -i :4000
Have you ever found yourself typing out long commands in the terminal, only to need a part of that command in a subsequent command? You can save time by reusing the last argument of the previous command using the ESC key followed by a period .
First, you use cat to display the contents of the file example.txt:
cat path/to/your/files/example.txt
Now, if you want to perform an operation on the same file, you don't need to type out the entire path again. Instead, use the ESC key followed by the period .
vim ESC.
# After pressing ESC. this becomes:
vim path/to/your/files/example.txt
In this example, ESC. automatically inserts the last argument from the previous command, which is the file path path/to/your/files/example.txt.
Note: You can use
ESC .repeatedly to cycle through previous arguments in your command history.
use ctrl+a to jump the cursor to the beginning of the line, and ctrl+e to jump the cursor to the end of the line.
The * character can be used as a wildcard to match sequences of unknown characters in the command line.
For example, lets say my elixir project has a few tests that I want to run in a directory: MyProject/tests. The folder is filled with a bunch of random files, but the ones that i want to run have a similar name, tests/user_views_home and tests/user_views_show. We could use a wild card to match on both of these file names and run the tests (assuming there are no other files that match) like this:
mix test MyProject/tests/user_views*
By using the wc command, you can print out word count information to the terminal. By default, if you use the wc command along with a file path, the command will return 3 values: the line count, word count, and character count of that file.
You can also pipe the wc command into any other terminal command to receive the word count information about the previous command. For instance, you could use the command ls | wc to see the word count info for the current directory's file list.
If you want the wc command to only output either the line count, word count, or char count, you can pass it the following flags: wc -l for line count, wc -w for word count, and wc -c for the char count.
You can use this command to add keys to the current user authorized_keys file. This command works for public keyservers, but in my case, I used it for Github. Handy when setting up a new machine or adding a new user's keys to a system.
# Example - ssh-import-id gh:GITHUB_USERNAME
ssh-import-id gh:avogel3
If the PROTOCOL (gh above) portion is absent, it defaults to lp. Acceptable values are lp = launchpad and gh = github. You can also configure this command to handle a specific URL when not specifying a protocol.
https://manpages.ubuntu.com/manpages/xenial/man1/ssh-import-id.1.html
If you need info about github like what their current SSH fingerprints are or IP address ranges of their services... it turns out that much of that is readily available via the meta information api endpoint
Going through the docs makes it seem like you need to do things like pesky authentication but you can just go right to https://api.github.com/meta and you'll find most of the info you're looking for.
Github had a situation which caused the need to potentially remove Github's keys from your known_hosts file.
This can be done easily with the SSH command:
ssh-keygen -R github.comThis is great... except when it isn't. I specifically ran into an issue where my connection was trying to utilize specific github IP addresses. Have no fear... it turns out that the same command can be utilized.
ssh-keygen -R 140.82.114.4I actually had to do a few of them
ssh-keygen -R 140.82.113.3etc...
You could potentially use github's meta information endpoint to find address ranges, but that's a problem for a more clever person.
Today I Learned about fzy, an interactive fuzzy text selector.
Here's one neat use case - interactively searching through rails routes:
h/t Matt Polito
Today I learned how to parse a nested json with jq, but the nested json is a string. It's just easier to show an example so here we are:
{
"a": "{\"b\": \"c\"}"
}
This is not a common situation but I found that out today on a codebase and my first thought was to call jq get the content of the node a and then pipe it into another jq command. It would look like this:
echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq '.a' | jq
# => "{\"b\": \"c\"}"As we can see the result is not a json, but a string so we cannot access inner nodes just yet.
And the solution to this problem is to use the -r flag on the first jq call to output the result in a raw format, so the " surounding double quotes will disappear. And with that in place we can easily parse the nested/nasty json:
echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq
# => {
# => "b": "c"
# => }Then finally:
echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq '.b'
# => "c"It's possible to change the password of your current ssh key if you have the current password or it is not currently password protected. You can use the command:
ssh-keygen -p
From the man pages -
Requests changing the passphrase of a private key file instead of
creating a new private key. The program will prompt for the file
containing the private key, for the old passphrase, and twice for
the new passphrase.
Today I learned how to group by json data by a key using jq. In ruby that's very trivial, it's just about using the group_by method like that:
[
{name: "John", age: 35},
{name: "Bob", age: 40},
{name: "Wally", age: 35}
].group_by{|u| u[:age]}
# {
# 35=>[{:name=>"John", :age=>35}, {:name=>"Wally", :age=>35}],
# 40=>[{:name=>"Bob", :age=>40}]
# }
But using jq I had to break it down to a few steps. So let's say that I have this json:
[
{"name": "John", "age": 35},
{"name": "Bob", "age": 40},
{"name": "Wally", "age": 35}
]
The idea is that we'll call the group_by(.age)[] function to return multiple groups, then I pipe it to create a map with the age as the key. Finally we'll have these bunch of nodes not surounded by an array yet, so I am pipeing to a new jq command to add with slurp:
cat data.json |
jq 'group_by(.age)[] | {(.[0].age | tostring): [.[] | .]}' |
jq -s add;
# {
# "35": [{"name": "John", "age": 35},{"name": "Wally", "age": 35}],
# "40": [{"name": "Bob", "age": 40}]
# }
Today I learned that we can specify a default value when getting a node in json using jq:
{
"users": [
{"name": "John"},
{}
]
}cat my.json | jq '[.users[] | .name // "my-default"]'
# [
# "John",
# "my-default"
# ]
We just did a PostgreSQL bump in Heroku from 13.8 => 14.5 (the latest Heroku supports at this day). The process was very smooth and kind of quick for a 1GB database. Here's the script we end up running:
# Change the following `basic` to the right plan for you
heroku addons:create heroku-postgresql:basic -r heroku-staging
heroku pg:wait -r heroku-staging
heroku pg:info -r heroku-staging
# Now grab the NEW and OLD URLS to change the following commands:
heroku maintenance:on -r heroku-staging
# It took less than 2 mins for a 1GB database
heroku pg:copy DATABASE_URL CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
# It's usually fast, it depends on how long the app takes to reboot
heroku pg:promote CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
heroku maintenance:off -r heroku-staging
heroku addons:destroy CHANGE_HERE_OLDCOLOR_URL -r heroku-staging
You can access the last command's last argument with $_
Example:
> echo '🍟 I Eat Food'
🍟 I Eat Food
> echo $_
🍟 I Eat FoodExample Two:
> dropdb my_database
> createdb $_
TIL by accident that Cmd + Shft + E on Mac iTerm2 can toggle a timeline of your commands as an overlay on the right side of our panel.
Check this out:
You can download files with a nice progress bar using curl's -# flag:
curl -# -O https://files.example.com/large/long_video.mp4
################# 38.6%This might be preferable to the verbose output:
curl --no-progress-meter -O https://files.example.com/large/long_video.mp4
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
17 433M 17 75.6M 0 0 28.7M 0 0:00:15 0:00:02 0:00:13 28.8MShell function:
function renametab () {
echo -ne "\033]0;"$@"\007"
}Usage:
renametab Bob\'s BurgersOutput:
The cp command has a -n flag that you can use to prevent the copied file from overriding another file if one with the same name exists
When a naming collision occurs and the -n flag is present, the copy command does nothing
You can use carets in bash to quickly do string replacements in your previous commands:
$ echo "wibble wubble"
wibble wubble
$ ^bb^gg
echo "wiggle wuggle"
wiggle wuggleHere's the documentation
h/t Dillon
Today I learned that uniq has an -c option that counts the number of times the line occurred and prepends for each uniq result. That's very handy tool:
$ echo "foo\nfoo\nbar\nfoo" | sort | uniq -c
1 bar
3 foo
I was playing around with some ffmpeg filters, like cropping, scaling and overlays and I was tired of waiting for the video to be fully re-encoded in order to see the changes.
ffmpeg -i video.mp4 -vf "crop=in_w:in_h/2:0:0" -c:a copy output.mp4
I'm glad that this is not a problem because you can use ffplay to preview the changes instantly without having to wait:
ffplay -i video.mp4 -vf "crop=in_w:in_h/2:0:0"
I learned that in unix, root (e.g. /) is the only directory that is the parent directory of itself.
$ ls -lai / | grep '\./'
2 drwxr-xr-x 20 root wheel 640 Jan 1 2020 ./
2 drwxr-xr-x 20 root wheel 640 Jan 1 2020 ../
In the above example, the files . and .. both have the same i-node: 2
Source: Brian W. Kernighan, & Rob Pike (1984) The UNIX Programming Environment. Prentice-Hall, Inc
I have a cronjob to open macOS's Photo Booth every weekday so I can take a picture of my work life. Unfortunately, it opens the program every weekday; I'd rather it quickly closes itself if I'm not there or otherwise occupied. Today I used pkill in the cronjob to terminate the program five minutes after opening:
20 9 * * 1-5 pkill "Photo Booth"
pkill kills a process by name. You can figure out how to make pkill effective using pgrep, a companion program that searches for running processes by name. Using it, I learned that the string "Photo Booth" was specific enough to find and kill Photo Booth:
$ pgrep -l "Photo"
292 Photo Booth"Photo Booth", PID 292 (today), is the process I programmatically kill every weekday at 9:20 AM.
I use TextEdit from time to time to print code, explained in this post. Today I learned that you can open a file in that program via the command line in OSX:
$ open -e your-file.txtHappy hacking and printing.
Here's a situation: you're watching a server log in Tmux, about to trigger an action that will produce log data you care about. You hit return a bunch of times to create a visual break in the server log. Then you can scroll up and see the beginning of your revelant history.
What actually happens? Sometimes, the server logs tons and tons of information, and your visual break gets buried way above the fold. It's hard to find the break, and you're searching through all that information, plus anything that happened before.
There's a better way! Tmux's clear-history command "removes and frees the history of the specified pane." In the Hashrocket Dotmatrix, we combine that with send keys -R, which "causes the terminal state to be reset." Here's the mapping:
bind-key C-k send-keys -R \; clear-history
Type the Tmux leader, then C-k, and your terminal pane will be visually cleared and cleared of its history, making reading and reverse searching much easier.
h/t Gabe Reis
I've often wondered what the rc portion of my .vimrc filename, and other dotfile filenames, means. The history is fascinating. TL;DR it most likely stands for "run commands."
I had this json file that was an array of objects and some of those objects had different keys. I wanted to visualize that data in a spreadsheet (data science, AI, machine learning stuff) so I thought about having a CSV file where each JSON key would become a CSV column.
// file.json
[
{
"type": "Event1",
"time": 20
},
{
"type": "Event2",
"distance": 100
}
]
jq to the rescue:
cat file.json | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' > file.csvHere is the output:
"distance","time","type"
,20,"Event1"
100,,"Event2"
I wrote a script the other day designed to help me download and edit files faster. In part of the script, I wanted to open Vim in an existing Tmux pane, and in the process I learned about the tmux send-keys command. Here's how it works:
$ tmux send-keys -t 3 "vim" Enter
send-keys, aliased send, sends your string of commands to your pane (t) of choice. Running the above opens Vim in pane #3.
The -t flag accepts negative numbers, too, like an array index. In my version of the above command, I send the keys to pane -1, the last pane on the screen, which is where I keep Vim in my Tmux session.
zsh helpfully comes installed with help files for all the builtins and a run-help command to help you access those help files. There is a trick though, before setting any environment variables here's what happens:
$ run-help
There is no list of special help topics available at this time.
This is because the HELPDIR isn't set. You have to find the install location for zsh's help files and set the env var to that dir. On my system that looks like this:
export HELPDIR='/usr/share/zsh/help'
Then when you run run-help you should see a list of builtins for which there is help documentation. This is the same documentation that you can get via man builtins but much more readable and discoverable. run-help will call man as well if it can't find you're arg in the help files.
For me run-help is cumbersome to type so I alias it. Here's what goes into my .zshrc:
export HELPDIR='/usr/share/zsh/help'
alias help=run-help
zsh is now much more helpful!
zsh comes with it's very own tetris game. No plugins needed!
You do need to autoload the tetriscurses function:
autoload -Uz tetriscurses
And then run tetriscurses.
While in the game, you can press H to learn which keys do what, and that looks like this:
left: h, j, left
right: right, n, l
rotate: up, c, i
soft drop: down, t, k
hard drop: space
quit: q
press space to returnalso, maybe you want an alias for this?
autoload -Uz tetriscurses
alias tetris=tetriscurses
I'm putting the above straight into my .zshrc! Happy Sunday!
Sometimes linux can be a maze of symbolic links. On my system, the java command exists at /usr/bin/java which is a link that points to /etc/alternatives/java which is a link that points to /usr/lib/jvm/java-8-oracle/jre/bin/java.
Instead of looking up each of the links of these files with ls -l, readlink -e will the links all the way through to the eventual file. In my case that would look like this:
$ readlink -e `which java`
# returns /usr/lib/jvm/java-8-oracle/jre/bin/java
You can learn more with man readlink.
On Mac, there is a readlink command, but there is no -e flag and it is not recursive.
What makes an alias global? Well, the -g flag of course. And what does this globality give you? Well, the ability to invoke an alias anywhere in the command line.
If I like the word 'Potateos' but I don't ever have the energy to type the whole thing then I can create a global alias for that word:
> alias -g PO="Potatoes"
> echo PO
Potatoes
That's convenient and cool. What is it actually for? Maybe redirecting errors to /dev/null:
> alias -g NO='2> /dev/null'
> echo foo >> /dev/stderr
foo
> (echo foo >> /dev/stderr) NO
# no output, it got swallowed!Looks weird and maybe not useful, but maybe you can creatively find a useful way to use it:
I learned about this zsh functionality and other functionality here.
I stumbled across this zsh tricks post yesterday and am blown away by the hash command, which allows you to see and manipulate the hash table for either commands or for directory shortcuts.
hash by itself in zsh will output the location for all the commands.
hash -d shows you all of the named directories, and check this out you can navigate to one of those directories with ~shortcutname, like this:
$ hash -d | grep bin
bin=/bin
daemon=/usr/sbin
proxy=/bin
sync=/bin
$ cd ~daemon
$ pwd
/usr/sbinYou can create your own directory shortcuts like this:
$ hash -d mydir=/home/me/very/long/path
And then cd to it:
$ cd ~mydir
$ pwd
/home/me/very/long/pathCrazy! Read more in the zsh docs.
I'm currently working on an app that forwards logging around to various locations on the Linux server. It's a bit tricky for me to figure out where any action I take in the browser is being logged. I need those logs!
A nice way to figure out where the logging is happening is to narrow it down to one directory (say, /var/log/) then ls that directory, ordering by most recently updated. The items at the top of the list have been recently updated, and thus probably contain valuable loggings!
$ ls -lt
I'm throwing on the -l flag for more detail. If there are lot of logs, filter it down with head:
$ ls -lt | headThanks for the idea, Kori!
I always have trouble remembering how to get the name of the current directory. So strange pneumonics is the way to go.
The first amigo is a shell variable:
echo $PWD
# returns '/home/chris/tils'
There is also a pwd command that returns the same thing.
The second amigo is basename which gives you the current directory name without its path:
basename $PWD
# returns 'tils'
The third amigo is dirname which gives you the path without the current directory name:
dirname $PWD
# returns '/home/chris'So now I can do things like
alias tnew=tmux new -s $(basename $PWD)because I always, always, name my tmux session after the name of the current directory.
Today while doing some sleuthing, I learned about the host command. host "is a simple utility for performing DNS lookups." It helped me connect a series of domains to their respective AWS EC2 servers, without a visit to the domain registrar.
Example:
$ host jakeworth.com
jakeworth.com has address 184.168.131.241
jakeworth.com mail is handled by 10 mailstore1.secureserver.net.
jakeworth.com mail is handled by 0 smtp.secureserver.net.
More info: man host