Today I Learned

hashrocket A Hashrocket project

202 posts by viniciusnegrisolo twitter @vfnegrisolo

Connecting to github via ssh with different users

Due to github permissions on projects that I am working on I wanted to switch which user I am connecting to github on a per project base. I also want to keep using ssh keys for that, so I found out that we can have 2 different ssh keys pointint to the same domain, we just had to do a simple twich:

Host github.com
    HostName github.com
    IdentityFile ~/.ssh/id_ed25519

Host vinny.github.com
    HostName github.com
    IdentityFile ~/.ssh/vinny_id_ed25519

So I end up creating 2 ssh keys, 1 per user that I need to access on github. Then left the original ssh key as regular github.com domain, so no changes there, and on the projects that I need to access with the other user I had to clone with:

git clone git@vinny.github.com:MyOrganization/my-project.git
Bash

And if you have an exisintg project that you want to move to this new github user you'll have to change the remote on that:

git remote remove origin
git remote add origin git@vinny.github.com:MyOrganization/my-project.git
Bash

By the way, as you are changing which keys to use with an existing domain, you may have to restart the ssh-agent in order for this to work:

eval "$(ssh-agent -s)"
Bash

Expo Router add index page into stack

We had a situation that we wanted to guarantee that an "index" page under a tab navigation was always there, this way we could go back using the navigation to the index page, even if the user entered to that tab navigation via another screen. I learned that we can use withAnchor prop in conjunction with the unstable settings initialRouteName. This worked really fine and now our back buttons always go to each tab's index screens.

Check this out Expo Router navigation patterns

Validating Keyword lists in Elixir

I found out that Keyword.validate/2 is a great way to guarantee that a function would be called with the right keyword options. Check this out:

def do_something(%User{} = user, opts \\ []) when is_list(opts) do
  {:ok, opts} = Keyword.validate(opts, [:preload])
  preload = Keyword.get(opts, :preload, [])
  #...
end
Elixir

Here's some interesting cases:

# fails if you pass an invalid option
Keyword.validate([one: "1", two: "2"], [:one])
# => {:error, [:two]}

# fails if you pass a valid option that is not expected that many times
Keyword.validate([one: "1", one: "again"], [:one])
# => {:error, [:two]}

# it can set default values
Keyword.validate([one: "1"], [:one, two: "5"])
# => {:ok, [two: "5", one: "1"]}

# it succeeds if you pass the right options
Keyword.validate([one: "1", two: "2"], [:one, :two])
# => {:ok, [two: "2", one: "1"]}
Elixir

Ruby Data class to replace Struct

Ruby has a new-ish class to build "immutable" structs, check this out:

Measure = Data.define(:amount, :unit)
distance = Measure.new(100, 'km')

distance.amount
#=> 100

distance.unit
#=> "km"
Ruby

And if you try to use a setter, then it will fail:

 distance.amount = 101
(irb):7:in `<main>': undefined method `amount=' for an instance of Measure (NoMethodError)
Ruby

Change creation strategy in FactoryBot

I found out that's possible to change the FactoryBot strategy by invoking the to_create method inside the factory definition.

We had to do that to make factory bot linting to work on a factory that acts like an enum. So we did something like this:

FactoryBot.define do
  factory :role do
    to_create do |instance|
      instance.attributes = instance.class
        .create_with(instance.attributes)
        .find_or_create_by(slug: instance.slug)
        .attributes

      instance.instance_variable_set(:@new_record, false)
    end
  end
end
Ruby

The said part here is that FactoryBot expects us to mutate that instance in order to work.

Comparing Nullable values in PostgreSQL

PostgreSQL treats NULL values a bit different than most of the languages we work with. Said that, if you try this:

SELECT * FROM users WHERE has_confirmed_email <> TRUE
SQL

This would return users with has_confirmed_email = FALSE only, so NULL values that you have on your DB would be ignored. In this case if you want to get users with has_confirmed_email as FALSE or NULL then you can use IS DISTINCT FROM:

SELECT * FROM users WHERE has_confirmed_email IS DISTINCT FROM TRUE
SQL

Phoenix chunk huge files

So Phoenix, actually Plug, has a way to chunkify a response back to the client. This allowed us to create a stream out of an ecto query, then pipe that stream into a csv encoder and finally to Plug.Conn.chunk/2, which end up solving the huge memory spikes that we had when downloading a CSV. Here's a pseudo-code:

my_ecto_query
|> Repo.stream()
|> CSV.encode() # external lib that encodes from a stream into CSV
|> Enum.reduce_while(conn, fn chunk, conn ->
  case Conn.chunk(conn, chunk) do
    {:ok, conn} -> {:cont, conn}
    _ -> {:halt, conn}
  end
end)
Elixir

How to debug in Elixir

If you want to debug an elixir code execution you can use the dbg/2 macro. I was already using that nice macro to show the results on each step of a pipe execution on the logs, but I learned that if we use iex to start phoenix server then the dbg/2 calls will act like adding a breakpoints so we can pry on it. We need to start the server with:

iex -S mix phx.server
Bash

As a side note this debugging worked on the elixir 1.14.3 version but I saw that on the latest versions there's a new option to be passed to iex --dbg pry in order to swap the dbg implementation from the fancy IO output to the IEx.pry one.

How to get javascript params in LiveView?

I found out that Phoenix LiveView has a function get_connect_params to be used when you want to get some value sent at the connection time from the javascript.

This way I can send params like:

// app.js
const params = {
  _csrf_token: csrfToken,
  timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
};

const liveSocket = new LiveSocket('/live', Socket, { params });
JavaScript

And get those like on my LiveView:

# my_live_view.ex
@impl true
def mount(_params, _session, socket) do
  tz = get_connect_params(socket)["timezone"] || "America/New_York"
  {:ok, assign(socket, :timezone, tz)}
end
Elixir

Skip hidden fields on Phoenix inputs_for

I just found out that Phoenix Component inputs_for accepts a skip_hidden attr which was very useful today. We have a has_many relationship that we are using <.inputs_for> to decompose the nested fields and as it was generating some hidden fields the style was breaking.

We are actually using a Tailwind divide on the parent tag and those hidden fields were adding extra separation borders. A way to solve that was calling <.inputs_for twice as this:

<div>

  <.inputs_for field={@form[:posts]} />

  <div class="divide-y">
    <.inputs_for :let={form} field={@form[:posts]} skip_hidden>
      <.input type="text" field={form[:title]} label="Title" />
    </.inputs_for>
  </div>
</div>
Elixir

Phoenix LiveView slot attributes

Today I learned how to define an attribute on a Phoenix LiveView slot using a do block:

slot :column
  attr :field, :atom
  attr :sortable, :boolean, default: false
end

def table(assigns) do
  ~H"""
  <table>
    <tr>
      <th :for={col <- @column}>
        <%= col.label %>
        <.sort_by :if={col.sortable} field={col.field}/>
      </th>
    </tr>
  </table>
  """
end
Elixir

This way we can use our slots with attributes:

def render(assigns) do
  ~H"""
    <.table>
      <:colum field={:name} sortable>Name</:colum>
      <:colum field={:price}>Price</:colum>
      <:colum>Actions</:colum>
      ...
    </.table>
  """
end
Elixir

Check this out

Parsing nested string json

Today I learned how to parse a nested json with jq, but the nested json is a string. It's just easier to show an example so here we are:

{
  "a": "{\"b\": \"c\"}"
}
JSON

This is not a common situation but I found that out today on a codebase and my first thought was to call jq get the content of the node a and then pipe it into another jq command. It would look like this:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq '.a' | jq
# => "{\"b\": \"c\"}"
Bash

As we can see the result is not a json, but a string so we cannot access inner nodes just yet.

And the solution to this problem is to use the -r flag on the first jq call to output the result in a raw format, so the " surounding double quotes will disappear. And with that in place we can easily parse the nested/nasty json:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq
# => {
# =>   "b": "c"
# => }
Bash

Then finally:

echo '{"a": "{\\"b\\": \\"c\\"}"}' | jq -r '.a' | jq '.b'
# => "c"
Bash

The Outlet of Rails Stimulus

Besides values and targets Rails Stimulus has now outlets. Now we can invoke functions from one controller to the other like that:

// result_controller.js
export default class extends Controller {
  markAsSelected(event) {
    // ...
  }
}
JavaScript
// search_controller.js
export default class extends Controller {
  static outlets = [ "result" ]

  selectAll(event) {
    this.resultOutlets.forEach(result => result.markAsSelected(event))
  }
JavaScript

Javascript private class functions

Today I learned that you can make a class function to be private by prefixing it with a #. Check this out:

class MyClass {
  myPublicFunc() {
    console.log("=> hey myPublicFunc");
    this.#myPrivateFunc();
  }

  #myPrivateFunc() {
    console.log("=> hey myPrivateFunc");
  }
}

const myClass = new MyClass();

myClass.myPublicFunc();
// => hey myPublicFunc
// => hey myPrivateFunc

myClass.#myPrivateFunc();
// Uncaught SyntaxError:
//   Private field '#myPrivateFunc' must be declared in an enclosing class
JavaScript

We can also use that trick to make static methods, fields or static fields as well.

Thanks Andrew Vogel for this tip 😁

Composite Primary Keys using Elixir Ecto

Ecto allows us to map a table with composite primary keys into our schemas. The way to do that is by using the primary_key: true option for the Ecto.Schema.field/3 instead of dealing with the special module attr @primary_key. Check this out:

defmodule MyApp.OrderItem do
  use Ecto.Schema

  @primary_key false
  schema "order_items" do
    field :product_id, :integer, primary_key: true
    field :order_id, :integer, primary_key: true
    field :quantity, :integer
  end
end
Elixir

Identity Primary Keys on Ecto

Ecto, allow the type :identity to be used since 3.5 which is cool:

create table(:user, primary_key: false) do
  add :id, :identity, primary_key: true
  add :name, :string, null: false

  timestamps()
end
Elixir

That generates this SQL:

CREATE TABLE public.users (
    id bigint NOT NULL,
    name character varying(255) NOT NULL,
    inserted_at timestamp(0) without time zone NOT NULL,
    updated_at timestamp(0) without time zone NOT NULL
);

ALTER TABLE public.users ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY (
    SEQUENCE NAME public.users_id_seq
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1
);
SQL

The only issue is that there's no option to change from BY DEFAULT to ALWAYS 😕

Phoenix component attr definition

The new Phoenix.Component attr/3 function is awesome. It does compile time validations to all the places that are calling the component and also it comes with nice and useful options. For instance the default option:

attr :name, :string, default: "there"

def greet(assigns) do
  ~H"""
  <h1>Hey <%= @name %>!</h1>
  """
end
Elixir

That's very useful. That would totally replace most usages of assign_new like I used to do:

def greet(assigns) do
  assigns = assign_new(assigns, :name, fn -> "there" end)

  ~H"""
  <h1>Hey <%= @name %>!</h1>
  """
end
Elixir

This way we can call:

<.greet />
Elixir

And have this html generated:

<h1>Hey there!</h1>
HTML

Group Json data by a key

Today I learned how to group by json data by a key using jq. In ruby that's very trivial, it's just about using the group_by method like that:

[
  {name: "John", age: 35},
  {name: "Bob", age: 40},
  {name: "Wally", age: 35}
].group_by{|u| u[:age]}

# {
#   35=>[{:name=>"John", :age=>35}, {:name=>"Wally", :age=>35}],
#   40=>[{:name=>"Bob", :age=>40}]
# }
Ruby

But using jq I had to break it down to a few steps. So let's say that I have this json:

[
  {"name": "John", "age": 35},
  {"name": "Bob", "age": 40},
  {"name": "Wally", "age": 35}
]
JSON

The idea is that we'll call the group_by(.age)[] function to return multiple groups, then I pipe it to create a map with the age as the key. Finally we'll have these bunch of nodes not surounded by an array yet, so I am pipeing to a new jq command to add with slurp:

cat data.json |
  jq 'group_by(.age)[] | {(.[0].age | tostring): [.[] | .]}' |
  jq -s add;

# {
#   "35": [{"name": "John", "age": 35},{"name": "Wally", "age": 35}],
#   "40": [{"name": "Bob", "age": 40}]
# }
Bash

Upgrade Heroku PostgreSQL

We just did a PostgreSQL bump in Heroku from 13.8 => 14.5 (the latest Heroku supports at this day). The process was very smooth and kind of quick for a 1GB database. Here's the script we end up running:

# Change the following `basic` to the right plan for you
heroku addons:create heroku-postgresql:basic -r heroku-staging
heroku pg:wait -r heroku-staging
heroku pg:info -r heroku-staging
# Now grab the NEW and OLD URLS to change the following commands:

heroku maintenance:on -r heroku-staging
# It took less than 2 mins for a 1GB database
heroku pg:copy DATABASE_URL CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
# It's usually fast, it depends on how long the app takes to reboot
heroku pg:promote CHANGE_HERE_NEWCOLOR_URL -r heroku-staging
heroku maintenance:off -r heroku-staging

heroku addons:destroy CHANGE_HERE_OLDCOLOR_URL -r heroku-staging
Bash

Migrating Data in Ecto

I usually have created a mix task for data migrations to avoid putting the app down, but today I learned that ecto migrations accept an arg --migrations-path to their commands which allow us to have 2 separate folders for migrations.

With that we can easily use the default priv/repo/migrations folder for automatic migrations (for running on deployments) and a separate folder, let's say priv/repo/data_migrations that we can run when it's more convenient.

So in prod we run migrations on deploy and data_migrations on a quite time for the app to avoid downtime. And in dev and test env we just run them all as we usually have smaller dataset, so not a big deal.

Here's a good post about this strategy.

Ruby Private Class Methods

Today I learner that Ruby Module has private_class_method, this way we can for example make the new method as private on service objects:

class MyServiceObject
  private_class_method :new

  def self.call(*args)
    new(*args).call
  end

  def initialize(foo:)
    @foo = foo
  end

  def call
    @foo
  end
end
Ruby

Note that in this example the new method is private, so the .call will work, but the .new().call will raise an error:

irb> MyServiceObject.call(foo: "FOO")
=> "FOO"

irb> MyServiceObject.new(foo: "FOO").call
NoMethodError: private method `new' called for MyServiceObject:Class
Ruby

Rails composed_of

I learned that Rails ActiveRecord has composed_of method which allow us to group model attributes under a common PORO like object. Let's say we have an User with address_street and address_city fields, stored in the same table, so we can still group the address attributes:

class User < ActiveRecord::Base
  composed_of :address, mapping: [%w(address_street street), %w(address_city city)]
end
Ruby

This way, besides the user.address_street and user.address_city, we can also access the same values as:

address = user.address
puts address.street
puts address.city
Ruby

Rails AR readonly!

ActiveRecord has some readonly features that marks a model (or relation) not to be updated. This way if we try:

irb> user = User.last
=> #<User id: 123, ...>

irb> user.readonly!
=> true

irb> user.update(updated_at: Time.current)
   (0.3ms)  SAVEPOINT active_record_1
   (0.3ms)  ROLLBACK TO SAVEPOINT active_record_1
ActiveRecord::ReadOnlyRecord: User is marked as readonly
Ruby

Then we get a rollback with the readonly error.

An interesting thing is that the touch AR method does not respect this readonly feature (I tested on rails 6.0.4.1), check this out:

irb> user = User.last
=> #<User id: 123, ...>

irb> user.readonly!
=> true

irb> user.touch
   (0.3ms)  SAVEPOINT active_record_1
  User Update (1.4ms)  UPDATE "users" SET "updated_at" = $1 WHERE "users"."id" = $2  [["updated_at", "2022-03-08 20:39:40.750728"], ["id", 123]]
   (0.2ms)  RELEASE SAVEPOINT active_record_1
=> true
Ruby

Elixir Compilation Cycles with `--fail-above`

Elixir 1.13.0 introduced a --fail-above flag for the mix xref task which will fail that task execution under a certain criteria.

With that we can verify, for example, that our code is ok to have 1 cycle that goes over length of 4, but not 2 cycles. Let's see how it works:

$ mix xref graph --format cycles --min-cycle-size 4 --fail-above 1
2 cycles found. Showing them in decreasing size:

Cycle of length 6:

    lib/my_app_web/endpoint.ex
    lib/my_app_web/router.ex
    lib/my_app_web/views/layout_view.ex
    lib/my_app_web/live/components/sorting_link.ex
    lib/my_app_web/live/components/icon.ex
    lib/my_app_web/endpoint.ex

Cycle of length 5:

    lib/my_app_web/live/components/sorting_link.ex
    lib/my_app_web/live/components/icon.ex
    lib/my_app_web/router.ex
    lib/my_app_web/views/layout_view.ex
    lib/my_app_web/live/components/sorting_link.ex

** (Mix) Too many cycles (found: 2, permitted: 1)
Elixir

In this case xref found 2 cycles with a length greater than 4, and as I allowed only 1 then we can see the error.

PostgreSQL Indexes on Partition tables

When we CREATE INDEX in a partitioned table, PostgreSQL automatically "recursively" creates the same index on all its partitions.

As this operation could take a while we can specify the ONLY parameter to the main table index to avoid the index to be created on all partitions and later creating the same index on each partition individually.

CREATE INDEX index_users_on_country ON ONLY users USING btree (country);

CREATE INDEX users_shard_0_country_idx ON users_shard_0 USING btree (country);
CREATE INDEX users_shard_1_country_idx ON users_shard_0 USING btree (country);
SQL

Elixir __struct__/0

When we define a struct via defstruct macro we end up getting a __struct__/0 function on both struct module definition and on each struct map. The intriguing part is that the implementation of the module and the map are different, check this out:

iex(1)> defmodule Book do
...(1)>   defstruct title: nil, pages_count: 0
...(1)> end

iex(2)> Book.__struct__()
%Book{pages_count: 0, title: nil}

iex(3)> Book.__struct__().__struct__()
Book
Elixir

As we can see Book.__struct__() returns a new %Book{} struct with its defaults, meanwhile %Book{}.__struct() returns the Book module.

Elixir IEX multi-line command

A change on Elixir 1.12.0 made possible to pipe |> multi-line commands in iex where the |> operator is in the beginning of new lines.

That means that we can:

iex(1)> :foo
:foo
iex(2)>       |> to_string()
"foo"
iex(3)>       |> String.upcase()
"FOO"
Elixir

The docs also mention that all other binary operators works the same way, except +/2 and -/2, so that's also valid:

iex(1)> [:foo]
[:foo]
iex(2)> ++ [:bar]
[:foo, :bar]
iex(3)> |> Enum.join(" ")
"foo bar"
iex(4)> |> String.upcase()
"FOO BAR"
Elixir

Each line will run at a new command, so if you assign a variable in the first line you may not have what you expect, so watch out.

Phoenix Live View enable Profiling

Phoenix LiveView has a way to enable Profiling in the client side by just adding this into the app.js file:

// app.js
liveSocket.enableProfiling();
JavaScript

That will enable a log into your browser console such as:

toString diff (update): 1.224853515625 ms
premorph container prep: 0.006103515625 ms
morphdom: 397.676025390625 ms
full patch complete: 411.117919921875 ms

In this case we can see that the morphdom library is taking a considerable time to apply my DOM patches as my page has a huge html table full of data.

BTW, this function adds to 2 other very useful ones for debugging the client:

  • enableDebug ()
  • enableLatencySim(ms)

Rails ActiveRecord count on groups

Rails ActiveRecord can count queries with GROUP BY clauses, and in this case the result is not just an integer, but a Hash with the grouped values and the count for each group. Check this out:

Project.group(:status, :city).count
# SELECT COUNT(*) AS count_all, "projects"."status" AS projects_status, "projects"."city" AS projects_city
# FROM "projects"
# GROUP BY "projects"."status", "projects"."city"
=> {
  [:pending, "Jacksonville Beach"] => 21,
  [:finished, "Jacksonville Beach"] => 1061
  [:pending, "Chicago"] => 10,
  [:finished, "Chicago"] => 980,
}
Ruby

So rails manipulates the select statement to have all grouped by fields and the count(*) as well, which is pretty neat.

Get Elixir GenServer current state

Today I learned that we can use :sys.get_state/1 to get the current state of a GenServer.

Check this out:

iex(1)> pid = Process.whereis(MyApp.Repo)
iex(2)> :sys.get_state(pid)

{:state, {:local, MyApp.Repo}, :one_for_one,
 {[DBConnection.ConnectionPool],
  %{
    DBConnection.ConnectionPool => {:child, #PID<0.421.0>,
     DBConnection.ConnectionPool,
     {Ecto.Repo.Supervisor, :start_child,
      [
        {DBConnection.ConnectionPool, :start_link,
         [
           {Postgrex.Protocol,
            [
              types: Postgrex.DefaultTypes,
              repo: MyApp.Repo,
              telemetry_prefix: [:my_app, :repo],
              otp_app: :my_app,
              timeout: 15000,
              database: "my_app_dev",
              hostname: "localhost",
              port: 5432,
              show_sensitive_data_on_connection_error: true,
              pool_size: 10,
              pool: DBConnection.ConnectionPool
            ]}
         ]},
        MyApp.Repo,
        Ecto.Adapters.Postgres,
        %{
          cache: #Reference<0.1645792067.2416050178.91334>,
          opts: [
            timeout: 15000,
            pool_size: 10,
            pool: DBConnection.ConnectionPool
          ],
          repo: MyApp.Repo,
          sql: Ecto.Adapters.Postgres.Connection,
          telemetry: {MyApp.Repo, :debug, [:my_app, :repo, :query]}
        }
      ]}, :permanent, false, 5000, :worker, [Ecto.Repo.Supervisor]}
  }}, :undefined, 0, 5, [], 0, :never, Ecto.Repo.Supervisor,
 {MyApp.Repo, MyApp.Repo, :my_app, Ecto.Adapters.Postgres, []}}
Elixir

Ensure Ruby returns the correct value

Ruby has implicit returns for any possible block that I can think of, except ensure. There might be more, but this is the only one that I can think of now.

So in order to return something from inside the ensure block we must use the return reserved word explicitly. Check this out:

def check_this_out
  yield if block_given?
  :ok
rescue
  :error
ensure
  :ensured
end

irb()> check_this_out { "should work" }
=> :ok

irb()> check_this_out { raise "should fail" }
=> :error
Ruby

As we can see even though the ensure code runs on all calls it does not return :ensured.

Here's the code with the explicit return:

def check_this_out_explicit_ensure_return
  yield if block_given?
  :ok
rescue
  :error
ensure
  return :ensured
end

irb()> check_this_out_explicit_ensure_return { "should work" }
=> :ensured

irb()> check_this_out_explicit_ensure_return { raise "should fail" }
=> :ensured
Ruby