Today I Learned

hashrocket A Hashrocket project

38 posts about #testing surprise

Stub Environment Variables in Vitest

You can stub environment variables in Vitest using vi.stubEnv. This will stub the value on process.env and import.meta.env.

You can also reset all env vars back to their original value with vi.unstubAllEnvs.

process.env.COOL_ENV_VAR; // => "test"
import.meta.env.COOL_ENV_VAR; // => "test"

vi.stubEnv('COOL_ENV_VAR', "stubbed");

process.env.COOL_ENV_VAR; // => "stubbed"
import.meta.env.COOL_ENV_VAR; // => "stubbed"

vi.unstubAllEnvs();

process.env.COOL_ENV_VAR; // => "test"
import.meta.env.COOL_ENV_VAR; // => "test"

Docs

Execute CDP from a Capybara test

Selenium Chrome:

With capybara you can access the page's driver by using page.driver. Next you can access the browser methods on the driver withpage.driver.browser, then the .execute_cdpmethod can be used to execute chrome devtools commands on your webdriver browser.

In a capybara test, it could like: 

test_browser = page.driver.browser

coordinates = { latitude: 35.689487,
                longitude: 139.691706,
                accuracy: 100 }

test_browser.execute_cdp('Emulation.setGeolocationOverride', **coordinates)

This can be used to mock some client-side information such as user device metrics, geo-location, or even emulate slow CPUs 😳

About rspec test tags

You want a way to run a specific sub-set of your specs. Well RSpec has a tag metadata that you can add to make things a lot nicer:

describe "my awesome code", :awesome do
  # some set of awesome tests
end

Now I just run rspec --tag awesome and you're just going to run your block!

It also takes a negation, which means that if you want to run everything else you just use rspec --tag ~awesome.

You can also use this to add behavior to the way examples are dealt with via RSpec.configure

RSpec.configure do |config|
  config.append_after : each do |ce|
    if ce.metadata[:awesome]
         # do something awesome to the example
    end
  end
end

About `:aggregate_failures` for rspec

When writing rspec tests you can separate your test cases as in:

describe "foo" do
  specify { expect(some_behavior).to be_well_behaved }
  specify { expect(some_other_behavior).to be_well_behaved }
end

Or if you prefer to write:

describe "foo" do
  it "handles some behaviors" do
    expect(some_behavior).to be_well_behaved
    expect(some_other_behavior).to be_well_behaved
  end
end

The second example has the downside that it will fail out of the it block if the first expect doesn't pass. To get around this, you can use:

describe "foo" do
  it "handles some behaviors", :aggregate_failures do
    expect(some_behavior).to be_well_behaved
    expect(some_other_behavior).to be_well_behaved
  end
end

Which will run the specs and collect their failures.

RSpec's Render Template is Permissive

In a controller test, when you expect a request to render a template, you'd write something like this:

expect(response).to render_template(new_user_path)

If your controller action looked like the following, the expectation would pass:

class UsersController < ActionController::Base
  def action
    ...
    render :new
  end
end

render_template delegates to ActionController::TemplateAssertions#assert_template. What I didn't know, however, is how permissive (magical) it is.

Let's say instead of the action's render :new you returned a json object with the template, rendered to a string, as one of its values:

def action
  response_body = render_to_string(layout: false, template: "users/new.html.haml")
  render json: { page: response_body }
end

expect(response).to render_template(new_user_path) still passess!!!

Also if you say expect(response).to render_template("new.html.haml"), it also passes. How does it know you meant the user's new.html.haml and not some other arbitrary one? Maybe because you're in the UsersControllerSpec. You can also expect...render_template for templates unrelated to the controller's scope.

I suspect that this permissiveness is caused by Rails registering the rendering of templates, irrespective of whether they are rendered to strings or directly as a response to a controller action. expect...render_template maybe then looks through the registered renders 🤷🏾‍♂️. Either way, it's magical!

Bootstrap CircleCI Setup with Docker 🐳

CircleCI maintains a collection of Docker containers, pre-installed with different languages and tools.

These include language image variants that can eliminate a lot of CI setup churn. I recently used the -browsers variant, which ships with Chrome, Firefox, Java 8, and Geckodriver.

Here's how you could use such a container.

# .config/circle.yml

version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.5.1-browsers
    steps:
      - run: rake

Open devtools when running a Selenium Chrome test

The Network tab in Chrome devtools doesn't record requests unless devtools is open. This makes debugging specific issues in tests much harder. It's great being able to see which api requests were made and what payloads they returned.

You can start Chrome with devtools open though with the the chrome option --auto-open-devtools-for-tabs.

If you are using Selenium with Chrome in a Ruby integration test, you can pass the option 

opts = {
  browser: :chrome,
  options: Selenium::WebDriver::Chrome::Options.new(
  args: %w(--auto-open-devtools-for-tabs --window-size=2400,2400)
)
}

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, opts)
end

Opening devtools automatically may restrict your window size enough to disrupt some of your tests in which case you can set -window-size to a value that accomodates your website.

Chrome DevTools Audit Panel

One highlight of the Chrome 63 update was the addition of four new audits to DevTools.

'Audits' are one of the many DevTools panels lurking in the background, waiting to make your application better; I hadn't noticed them until today. They will analyze and provide a downloadable report on your application's Progressiveness, performance, accessibility, and best practices.

'Today I Learned' scored 91 out of 100 on accessibilty. That's something I'd like to improve, and this panel could help direct that journey.

Message order constraint in RSpec

Sometimes you need to make sure your code is executing in a specific order. In the example below we have a Payment double that needs to first call processing! and then approved!. So you write a test like this:

it "approves the payment" do
  payment = double("Payment")

  expect(payment).to receive(:processing!)
  expect(payment).to receive(:approved!)

  payment.processing!
  payment.approved!
end

If you change the order of the method calls your test will still pass:

...
  payment.approved!
  payment.processing!
...

Finished in 0.01601 seconds (files took 0.20832 seconds to load)
1 example, 0 failures

To guarantee the order, RSpec has an option to specify an order constraint. So we want to make sure processing! is called before approved! :

 expect(payment).to receive(:processing!).ordered
 expect(payment).to receive(:approved!).ordered

Now if you run the test with the wrong order, you will see the error:

it "approves the payment" do
  payment = double("Payment")

  expect(payment).to receive(:processing!).ordered
  expect(payment).to receive(:approved!).ordered

  payment.approved!
  payment.processing!
end

Failure/Error: payment.approved!
   #<Double "Payment"> received :approved! out of order

RSpec Covers a Range

Today I used a feature of RSpec I've never used before, cover.

cover returns true if it's argument is covered by the given range:

expect(0..1).to cover(order.risk_score)

In my use case, I had an attribute risk_score that is calculated by a third-party API. It should always be between zero and one, but it changes on every test run because the API considers the test user more and more risky each time. cover allowed me to test this attribute, and the underlying logic, in a flexible way.

Append an RSpec Failure Message

Have you ever wanted to customize an RSpec failure message? It's possible, but we lose the original message. What if we want both?

Here's one technique:

begin
  expect(actual).to eq(expected)
rescue RSpec::Expectations::ExpectationNotMetError
  $!.message << <<-MESSAGE

You broke it.

Maybe you deleted the fixtures? Try running `rake db:create_fixtures`.
  MESSAGE
  raise
end

This rescues the failure ExpectationNotMetError, then shovels our HEREDOC string onto the message of $!, a built-in Ruby global variable representing the last error. Then, we raise.

The result is our RSpec error, with the provided message, followed by our custom text.

Cucumber Suite Hooks

Adding hooks that run before and after your RSpec test suite looks like this:

# spec/helper/spec_helper.rb

RSpec.configure do |config|
  config.before(:suite) do
    My::Service.build!
  end

  config.after(:suite) do
    My::Service.tear_down!
  end
end

But for Cucumber, the API isn't quite as obvious. I found some interesting discussion online about this subject, and eventually settled on the following:

# features/support/env.rb

My::Service.build!

at_exit do
  My::Service.tear_down!
end

Anything in features/support/env.rb gets run before the Cucumber test suite, so the first method call functions like a before suite block. at_exit is Ruby Kernel and runs when the program exits. This implementation works roughly the same as a before and after block.

ExtractRspecLet

From the Hashrocket Vault...

Today I got to see the :ExtractRspecLet command from the vim-weefactor plugin. It does what the names suggests, converting this:

# spec/model/foobar_spec.rb

foo = FactoryGirl.create :foobar

To this:

# spec/model/foobar_spec.rb

let(:foo) { FactoryGirl.create :foobar }

It also moved the new let from inside my it block to right underneath my context block. Awesome!

h/t Josh Davey and Dillon Hafer

Capybara Find with Wait

Integration tests covering JavaScript features are a prime location for timing-related flickers. Often our implementation and expectations are correct, but the test gives up and bails before the desired behavior can be observed.

Patience!

With Capybara, one way around this is to just wait a little longer. This code:

find('.mj-modal iframe', wait: 5)

will wait for the iframe up to five seconds, overriding Capybara's default max wait time of two seconds. On certain test runs this can make all the difference.

Capybara tests on a Rails Engine

If you have a Rails Engine you probably have a dummy app attached in test/dummy or spec/dummy to run and manual test the engine, right?

So you can use this dummy app to have some feature tests with capybara.

The trick is to change the RAILS_ROOT to point to the correct file.

For cucumber tests add the following into features/support/env.rb:

ENV["RAILS_ROOT"] ||= File.expand_path(File.dirname(__FILE__) + '/../../spec/dummy')

For rspec tests add the following into spec/rails_helper.rb:

ENV["RAILS_ROOT"] ||= File.expand_path(File.dirname(__FILE__) + '/dummy')

Rerun Failed Cucumber Tests

An underused trick in Cucumber is the rerun format.

Run your suite like so:

$ cucumber -f rerun -o rerun.txt

This runs your tests with the rerun formatter, sending the output to rerun.txt. After breaking a test, here's my generated file:

$ cat rerun.txt                                                                                                                                                 
features/visitor_views_posts.feature:9

Now we can run again against just the failing example!

$ cucumber @rerun.txt

Use this to drill down through a giant integration test suite.

Full Backtrace in RSpec

Sometimes an RSpec error message can seem overwhelmingly verbose, other times frustratingly terse. Today I learned that this can be customized in Rails, with the following configuration:

# spec/spec_helper.rb
config.full_backtrace = false # or true?

Turn this on, and every failure will include a stack trace. Turn it off, and it won't. You can override a false setting at the command line, like so:

$ rspec spec/some/thing_spec.rb --backtrace

Great for fast debugging.

See rspec --help for more information.

Cucumber step inception

I learned that you can call a cucumber step inside another one. Super cool!

Image a simple step like this:

Then /^I should see "(.*)"$/ do |text|
  expect(page).to have_content(text)
end

Now you have modals on your web app and you need to restrict this assertion to inside the modal.

So you can start to duplicate the step above to match the within modal and all the ones that could be reusable with modals.

Or you can create a new generic step that scope into the modal and calls your original step, such as:

And /^(.*?) within the modal$/ do |step_definition|
  within ".modal", visible: true do
    step step_definition
  end
end

Now you can call both steps in your gherkin files:

Then I should see "Hello World!"

or

Then I should see "Hello World!" within the modal

Just be careful with generic and reusable steps. The idea is to help developers and not confuse them.

reference: Cucumber wiki - call step from step definition

h/t Taylor Mock

Capybara's RackTest driver limitation

By default, if you click a link through Capybara and it takes you into an external site, the current_url will work fine, but the content on the page does not change.

RackTest is the Capybara's default driver and it has its limitations.

you cannot use the RackTest driver to test a remote application, or to access remote URLs

If you really need to do that know other drivers features and try them.

h/t by Nick Palaniuk

Conveying Intent with RSpec subject

Today during a test refactor, I discovered RSpec's subject method. We used it a lot, because it helps convey the test's intent.

subject can be used as a convenience method, and to define the intent of a test. If you've ever seen a comment # This is what we are testing, subject would be an upgrade to that.

Here are three passing examples of this method in practice, from most to least verbose.

Explicit, with a named subject:

RSpec.describe Array, "with some elements, explicit named subject" do
  subject(:array) { [1, 2, 3] }
  it "should have the prescribed elements" do
    expect(array).to eq [1, 2, 3]
  end
end

Explicit, with no name:

RSpec.describe Array, "with some elements, explicit subject" do
  subject { [1, 2, 3] }
  it "should have the prescribed elements" do
    expect(subject).to eq [1, 2, 3]
  end
end

Implicit:

RSpec.describe Array, "with some elements, implicit subject" do
  it "should have the default elements" do
    expect(subject).to be_empty
  end
end

Substitute this for let on your described class. There are a ton of interesting uses and edges for this method, so dig into those docs.

h/t Mike Chau

Documentation

Test Validation Errors with RSpec

We can use RSpec unit tests to confirm model validations, like this:

# app/models/developer.rb
validates :email, format: { with: /foo/ }

# spec/models/developer_spec.rb
developer.email = 'bar'
expect(developer).to_not be_valid

But there's a false positive here; expect returns true, but we can't be sure why it returned true. It may have been an invalid email, because the regex doesn't match bar, or it may be invalid for many other reasons, such as an incomplete fixture.

Another method I re-learned today is this:

# spec/models/developer_spec.rb
developer.email = 'bar'
expect(developer).to_not be_valid
expect(developer.errors.message[:email]).to eq ['is invalid']

The first expect triggers the validations; the second proves the right error was included. This makes a difference when flash messages or other code depend on your errors stack.

Set custom user agent on #Capybara

Sometimes you want to emulate a request to your site from a device with a specific user-agent. This is particularly useful if you have a different behavior when users use those devices (e.g. wrapped mobile app using Cordova)

Setting the user agent can be accomplished by setting the header on the current session:

Capybara.current_session.driver.header('User-Agent', 'cordova')

Mark Test Pending With xit

Mark any RSpec test temporarily pending by changing it or specify to xit.

Take this test, from the 'Today I Learned' codebase:

# spec/models/channel_spec.rb

describe Channel do
  it 'should have a valid factory' do
    channel = FactoryGirl.build(:channel)
    expect(channel).to be_valid
  end
end

Change it to this:

# spec/models/channel_spec.rb

describe Channel do
  xit 'should have a valid factory' do
    channel = FactoryGirl.build(:channel)
    expect(channel).to be_valid
  end
end

Here's the test output:

$ rspec spec/models/channel_spec.rb:4
Run options: include {:locations=>{"./spec/models/channel_spec.rb"=>[4]}}

Channel
  should have a valid factory (PENDING: Temporarily skipped with xit)

...

Use this when refactoring code that breaks tests, to avoid deleting or commenting-out that hard-won test logic while you get back to green, test by test.

h/t Dorian Karter

Relish

Expect a Case-Insensitive Match

We had a testing issue this week when a button's text save changes! was rendering as SAVE CHANGES!, due to the CSS property text-transform: uppercase. How do you test that?

One technique is to use RSpec's match method, and include case insensitivity with i (explored earlier here):

> button_text = "foo"
=> "foo"
> expect(button_text).to match("FOO")
RSpec::Expectations::ExpectationNotMetError: expected "foo" to match "FOO"
# stuff
> expect(button_text).to match(/FOO/i)
=> true

We ultimately hard coded the expectation to match("FOO"), because allowing fOo and FoO seemed too permissive. But it remains an option.

Select A Select By Selector

Generally when using Capybara to select from a select input, I reference it by its name which rails associates with the label:

select("Charizard", from: "Pokemon")

However, not all forms are going to have a label paired with every select input. We don't want to let our test coverage suffer, so we are going to need a different way to select. Fortunately, Capybara allows us to chain select off a find like so:

find('#pokemon_list').select('Charizard')

Expecting A Selector and Its Content

In integration tests with Capybara, I find myself writing code like this a lot:

within 'h1' do
  expect(page).to have_content('Today I Learned')
end

Today I remembered that there is another way:

expect(page).to have_selector('h1', text: 'Today I Learned')

I prefer this because it captures the scope and the content in one line. RSpec integration tests in particular can get pretty long, and I think this is a good opportunity to keep them concise.

Testing Edit Forms

Today I found a way to assert that an edit form's inputs include a record's saved data. I think it strikes a good balance between broad and narrow scope.

# spec/features/user_edits_kit_spec.rb

within 'form' do
  expect(page).to have_selector("input[value='Default copy.']")
end

This asserts that some content is inside an input field in the form, rather than just anywhere on the page. You can narrow the scope as needed.

RSpec multiple formats with different out buffers

On RSpec there is a way to use multiple formatters from the command line and pipe each into a different output buffer (file/stdout).

To do this use rspec -f p -f j --out result.json spec

-f is alias for --format which takes the arguments

[p]rogress (default - dots)
[d]ocumentation (group and example names)
[h]tml
[j]son

--out will save the last formatter mentioned to a file.

The original formatter will print to $stdout so you end up having the normal display from RSpec and a JSON file which you can use to parse with your own script.

I wish I could cURL that network request

So OK, you've got this webpage that's making an xhr (XML http request) and you really want to iterate on the results you're getting back from that request. cURL seems like the answer there, but sometimes constructing a cURL request on the command line isn't worth the effort.

Enter Chrome.

In the network panel of Chrome's developer tools you can ctrl-click the request in question and choose Copy as cURL. Presto! Paste it into the cmd line and start iterating on that endpoint!

RSpec let!

RSpec's let method comes in two varieties.

let can be used to define a memoized helper method.

let(:current_user) {  FactoryGirl.create :user }

But let is lazily-evaluated, meaning it isn't evaluated until the first time the method is invoked. The value returned will be cached across multiple calls in the same example.

let! forces the method to be evaluated before each example in a before hook.

let!(:current_user) {  FactoryGirl.create :user, active_range: (Time.now)...(Time.now + 1.second) }

You might use `let! when you need a method to be invoked before each example, or when measuring precise time-dependent behavior where lazy evaluation could produce inconsistent results. Source

Cleanup Cucumber Steps

Code that is never executed is bad. Try this on your brownfield test suite:

$ cucumber --dry-run -f stepdefs

The stepdefs flag reports the location of each step, and also which steps aren't being used by any test. Delete those dusty old steps!

--dry-run skips running the actual steps, a significant time-saver.

RSpec Specify

There are more ways to create a test block with RSpec than it. One is specify, and you can use it to make your tests more readable. Consider this block:

describe 'doing everything' do

  it 'in space' do
    # actions
  end

The sentence 'it in space' is awkward. You can replace it with 'specify', which brings us very close to a grammatically correct sentence. This is Ruby after all; we try to make it readable.

describe 'doing everything' do

  specify 'in space' do
    # actions
  end

RSpec Profile

RSpec includes an easy way to measure test speed, the -p (profile) flag. It takes an argument [COUNT] representing the number of blocks to measure, defaulting to 10.

Here's some sample output:

Top 3 slowest examples (0.03332 seconds, 57.9% of total time):
  Developer should have a valid factory
    0.0199 seconds ./spec/models/developer_spec.rb:4
  Post should have a valid factory
    0.00816 seconds ./spec/models/post_spec.rb:4
  Post should require a body
    0.00525 seconds ./spec/models/post_spec.rb:16

Assert No Selector

Capybara assertions can be very simple. Today I learned about the assert_no_selector and assert_selector methods.

Then 'assert_no_selector works' do
  assert_no_selector '.nonexistent-text'
end

Then 'assert_selector works' do
  assert_selector '.existent-text'
end

And here's the view that makes them pass:

# app/views/home/index.html.haml
.existent-text

No RSpec required!