The Terminal Programmer

Colorful assertion failure diffs in MiniTest

Suraj N. Kurapati

A colorful MiniTest diff, after patching.

  1. Problem
    1. Approach
      1. Solution
        1. ~/bin/gdiff
          1. Ruby on Rails integration

        Problem

        After a long day of practicing test driven development in Ruby, I grew weary of mentally parsing the colorless diffs that MiniTest displays when assert_equal fails:

        A typical MiniTest diff, before patching.

        Approach

        I dove into the MiniTest codebase using ack, hoping to find some way to pipe its diff output into colordiff during display, and discovered that MiniTest already shells out to diff(1) to produce its diff output.

        In fact, MiniTest tries several different flavors of diff(1) before falling back to the standard one. One of these flavors is the mysterious gdiff, which neither exists on my Arch Linux system nor is available in its package repositories. That’s the ticket!

        Solution

        I quickly masqueraded colordiff as gdiff within my user account:

        ln -s `which colordiff` ~/bin/gdiff

        re-ran my failing test, and saw colored diffs in the failure report!

        ~/bin/gdiff

        Three years later, in the summer of 2014, I wrote my own gdiff script in Ruby to further enhance the readability of MiniTest’s assert_equal diffs by intelligently combining two complementary approaches of visualizing intra-line differences into a single format:

        A colorful MiniTest diff, after patching.

        There are several things happening in the screenshot above:

        1. The changes are first visualized as a line-wise diff using the diff-highlight script (if available) that comes bundled with Git itself.

        2. A redundant hunk header (in this case: @@ -1 +1 @@) is displayed in reverse video to indicate where the first visualization ends and where the second begins.

        3. The changes are visualized again as a character-wise diff by squashing common portions of changed lines using Git’s native --word-diff-regex algorithm unless the hunk being visualized only consists of whitespace changes.

        As you can see, the second visualization complements the first by showing you the character-wise differences within the line-wise differences! This combination provides a more meaningful difference visualization overall.

        In addition, this gdiff(1) script goes beyond standard diff(1) to:

        Finally, here is the source code of my gdiff(1) script, also available on GitHub, ready for use. Simply download it into your ~/bin directory and happy diffing! :)

        #!/usr/bin/env ruby
#
# MiniTest runs this command to show the differences between expected and
# actual values in assert_equal(), so we make it word-aware and colorful.
#
# You can make this script use blue instead of green by setting the COLORBLIND
# environment variable, as described at https://pypi.python.org/pypi/fragments
#
# See http://sunaku.github.io/minitest-colordiff.html for more information.
#

require 'shellwords'

FILE_BEGIN_REGEXP = /\A\e\[1mdiff/
HUNK_BEGIN_REGEXP = /(?=^\e\[36m@@)/

# highlight the different inside changed lines while keeping unified output
linewise_diff = `git diff --color=always #{ARGV.shelljoin} |
( perl /usr/share/doc/git/contrib/diff-highlight/diff-highlight || cat )`

linewise_hunks = linewise_diff.split(HUNK_BEGIN_REGEXP)

# squash common portions of changed lines and only highlight the differences
# http://neurotap.blogspot.com/2012/04/character-level-diff-in-git-gui.html
charwise_diff = `git diff --color=always --color-words \
--word-diff-regex='[^[:space:]]|([[:alnum:]]|UTF_8_GUARD)+' #{ARGV.shelljoin}`

charwise_hunks = charwise_diff.split(HUNK_BEGIN_REGEXP).map do |hunk|
  # nullify character-wise diffs that don't give any whitespace information
  if hunk =~ /\S\e\[3[12]m.*?\e\[m/
    # highlight headers to indicate that these character-wise diffs are
    # alternate representations of the line-wise diffs that precede them
    # and also highlight the changes themselves in character-wise diffs
    hunk.gsub(/\A\e\[36(?=m@@)|\e\[3[12](?=m.*?\e\[m)/, '\&;7')
  end
end

# combine the diffs by joining the charwise output into the linewise output
# and use uniq() to show only one copy of common hunks between both outputs
diff = linewise_hunks.zip(charwise_hunks).map(&:uniq).flatten.map do |hunk|
  if hunk =~ FILE_BEGIN_REGEXP
    # strip diff headers and git index headers which mention /tmp/* files
    hunk.sub(/\A.+\bexpect.+\bbutwas.+\n\e\[1mindex.+\n/, '').

    # provide human-friendly descriptions of MiniTest value assertions
    sub(/(?<=m-{3}\s)\S+?\bexpect\S+/,  'I expected it to be...').
    sub(/(?<=m\+{3}\s)\S+?\bbutwas\S+/, 'but it actually was...').

    # color-code file headers according to difference highlighting colors
    sub(/^\e\[1(?=m-{3})/, '\&;31').sub(/^\e\[1(?=m\+{3})/, '\&;32')
  else
    hunk
  end
end.join

# use blue instead of green for users who indicate that they are colorblind
diff = diff.gsub(/(\e\[[\w;]*\b3)2m/, '\14m') if ENV['COLORBLIND']

print diff

        Ruby on Rails integration

        Installing the above gdiff script is not enough for Ruby on Rails (which provides its own, plain “expected but was” message for assertion failures) so the following monkeypatch is necessary in your test helper to override Ruby on Rails’ assertion failure messages with MiniTest’s native diffing.

        class Minitest::Unit
  alias _0dcfe0e3_de10_40df_bc94_da27aa85c215 puke
  def puke klass, meth, e
    if e.message =~ /\\A<(.*?)> expected but was\\n<(.*?)>\\.\\Z/m
      e.message.replace Object.new.extend(Minitest::Assertions).diff($1, $2)
    end
    _0dcfe0e3_de10_40df_bc94_da27aa85c215 klass, meth, e
  end
end

        After adding the above snippet to your test/test_helper.rb file, you should now see the forementioned gdiff script being used in Ruby on Rails!

        Updates