Have you ever wondered what happens when you run a Minitest test suite? How does it work?
This article will demystify Minitest's magic by going deep into its source code. After reading
it, you'll no longer consider Minitest a magical black box and will understand how Minitest
discovers your tests methods and executes them.
Reinvent the wheel
Although the standard programming wisdom
instructs us to avoid it,
reinventing the wheel is a great way to learn
the basic principles about the wheel.
So how does Minitest work?
Let's write a simplest code possible that demonstrates
how it's used:
Now we're ready for 2, so let's call the methods that start with test_.
Since these methods are instance methods, it makes sense to instantiate
descendant classes we have tracked:
Traceback (most recent call last):
6: from test.rb:22:in `<main>'
5: from test.rb:22:in `each'
4: from test.rb:24:in `block in <main>'
3: from test.rb:24:in `each'
2: from test.rb:25:in `block (2 levels) in <main>'
1: from test.rb:25:in `public_send'
test.rb:18:in `test_two_plus_two': undefined method `assert' for #<MathTest:0x00007f8eeb0f4ef8> (NoMethodError)
This is just what we've expected since we didn't define
the assert method yet. Let's add it:
However, this is not precisely how Minitest works. Minitest doesn't magically add code to the bottom
of our tests to print its report. Remember, we require
Minitest before our test code, so we need to keep our "test library"
code before our tests.
We need to print the report when everything else has finished
executing, and luckily Ruby comes with the Kernel#at_exit
method we can use for this:
Which uses at_exit to instruct Minitest to run after all the other code has been executed and the program is exiting. The first line skips invoking Minitest if
the exception is raised and it's not SystemExit with zero status. [2]
This hook is only set up if @@installed_at_exit has not been set, ensuring
the hook will only be set up once. This allows requiring minitest/autorun multiple
times and not having to worry about what will happen with the at_exit hook (imagine
multiple test files each requiring minitest/autorun).
Notice the little trick with exit_code being set to nil first, before
assigning it to the result of Minitest.run. This is used to ensure that
the at_exit block runs regardless of the result of Minitest.run.[3]
This second at_exit hook will be executed after Minitest finishes its execution,
so this is setting up the second layer of code that's going to be run
when the program is exiting.
In its block @@after_run is being called in reverse order. Where is it coming from?
### A simple hook allowing you to run a block of code after everything# is done running. Eg:## Minitest.after_run { p $debugging_info }defself.after_run&block@@after_run<<blockend
Nice, so this is how Minitest keeps track of its after_run callbacks, it appends blocks
passed to Minitest.after_run to an ordinary array - nothing magical.
Now that we've demystified other parts of the at_exit hook that Minitest uses
to deploy its magic, let's take a look at the one thing that's left:
### This is the top-level run method. Everything starts from here. It# tells each Runnable sub-class to run, and each of those are# responsible for doing whatever they do.## The overall structure of a run looks like this:## Minitest.autorun# Minitest.run(args)# Minitest.__run(reporter, options)# Runnable.runnables.each# runnable.run(reporter, options)# self.runnable_methods.each# self.run_one_method(self, runnable_method, reporter)# Minitest.run_one_method(klass, runnable_method)# klass.new(runnable_method).rundefself.runargs=[]self.load_pluginsunlessargs.delete("--no-plugins")||ENV["MT_NO_PLUGINS"]options=process_argsargsreporter=CompositeReporter.newreporter<<SummaryReporter.new(options[:io],options)reporter<<ProgressReporter.new(options[:io],options)self.reporter=reporter# this makes it available to pluginsself.init_pluginsoptionsself.reporter=nil# runnables shouldn't depend on the reporter, everself.parallel_executor.startifparallel_executor.respond_to?(:start)reporter.startbegin__runreporter,optionsrescueInterruptwarn"Interrupted. Exiting..."endself.parallel_executor.shutdownreporter.reportreporter.passed?end
### Internal run method. Responsible for telling all Runnable# sub-classes to run.defself.__runreporter,optionssuites=Runnable.runnables.reject{|s|s.runnable_methods.empty?}.shuffleparallel,serial=suites.partition{|s|s.test_order==:parallel}serial.map{|suite|suite.runreporter,options}+parallel.map{|suite|suite.runreporter,options}end
So far so good, but we're little confused now about this Runnable.runnables invocation.
It looks like it's an array, but where did it come from?
Ah! It uses the Class#inherited to track all
the classes that have inherited Runnable. It's a bit weird to discover that @@runnables is being
appended to, but not assigned to an array yet. Here's the missing piece:
This clears things up about Runnable.runnables being an array of classes that inherit
Runnable, but we still know nothing about the nature of those classes.
Doing a quick grep for < Runnable gives us a suspect:
### Internal run method. Responsible for telling all Runnable# sub-classes to run.defself.__runreporter,optionssuites=Runnable.runnables.reject{|s|s.runnable_methods.empty?}.shuffleparallel,serial=suites.partition{|s|s.test_order==:parallel}serial.map{|suite|suite.runreporter,options}+parallel.map{|suite|suite.runreporter,options}end
But we now know (or do we, cough, cough?) that Runnable.runnables is [Result]. Looks like
we're rejecting classes from that array that have
empty runnable_methods.
Tracking this method down leads us to the dead end:
### Each subclass of Runnable is responsible for overriding this# method to return all runnable methods. See #methods_matching.defself.runnable_methodsraiseNotImplementedError,"subclass responsibility"end
Turns out, this nifty re-opening
of Runnable class has a purpose. The Runnable.inherited is defined
on line 977,
while Result < Runnable happens on line 496. Looks like the order is important here. Who knew!?
# This happens firstclassRunnabledefself.reset# :nodoc:@@runnables=[]endresetdefself.runnables@@runnablesendendclassResult<Runnable;end# And then we re-open the classclassRunnable# re-opendefself.inheritedklass# :nodoc:self.runnables<<klasssuperendend# > Runnable.runnables# => []
Our suspect is free to go. We have found another one:
Ah! That's the last line of that file, so the inherited will kick in and catch this class. This suspect is now confirmed, and
we can happily declare that Runnable.runnables contains [Test].
### Returns all instance methods starting with "test_". Based on# #test_order, the methods are either sorted, randomized# (default), or run in parallel.defself.runnable_methodsmethods=methods_matching(/^test_/)caseself.test_orderwhen:random,:parallelthenmax=methods.sizemethods.sort.sort_by{randmax}when:alpha,:sortedthenmethods.sortelseraise"Unknown test_order: #{self.test_order.inspect}"endend
This returns all the instance methods of this class, that start with test_ by
using the Runnable#methods_matching:
### Internal run method. Responsible for telling all Runnable# sub-classes to run.defself.__runreporter,optionssuites=Runnable.runnables.reject{|s|s.runnable_methods.empty?}.shuffleparallel,serial=suites.partition{|s|s.test_order==:parallel}serial.map{|suite|suite.runreporter,options}+parallel.map{|suite|suite.runreporter,options}end
It looks like some partitioning is happening, but we don't care because:
### Defines the order to run tests (:random by default). Override# this or use a convenience method to change it for your tests.defself.test_order:randomend
Which will put everything in serial (parallel will be empty array) and this leads us to:
### Internal run method. Responsible for telling all Runnable# sub-classes to run.defself.__runreporter,optionssuites=Runnable.runnables.reject{|s|s.runnable_methods.empty?}.shuffleparallel,serial=suites.partition{|s|s.test_order==:parallel}serial.map{|suite|suite.runreporter,options}+parallel.map{|suite|suite.runreporter,options}end
Which finally calls Test.run, which Test inherits from Runnable:
### Responsible for running all runnable methods in a given class,# each in its own instance. Each instance is passed to the# reporter to record.defself.runreporter,options={}filter=options[:filter]||"/./"filter=Regexp.new$1iffilter=~%r%/(.*)/%filtered_methods=self.runnable_methods.find_all{|m|filter===m||filter==="#{self}##{m}"}exclude=options[:exclude]exclude=Regexp.new$1ifexclude=~%r%/(.*)/%filtered_methods.delete_if{|m|exclude===m||exclude==="#{self}##{m}"}returniffiltered_methods.empty?with_info_handlerreporterdofiltered_methods.eachdo|method_name|run_one_methodself,method_name,reporterendendend
This filters methods by the --name and --exclude options
provided from the command line. Let's get to the juicy part:
### Responsible for running all runnable methods in a given class,# each in its own instance. Each instance is passed to the# reporter to record.defself.runreporter,options={}filter=options[:filter]||"/./"filter=Regexp.new$1iffilter=~%r%/(.*)/%filtered_methods=self.runnable_methods.find_all{|m|filter===m||filter==="#{self}##{m}"}exclude=options[:exclude]exclude=Regexp.new$1ifexclude=~%r%/(.*)/%filtered_methods.delete_if{|m|exclude===m||exclude==="#{self}##{m}"}returniffiltered_methods.empty?with_info_handlerreporterdofiltered_methods.eachdo|method_name|run_one_methodself,method_name,reporterendendend
### Runs a single method and has the reporter record the result.# This was considered internal API but is factored out of run so# that subclasses can specialize the running of an individual# test. See Minitest::ParallelTest::ClassMethods for an example.defself.run_one_methodklass,method_name,reporterreporter.prerecordklass,method_namereporter.recordMinitest.run_one_method(klass,method_name)end
Which passes the potato to Minitest.run_one_method:
defself.run_one_methodklass,method_name# :nodoc:result=klass.new(method_name).runraise"#{klass}#run _must_ return a Result"unlessResult===resultresultend
This line creates a new instance of Minitest::Test class and passes the current method name as the first argument. Minitest::Test inherits initialize from Minitest::Runnable which looks like this:
So every test method we've created in our original test file gets its own Minitest::Test instance, which stores the name, failures, and number of assertions. A much better approach, compared to our hack from the beginning, but there are some similarities.
The final layer
We still have one layer before we reach the end since we're calling
run on an instance of Minitest::Test.
### Runs a single test with setup/teardown hooks.defrunwith_info_handlerdotime_itdocapture_exceptionsdobefore_setup;setup;after_setupself.sendself.nameendTEARDOWN_METHODS.eachdo|hook|capture_exceptionsdoself.sendhookendendendendResult.fromself# per contractend
Let's highlight the most important line in that method.
### Runs a single test with setup/teardown hooks.defrunwith_info_handlerdotime_itdocapture_exceptionsdobefore_setup;setup;after_setupself.sendself.nameendTEARDOWN_METHODS.eachdo|hook|capture_exceptionsdoself.sendhookendendendendResult.fromself# per contractend
This calls our original test method named test_two_plus_two. The name was stored in Minitest::Runnable#initialize, as explained earlier.
Other parts of this method are pretty self-explanatory, and I will not go into details. The goal of this article is achieved since we have tracked down and found out exactly what happens when we run our Minitest tests.
Hooray!
Notes and examples
The commit 1f2b132 is no longer
the latest commit in the repository. The first draft of this article was written 8 weeks ago, and I apologize for not publishing
it sooner.
This means it will
be skipped for any exceptions except exit 0 which raises SystemExit with success?
set to true. You can test this with the following code:
require'minitest/autorun'exit0
Which will result in the standard Minitest report being printed out. Try replacing
the exit status like this:
require'minitest/autorun'exit1
And notice that the Minitest report was not printed out. The same thing happens
if you raise an exception.