Clojure Tip: defspec

tl;dr test.check’s defspec lets you call the thusly defined tests like functions.

I’ve been working through Eric Normand’s excellent test.check courses to get a better handle on property-based testing (PBT). One of the great things about test.check (and most other PBT libraries) is that when you run into a test failure, the library will “shrink” the failing test input into the smallest version that still reproduces the failure. This is a boon for debugging, because you can take that shrunken input and feed it back into your code to find out what’s happening.

One of the other great things about test.check is that, since it’s a Clojure library, you get the test results as data (via a map), and can thus extract and use the information easily.

The tests I was writing were using test.check’s clojure.test integration, specifically the defspec macro. When I ran the tests, the test output was displayed, but I couldn’t get my grubby mitts on it without copying and pasting. That’s living like an animal.

So I read the docs, because there had to be something I could do. Here’s what defspec’s docstring says (emphasis added):

That’s right: not only do you get easy integration with Clojure’s testing machinery, you can call the test as a function and get the result data directly. Here’s a tiny example:

(ns foo
  (:require [clojure.string :as str]
            [clojure.test.check.clojure-test :refer [defspec]]
            [clojure.test.check.generators :as gen]
            [clojure.test.check.properties :as prop]))

(defn bad-even?
  "A terrible implementation of `even?` that
   returns true if the number contains the
   digit 3."
  [n]
  (if (str/index-of (str n) \3)
    true
    (zero? (mod n 2))))

(defspec even-returns-false-for-odd-numbers
  (prop/for-all [n (gen/fmap
                     #(-> % (* 2) inc)
                     gen/small-integer)]
    (false? (bad-even? n))))

Let’s say I run those tests. I’ll get some output similar to this:

Testing foo
{:type :clojure.test.check.clojure-test/shrunk, :clojure.test.check.clojure-test/property #clojure.test.check.generators.Generator{:gen #object[clojure.test.check.generators$gen_fmap$fn__443 0x7affc159 "clojure.test.check.generators$gen_fmap$fn__443@7affc159"]}, :clojure.test.check.clojure-test/params [-3]}

FAIL in (even-returns-false-for-odd-numbers) (foo.clj:12)
expected: {:result true}
  actual: {:shrunk {:total-nodes-visited 2, :depth 0, :pass? false, :result false, :result-data nil, :time-shrinking-ms 1, :smallest [-3]}, :failed-after-ms 3, :num-tests 5, :seed 1575309899069, :fail [-3], :result false, :result-data nil, :failing-size 4, :pass? false, :test-var "even-returns-false-for-odd-numbers"}

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
{:test 1, :pass 0, :fail 1, :error 0, :type :summary}

If you look at actual, you’ll see a handy map with a lot of details about the failing test. (In particular, that juicy smallest failing input!) You could just copy and paste it outta there … but why not just use the map directly? You could do something like this:

(let [result (even-returns-false-for-odd-numbers)
  (when-not (:pass? result)
    (let [input (-> result :shrunk :smallest first)]
      (bad-even? input))))

It’s a handy feature that helps shorten that feedback loop just a little bit more, and goes hand-in-hand with REPL-driven development.