JetBlack
A black-box testing utility for command line tools and gems. Written in Ruby, with RSpec in mind (but it's test framework agnostic). Features:
- Each session takes place within a unique temporary directory, outside the project
- Synchronously run commands then write assertions on:
- The
stdout/stderrcontent - The exit status of the process
- The
- Exercise interactive command line interfaces
- Manipulate files in the temporary directory:
- Create files
- Create executable files
- Append content to files
- Copy fixture files from your project
- Modify the environment without changing the parent test process:
- Override environment variables
- Escape the current Bundler context
- Adjust
$PATHto include your executable / Subject Under Test
- RSpec matchers (optional)
The temporary directory is discarded after each spec. This means you can write &
modify files and run commands (like git init) without worrying about tidying
up after or impacting your actual project.
Setup
gem "jet_black"
end
RSpec setup
If you're using RSpec, you can load matchers with the following require (optional):
require "jet_black/rspec"
Any specs you write in the spec/black_box folder will then have an inferred
:black_box meta type, and the matchers will be available in those examples.
Manual RSpec setup
Alternatively you can manually include the matchers:
require "jet_black"
require "jet_black/rspec/matchers"
RSpec.describe "my command line tool" do
include JetBlack::RSpec::Matchers
end
Usage
Running commands
session = JetBlack::Session.new
result = session.run("echo foo")
result.stdout # => "foo\n"
result.stderr # => ""
result.exit_status # => 0
Providing stdin data:
session.run("./hello-world", stdin: "Alice")
Running interactive commands
result = session.run_interactive("./hello-world") do |terminal|
terminal.expect("What's your name?", reply: "Alice")
terminal.expect("What's your location?", reply: "Wonderland")
end
expect(result.exit_status).to eq 0
expect(result.stdout).to eq <<~TXT
What's your name?
Alice
What's your location?
Wonderland
Hello Alice in Wonderland
TXT
If you don't want to wait for a process to finish, you can end the interactive session early:
result = session.run_interactive("./long-cli-flow") do |terminal|
terminal.expect("Question 1", reply: "Y")
terminal.end_session(signal: "INT")
end
File manipulation
session.create_file "file.txt", <<~TXT
The quick brown fox
jumps over the lazy dog
TXT
session.create_executable "hello-world.sh", <<~SH
#!/bin/sh
echo "Hello world"
SH
session.append_to_file "file.txt", <<~TXT
shiny
new
lines
TXT
# Subdirectories are created for you:
session.create_file "deeper/underground/jamiroquai.txt", <<~TXT
I'm going deeper underground, hey ha
There's too much panic in this town
TXT
Copying fixture files
It's ideal to create pertinent files inline within a spec, to provide context for the reader, but sometimes it's better to copy across a large or non-human-readable file.
-
Create a fixture directory in your project, such as
spec/fixtures/black_box. -
Configure the fixture path in
spec/support/jet_black.rb:require "jet_black"
JetBlack.configure do |config|
config.fixture_directory = File.expand_path("../fixtures/black_box", __dir__)
end -
Copy fixtures across into a session's temporary directory:
session = JetBlack::Session.new
session.copy_fixture("src-config.json", "config.json")
# Destination subdirectories are created for you:
session.copy_fixture("src-config.json", "config/config.json")
Environment variable overrides
result = session.run("printf $FOO", env: { FOO: "bar" })
result.stdout # => "bar"
Provide a nil value to unset an environment variable.
Clean Bundler environment
If your project's test suite is invoked with Bundler (e.g. bundle exec rspec)
but you want to run commands like bundle install and bundle exec with a
different Gemfile in a given spec, you can configure the session or individual
commands to run with a clean Bundler environment.
Per command:
session.run("bundle install", options: { clean_bundler_env: true })
Per session:
session.run("bundle install")
session.run("bundle exec rake")
$PATH prefix
Given the root of your project contains a bin directory containing
my_awesome_bin.
Configure the path_prefix to the directory containing your executable(s):
require "jet_black"
JetBlack.configure do |config|
config.path_prefix = File.expand_path("../../bin", __dir__)
end
Then the $PATH of each session will include the configured directory, and your
executable should be invokable:
RSpec matchers
Given the RSpec setup is configured, you'll have access to the following matchers:
have_stdoutwhich accepts a string or regular expressionhave_stderrwhich accepts a string or regular expressionhave_no_stdoutwhich asserts thestdoutis emptyhave_no_stderrwhich asserts thestderris empty
And the following predicate matchers:
be_a_success/be_successasserts the exit status was zerobe_a_failure/be_failureasserts the exit status was not zero
Example assertions
RSpec.describe "my command line tool" do
let(:session) { JetBlack::Session.new }
it "does the work" do
expect(session.run("my_tool --good")).
to be_a_success.and have_stdout(/It worked/)
end
it "explodes with incorrect arguments" do
expect(session.run("my_tool --bad")).
to be_a_failure.and have_stderr("Oh no!")
end
end
However these assertions can be made with built-in matchers too:
let(:session) { JetBlack::Session.new }
it "does the work" do
result = session.run("my_tool --good")
expect(result.stdout).to match(/It worked/)
expect(result.exit_status).to eq 0
end
it "explodes with incorrect arguments" do
result = session.run("my_tool --bad")
expect(result.stderr).to match("Oh no!")
expect(result.exit_status).to eq 1
end
end
More examples
- JetBlack's own higher-level tests
- A more complex scenario testing a gem in a fresh Rails app. Shows how to:
- Include the gem-under-test via the Rails app's Gemfile
- Use a clean Bundler environment to use the Gemfile of the new Rails app (instead of the Bundler context of the gem's test suite)