Testing with random values: is it really dangerous?

Last Saturday I presented a session about automation at the Winnipeg Code Camp.

I’ve been doing presentations a the WPG CC for the past three years, and is always a blast! Great sessions, lots of networking and great food is an awesome combination!

After my session, some of the attendes approached me and started to talk about CI, their gotchas about testing and woes .

One of them mentioned that he wasn’t sure about the way they were using “hardcoded” values with the tests. He felt that probably should be a better way of defining valid inputs that could avoid the “smell”.

My suggestions was to consider using auto generated values. Random values are a great tool to avoid hardcoding inputs. Using “factories” to create valid test values helps us to make sure we have better coverage of the domain of valid inputs for our tests.

How do I know what input is valid?

You may wonder “But, if the values are random, the test may pass once and then fail at runtime, that’s really, really, dangerous!!!”.

The answer is No, not at all. The key is to identify the domain that makes your test pass and use the random generation tools to get values that are part of that domain. How to do that? Keep reading …

When you write a unit test you may have more than one scenario. For each scenario you need to find a set of inputs that make it pass.

For example, let’s use a method that verifies if a string is a palindrome. Definition of palindrome in wikipedia:

A palindrome is a word, phrase, number or other sequence of units that can be read the same way in either direction

Our method may look something like this in C#:

bool IsPalindrome(string word)

So now, what should we do? Write one test that takes random strings as input, run it once and if we are lucky and passes then we are done? Woah, slow down your horses….. take a deep breath… maybe two….

Preconditions, postconditions and domains

The precondition of the method is the set of values that are valid in order to call the method. Valid means that we don’t get an exception when we use it, no matter what the result of the method is.

So, what are the values that are valid to use for IsPalindrome? It seems to me that any string should work, except null, therefore:

• Precondition: Any not null string instance.

Great! Now let’s think about the postcondition and discover our scenarios. The postcondition defines what are the expected results and under what conditions are calculated.

Following our definition of what the method does, we can have two possible outputs:

• True: if the input is palindrome
• False: if the input is not palindrome

Thus, we have two scenarios:

• When the word is palindrome it should return true
• When the word is not palindrome it should return false

Writing the test

We need to test two scenarios and the best way of doing that is to write one test per scenario.

The first test should check the first part of the postcondition, I want to get true as result, and to do that the domain I need to use would be all the words that actually are palindrome. Let’s write the scenario:

Given I have a palindrome phrase

When I check IsPalindrome

Then I should get true

I like to use the Given-When-Then syntax, it’s very descriptive. I’m using the MT Testing library that enforces GWT. Here is the scenario using C# and just one example:

protected override void GivenThat()
{
    this._phrase = "Madam Im adam";
}

protected override void WhenIRun()
{
   this.Actual = PalindromeChecker.IsPalindrome(this._phrase);
}

[It]
public void Should_validate_the_phrase_is_palindrome()
{
    this.Actual.Should().Be.True();
}

The test looks good, a bit of TDD and all green. Don’t worry about the implementation, you can download all the source and check it out at the later .

Using Factories

I’d like to make sure that for any palindrome phrase (that’s the domain we got from the pre and post condition) the test passes, no just for one example.

However, though generating all the palindrome words is a bit ambitious, we could settle for generating a collection of phrases that are palindrome and use that as input.

MbUnit provides an attribute named “Factory” that allows a static method to define a collection of values to be passed to the test, let’s look at the code:

        private readonly string _phrase;

        [Factory("PalindromeFactory")]
        public When_palindrome_checker_checks_a_palindrome_phrase(string phrase)
        {
            _phrase = phrase;
        }

        protected override void WhenIRun()
        {
            this.Actual = PalindromeChecker.IsPalindrome(this._phrase);
        }

        [It]
        public void Should_check_the_word_as_palindrome()
        {
            this.Actual.Should().Be.True();
        }

        protected static IEnumerable<string> PalindromeFactory()
        {
            yield return "Madam I'm adam";
            yield return "Draw pupil's lip upward";
            yield return ....
        }

Using the attribute, we indicate the runner that the test should be run for every value in the collection.

We run the test, and we can see in the result window something like:

### Step When_palindrome_checker_checks_a_palindrome_phrase("Madam I\’m adam"): passed ###
### Step When_palindrome_checker_checks_a_palindrome_phrase("Draw pupil\’s lip upward"): passed ###
### Step When_palindrome_checker_checks_a_palindrome_phrase("Gateman sees name, garageman sees name tag"): passed ###
### Step When_palindrome_checker_checks_a_palindrome_phrase("Go hang a salami; I\’m a lasagna hog"): passed ###
### Step When_palindrome_checker_checks_a_palindrome_phrase("I roamed under it as a tired, nude Maori"): passed ###
### Step When_palindrome_checker_checks_a_palindrome_phrase("Live not on evil"): passed ###

Using random strings

One case covered, one to go.

The second test should verify the second part of the postcondition. I want to get false as result, so the domain I need to use is all the words that are not palindrome. Let’s write the scenario:

Given I don’t have palindrome phrase

When I check IsPalindrome

Then I should get false

And the code:

        private string _phrase;

        protected override void GivenThat()
        {
            this._phrase = "This is not palindrome";
        }

        protected override void WhenIRun()
        {
            this.Actual = PalindromeChecker.IsPalindrome(this._phrase);
        }

        [It]
        public void Should_not_validate_the_phrase_as_palindrome()
        {
            this.Actual.Should().Be.False();
        }

The code works fine. However we have a similar situation as before, I’d like to have more values that belong to the domain that makes the method fail, that means any phrase that is not palindrome.

Factories are a good idea when we have a set of values that we want to use. In this case I don’t want to hardcode non palindrome phrases. I would prefer to have a way of creating those phrases without needing to enumerate all the examples.

Luckily MbUnit has another attribute that we can use to generate random strings. It’s called RandomStrings and it takes a regular expression to describe the string. Adding the attribute the code looks like this:

        private readonly string _phrase;

        public When_palindrome_checker_checks_a_non_palindrome_phrase(
            [RandomStrings(Count=10, Pattern=@"Weird [A-Z]{5,8} [0-9]{2}")]string phrase)
        {
            _phrase = phrase;
        }

        protected override void WhenIRun()
        {
            this.Actual = PalindromeChecker.IsPalindrome(this._phrase);
        }

        [It]
        public void Should_not_validate_the_phrase_as_palindrome()
        {
            this.Actual.Should().Be.False();
        }

I’m using a regular expression that uses numbers at the end to make sure the phrase is not palindrome.

We run the tests, and here is the output:

### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird VGSKTTZ 64"): passed ###
### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird VLMEWIJ 59"): passed ###
### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird NDICJK 08"): passed ###
### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird FEMBMSKK 88"): passed ###
### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird IHOIOWN 38"): passed ###
### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird ITIRCHAA 63"): passed ###

The moral of the story

Random values are a powerful tool, not dangerous per se. They are part of our toolbox as developers. Can we use it wrong? Sure, but that’s a completely different story ….

If you want to get the full code, download it from the examples in MT Testing. Check the Palindrome folder.

Do you have any other stories or comments about random values that you would like to share, please leave a comment, feedback is more than welcome.

Want to see an example with major complexity? Something that fits better your case? Tell me about it and I’ll add it to the MT Testing framework as an example of usage (and I’ll give you credit for it too!).