In a recent project failing cucumber scenarios due to unreliable environments were a very common occurrence. I ran a set of tests and there were failures, then I ran them again a minute a later and they were all green! This became very frustrating. These failures where normally in the form of timeouts due to an ever increasing load on the app box. Obviously my first reaction was to speak to the team responsible for the environment and try to get the problems resolved but due to limited time and resource nothing could be done immediately.
So while I was waiting for further hardware to be thrown at the environment I decided to use the cucumber rerun formatter.
For this demonstration we need to have a feature which has a scenario that passes and a scenario that fails.
Feature: Rerun formatter Scenario: Test that Passes Given I have 1 pound When I add 2 pounds Then I should see 3 pounds Scenario: Test that Fails Given I have 1 pound When I add 2 pounds Then I should see 4 pounds
So lets try and rerun that failed scenario. On the command line add the formatters pretty and rerun so it looks like this:
$ cucumber features/retest.feature -f pretty -f rerun --out rerun.txt
After you have executed this you will have noticed that a rerun.txt file has appeared in the root directory of your project, if you open that you can see that the failed test has been recorded, now we need to rerun that failed test.
$ cucumber features/retest.feature -f pretty -f rerun --out rerun.txt || cucumber @rerun.txt
Notice that the cucumber will run through the 2 scenarios as normal then it will rerun the failed scenario. The || means that the cucumber will execute the rerun text file if a failure has occurred, if you replace the || with ; cucumber will execute the rerun every time.
$ cucumber features/retest.feature -f pretty -f rerun --out rerun.txt ; cucumber @rerun.txt
If you are experiencing problems with failing tests this rerun should not be the first thing you implement. Investigate the reason for the failures and try your upmost to get them fixed. I would only recommend using rerun for this kind of scenario if all else has failed.