BDD with Cucumber and rspec - when is this redundant?

views:

343

answers:

+15 Q:

BDD with Cucumber and rspec - when is this redundant?

A Rails/tool specific version of: How deep are your unit tests?

Right now, I currently write:

Cucumber features (integration tests) - these test against the HTML/JS that is returned by our app, but sometimes also tests other things, like calls to third-party services.
RSpec controller tests (functional tests), originally only if the controllers have any meaningful logic, but now more and more.
RSpec model tests (unit tests)

Sometimes this is entirely necessary; it is necessary to test behavior in the model that is not entirely obvious or visible to the end-user. When models are complex, they should definitely be tested. But other times, it seems to me the tests are redundant. For instance, do you test method foo if it is only called by bar, and bar is tested? What if bar is a simple helper method on a model that is used by and easily testable in a Cucumber feature? Do you test the method in rspec as well as Cucumber? I find myself struggling with this, as writing more tests take time and maintaining multiple "versions" of what is effectively the same behaviors, which makes maintaining the test suite more time intensive, which in turn makes changes more expensive.

In short, do you believe there is there a time when writing only Cucumber features is enough? Or should you always test at every level? If you think there is a grey area, what is your threshold for "this needs a functional/unit test." In practical terms, what do you do currently, and why (or why not) do you think it's sufficient?

EDIT: Here's an example of what might be "test overkill." Admittedly, I was able to write this pretty quickly, but it was completely hypothetical.

I test complex model/lib methods with rspec then the main business logic in web with cucumber, so I'm sure that the main features of the web will work 100%, then if I got more time and resources I test everything else.

Gacha 2010-09-18 08:03:26

I have the same feelings, but I would like to hear more about your approach. How would you convince others that this is the right way to do it?

wuputah 2010-09-18 21:32:43

The main reason I do that is because I can't test it all, so I need to prioritize and test from more important to less. To cover more of web page I suggest to create simple specs for all models to see if everything saves/validates, then specs for all complex methods, and at last cucumber for all forms, main views. And then if you have more time/resources you can test not so important things.

Gacha 2010-09-27 12:27:22

+1 A:

Rails has a well-tested codebase, so I'd avoid re-testing stuff that is covered in those steps.

For example, unless it is custom code, it is pointless to test the results of validations at unit and functional levels. I'd test them at the integration level though. Cucumber features act as specifications for your project, so it is good to specify that you need a validation for x and y, even if the implementation is a single line declaration in the model.

edgerunner 2010-09-18 11:26:04

Yes, but: while it's easy enough with Shoulda to test validations in rspec, do you think you should test validations in both Cucumber and rspec? The point is not that things should be or not be tested, it's whether things should be tested at multiple levels. Validations aren't exactly what I was thinking of (particularly since Shoulda makes it so easy), but it's a reasonable example. Pretend that you had to write those rspec tests by hand!

wuputah 2010-09-18 21:24:19

To put it another way, what I meant was that you should test your original code at all levels, and test everything at the top level. Consider ActiveRecord lifecycle callbacks for example. You need not worry about if the callbacks are called on save. You need to worry about if the result you expect from that callback is on the record or not.

edgerunner 2010-09-18 23:30:58

Its easier to write simple specs for simple methods. Its much easier than writing cukes.

If you keep your methods simple - and keep your specs simple - by testing only the logic inside that method - you will find joy in unit testing.

If anything is redundant its cucumber tests. If you have well tested models and lib your software should work.

Vojto 2010-09-22 03:31:18

Until you realise that you forgot to put a submit button on your form and you can't get from A -> B in your app. The methods in question may work in isolation and all your unit tests pass, but when used in a real world situation, your application will not work. Integration tests are just as important as unit tests imo and should not be considered redundant.

Sidane 2010-09-22 11:46:55

Yes, that's one perspective.

Vojto 2010-09-22 16:34:29

The question is, to a degree, subjective, and one about best practices. I definitely understand the idea that integration tests can seem redundant if you have good compartmentalized tests. I am mostly coming from the other angle, that given integration tests exist, the rest can seem redundant at times. I personally think this is a better approach since, as Sidane points out, you can miss the "big picture" if you dont have integration tests. I think both opinions have merit, though.

wuputah 2010-09-23 14:55:52

I strongly disagree with declaration that having only integration tests is better approach. Integration testing just ensures the whole thing works. Unit testing on the other hand, if done correctly (test-first), ensures high quality of code.

Vojto 2010-09-24 15:11:34

Not _only_ integration tests, but putting the emphasis there - start at the outside and go in, instead of starting at the inside and going out.

wuputah 2010-09-27 15:34:22

+9 A:

Good question, one I've grappled with recently while working on a Rails app, also using Cucumber/RSpec. I try to test as much as possible at every level, however, I've also found that as the codebase grows, I sometimes feel I'm repeating myself needlessly.

Using "Outside-in" testing, my process usually goes something like: Cucumber Scenario -> Controller Spec -> Model Spec. More and more I find myself skipping over the controller specs as the cucumber scenarios cover much of their functionality. I usually go back and add the controller specs, but it can feel like a bit of a chore.

One step I take regularly is to run rcov on my cucumber features with rake cucumber:rcov and look for notable gaps in coverage. These are areas of the code I make sure to focus on so they have decent coverage, be it unit or integration tests.

I believe models/libs should be unit tested extensively, right off the bat, as it is the core business logic. It needs to work in isolation, outside of the normal web request/response process. For example, if I'm interacting with my app through the Rails console, I'm working directly with the business logic and I want the reassurance that methods I call on my models/classes are well tested.

At the end of the day, every app is different and I think it's down to the developer(s) to determine how much test coverage should be devoted to different parts of the codebase and find the right balance so that your test suite doesn't bog you down as your app grows.

Here's an interesting article I dug up from my bookmarks that is worth reading: http://webmozarts.com/2010/03/15/why-not-to-want-100-code-coverage/

Sidane 2010-09-22 13:34:59

Good link + didn't know about `rake cucumber:rcov`. Out of curiosity, do you unit test every model method, even simple things that have cucumber coverage? (See my example in the question) Do you consider it essential to the point where you won't consider code "finished" (i.e. not allowed to be committed, or merged to master) until it is tested at that level?

wuputah 2010-09-23 14:58:00

I usually aim for unit test coverage of most models/libs. It doesn't always happen though, when I'm in full flow, writing code, sometimes I end up taking short-cuts. I try to regularly check for noticeable gaps in my test coverage, be it manually or with tools like rcov, and then plug those gaps. And I don't stick to a hard and fast rule like not committing code untested at the unit level. Simple boolean methods like in your example I would not be concerned about if they didn't have unit test coverage, but methods with any sort of complexity I try to test.

Sidane 2010-09-23 16:00:51

ansaurus

tags:

views:

answers:

BDD with Cucumber and rspec - when is this redundant?

related questions