An Introduction to Web Test Automation with Selenium and Ruby

An Introduction to Web Test Automation with Selenium and Ruby

As readers of this blog and other random ramblings of mine across the internet will know - for more than 8 months or so I’ve been working a lot with Selenium and Ruby. It’s been a joy and a pain at the same time. Beguiled by how easy the Selenium tools were to use I got started only to discover that they weren’t as accessible as they at first appeared. A definite case of ‘easy when you know how’.

It seems there are a good number of folks out there using the tools to varying degrees and different ways which unfortunately adds to the confusion. It confuses because most folks taking the first steps with the Selenium toolset soon hit their first issues which seems commonly to be how to set the toolset up, which language to employ and even how to get over the many gotcha’s that Selenium has waiting for us.

Trawling over the forums, not just the smaller community forums either but the ‘official’ forums too, was not that helpful overall. Perhaps that’s because I was intent on asking very specific questions and/or my focus was on using Ruby. Maybe if I just wanted to use Java or buy someone’s course I’d have had better luck. In any case the lack of answers to my and other’s questions where the questions were something less than trivial was frankly disappointing. That is disappointing in a way that I feel folks are purposefully withholding the knowledge they have. There is a possibility that no one really knows too, but over five of the most trafficked forums? I doubt that. As if to confirm my deluded suspicions I even managed to find someone I would have considered a ‘Selenium Expert’ posting that they’d stopped posting - because the forums were full of ‘problem posters’. Well, I hate to spring a shock fest on you but that’s kind of the idea of public forums most of the time. FFS, can you say ‘testing community’?

So where to turn to if I’m a Selenium newbie wanting a clear idea of how to go about setting my Selenium web test framework up? Hoping for insight into other people’s success spelt out to such a degree that I can follow it step by step or translate it into my environment? I could try the official docs at Selenium HQ but it’s hardly an idiots guide, as good as they are getting as they become more complete. Where’s the guided, instructional, friendly, tutorial and walkthrough with helpful pictures and handy hints? They’re rhetorical questions and the reasons I wrote “An Introduction to Web Test Automation with Selenium and Ruby”.

Is the book an answer to all problems and perspectives you might want answers for? Not a chance. This book is an introduction to not a definitive guide and that’s the point. It’s written so that within a week the reader has a fully working Selenium-Ruby automation framework up and running which includes key process techniques for managing how it’s used. The book covers the Selenium tools of the IDE, Remote Control and Grid, provides numerous examples of the Selenium-Client keywords and commands in use, introduces the Ruby language and techniques for designing automated test scripts, it walks through the creation of 10 key test script templates and discusses ideas on managing test suites and test environments.

An Introduction to Web Test Automation with Selenium and Ruby is the book I was looking for 8 months ago and that I’ve written to capture the learning I’ve experienced bringing the framework to life on it’s now multiple live deployments. It’s my attempt to help other’s avoid wasting time that they don’t have and which needs to be focused on delivering great testing. There’s a lot to learn and it’s insane to think every tester wanting to do so is scrabbling around on forums and sites trying to piece the bits together. This book is a vehicle to kick start a more open and robust form of discussion and sharing by others in the community. That discussion will be of critical value and importance as there are a myriad of ways to implement the Selenium tools and the testing community needs to hear what they are.

Book Proposal
Having got just 2/3 the way through writing the book I decided it was time to start sending out the draft manuscript and see what publishers make of it. The first to get site of the manuscript is www.pragprog.com and it should hopefully be reviewed this week. Let’s see what the feedback is. Once I know where things are headed I’ll check in and see what I’m ‘allowed’ to share in advance of any publication.

So please do me and the Selenium would-be test community a favour and go put the word out, An Introduction to Web Test Automation with Selenium and Ruby, is out looking for a publisher.

Mark.

Behaviour Driven Development (BDD) and the Testing Profession

I posted earlier in the year about BDD. If you’ve not encountered BDD have a wander over to http://behaviour-driven.org/ and take a quick read or hit YouTube and watch Dave Astels (http://techblog.daveastels.com/) Google TechTalk http://www.youtube.com/watch?v=oOFfHzrIDPk. BDD isn’t some kind of Flavour of the Week new thing, it’s been around a good few years so hitting Google or Bing will turn up even more material.

I've read about approaches such as Agile Acceptance Testing (http://snipurl.com/pio2q) too and they offer good ideas about focusing on what’s important to the customer. What really struck home for me about BDD is the focus on Behaviour. As Michael Bolton said “... our clients don't value the code as such; they value the things the code does for them."

Behaviour Driven Testing (BDT)
There is a slight ‘gap’ in all this BDD goodness though and that’s the integration of the testing world’s perspective and practices. Especially those practices that could be applied, as is or modified, to fit alongside a team practicing BDD. AS much as I have the warm fuzzies for BDD it comes across as a far too developer centric perspective yet again. Maxim No. 27: ‘Software Development is more than just the writing of code’. What’s more, I’m a tester, as such it'd be odd to talk about me doing 'development' and having to say 'obviously I mean testing...' every time I mention the word development.

Hence why I don’t talk about BDD as such, I talk about Behaviour Driven Testing 'BDT', a methodology I developed to articulate how a test team could work in a BDD environment and consider what complimentary techniques/approaches might look like. If we’re enlightened enough to realise that ‘development’ is more than writing code and is all the tasks needed (BA, PM, Dev, Test, Support, ...) to get code out of the door then perhaps we could think just BDD but the world is a more fragmented and messy place so unfortunately we talk as if development just means writing code.

Replacing, renaming BDD?
I recently exchanged messages with David Chelimsky (http://www.davidchelimsky.net/) (and Bret Pettichord (http://www.pettichord.com/)) about BDT and a question that came why call BDD by the name BDT? A valid question where the short answer is “I’m not”. Just to be clear BDT is spawned from BDD and uses many BDD practices but the purpose, the objective is to deliver testing of the application code, not the development of the code. BDT sits next to BDD, I’m not renaming or replacing it, BDD is the wellspring that provides the framework and thinking for what BDT is.

When I read about BDD and then read the RSpec book (http://snipurl.com/pipiz) in a stroke I found the answer to many tester pains. BDD:BDT focuses testers back on Behaviour and away from the erroneous belief they are testing code (what do you think testers think they’re doing when they use Boundary Value Analysis?), it eliminates friction between developers and testers because, firstly we’re not testing code (directly) and secondly we both use Cucumber Features and now testers can write more tests (focused on Behaviour) that align with the work Development are doing, Cucumber is a catalyst to get customers, BA's and Test working together in ways they’ve been dreaming of and the first tool that allows Testers to help Developers increase velocity.

When testers then start writing edge and fail cases, end to end scenarios, etc. they like Developers can define those in Cucumber and implement them in the RSpec format. The RSpec approach ‘makes sense’ to testers as it reads so easily. If we’re not just doing manual testing combined with Selenium/Ruby it gives us a really accessible route into automation that focuses on Behaviour. With Selenium/Ruby we also get to use RSpec to structure our automation tests and kick-out neat reports too. Hang on, we’ve just found a way for Test to get really involved in (and in some aspects lead) the project, with all players, aligned with and supporting Development instead of clashing with them and we’ve got a kick-ass way to write automated tests that align with Development tests yet keep us focused on testing Behaviour not code. Is the paradigm shift this represents hitting home? ;)

With RSpec we have a way for testers to write scripted tests that mean something to them and that speaks to what they should focus on, Behaviour, not code. I suggest RSpec is as significant to the testing profession as Exploratory Testing (http://snipurl.com/pipjv). RSpec is the Aha! that smacks you in the face as hard as Exploratory Testing did and provides the paradigm shift in your tester thinking that both excites you about how you’ll work in the future and makes you lament for the pratting about you’ve been doing.

I could go on and on, point is all of the above is not (BD) development of application code. It’s an absolute focus on testing but testing with an equally militant focus on Behaviour. This is what BDT is about. It’s the testing methodology spawned from BDD that focuses testers back onto Behaviour and aligns them with the rest of the development effort. A BDD|BDT project helps testers fulfil roles the business hopes they will but until now didn’t have a way to do so. Are there other ways to approach the issue of development and test team integration, engagement of the customer by all teams in a way that appears congruent, enabling all stakeholders to contribute to each other’s success, etc? Maybe but I’ve yet to work with them.

A final thought, every time I present my BDT methodology to testers it’s great to see the impact of what it enables them to do sinks in. When customers hear about BDD:BDT, Cucumber and RSpec they ‘get it’.

Presentation
Have a look at the overview presentation here: http://www.cyreath.co.uk/papers/Cyreath_An_Introduction_to_BDT.pdf

Mark.

What test technology and tools are you working with?

I've been working a lot lately on elaborating a Selenium and Ruby web test automation framework. It's going well of itself and I've used the framework with a number of clients successfully in the last 3 months or so. As I always say - lot's more still to learn!

It came as little surprise that the use of Selenium and Ruby was not what most expected as say QTP or Java are more commonplace. This prompted me to think about:

What technologies and tools are you using now in the testing field?

From my side I'm using as tech or tools sat on my desktop:

* Selenium: IDE, RC and Grid
-- http://seleniumhq.org/
* Ruby: for creating Selenium-Ruby based test scripts
-- http://www.ruby-lang.org/en/
* Rake: Build programme
-- http://rake.rubyforge.org/
* NMQA Vienna: as a (free) test management tool
-- www.freetestmanagementtool.com
* MS Virtual PC: To run various OSs to use with Grid
-- http://snipurl.com/pij0i
* SciTE: Text editor
-- http://www.scintilla.org/SciTE.html
* SQL Server Management Studio Free Express edition for working with SQL DBs
-- http://snipurl.com/pij7

So, if you have to rebuild you laptop tomorrow what technologies and tools would you be putting back on there? If you went to a client site tomorrow to deliver some testing what would you expect to be working on?

Mark.

9000 hours of billing with no single bug found yet

In a discussion on Test Republic at http://www.testrepublic.com/forum/topics/code-coverage-with-test-cases Pradeep Soundararajan summarised the evil, immoral, corrupt dark heart of the commercial side of the testing profession so well I wanted to make sure his words were captured here so they wont be lost. The commercial reality is just that, a reality that we can’t escape but I hope we can generate amazing revenue while adding value and avoiding fleecing our clients. Too many consultancies have been sued because they didn’t – you know who you are.

Reply by Pradeep Soundararajan on July 28, 2009 at 6:17pm
Mark,

Thanks for not posting it earlier. It gave me an opportunity to get you post it.

I do exercises in my exploratory testing workshop that demonstrates that those who seem to care so much about RBT aren't actually caring about it. As Michael Bolton pointed out somewhere in Test Republic that those who profess so much about documentation themselves don't bother to read and write good documents.

A good RBT requires skills that people appear to be reluctant to build. By saying that I don't mean, "Yeah, I have built it". I have been trying to develop it as much as possible and constantly practicing it so that I am prepared for a war anytime.

My perhaps more militant stand against RBT (the misuse of terms and vocabulary for ‘coverage’ aside) is because the Behaviour Driven Testing approach I advocate more and more these days focuses on behaviour relevant to the user. Ticking off Requirements against test cases is of no use to the user. Answering the questions above and formulating testing around them AND the requirements is. "

Exactly. You would call it misuse and businessmen would call it fair-use. I am starting to realize that the scripted approach survives because there is more money out there for business people through that.

Mark, outsource work to me:

I would spend a couple of days analyzing your requirement document and bill you for X hours per person involved in my team.

I would spend a couple of days writing a test plan document ( and not refer to it ) and bill you for 2X hours per person involved in my team for preparing it.

I would spend a month or two writing test case document ( and refer only to it ) and bill you for 10X hours per person involved in my team for preparing it.

I would then again create a traceability matrix ( just to fool you and your boss about our coverage ) and bill you for 5X hours per person involved in my team for preparing it.

18X hours per person involved in the team. Assuming X is 50 hours and there are about 10 members in my team, that's 18 * 50 * 10 = 9000 hours of billing with no single bug found yet. If you are paying $20 an hour per person on an average, you would have actually given me a business of $1,80,000 without me or my team finding any bug yet.

Then comes the test case execution cycles and more billing. Why wouldn't a businessman be glad about the traditional approaches to test software?

Lets bother about the users of your product later during our maintenance billing phase ;-)

Lack of vendor support for Open Source

The lack of vendor support is a real issue for Open Source and Free tools. It may seem logical that paid-for-tools are going to get superior support by the folks who have actually created them and have a commercial interest in promoting them.

This can certainly be true and NMQA (who I work for) have the Vienna test management tool that we created and so support both through paid service contracts and queries raised by the test community. However, the matter isn’t as simple as proprietary tools get superior support over Open Source or Free ones.

For example, NMQA also offer a Selenium-Ruby automation framework (in various forms) )that we support as aggressively as Vienna that we wrote fully ourselves. The reason being is that we see no difference between the two in terms of the support a customer needs. For NMQA there’s no difference in the support a customer needs for a proprietary solution we’ve developed and written in proprietary code or an Open Source / free framework constructed of open source code that we’ve set-up for them.

It’s when a customer tries to hit the internet and read online documents and forum postings to do it themselves the trouble starts. Think about that for a second, inexperienced staff trawling through spurious sources of information as the way to learn and implement a key technology, what a ridiculous strategy. Yet it’s the one often taken. Open source tools are not an easy solution to adopt unless there is expertise available, in-house or via a consultancy. The learning curve that inexperienced internal staff will take on is usually too great a burden for organisations to support and won’t deliver anywhere near as fast as is needed. Add to that the lack of trusted sources of information and we begin to see why organisations are shying away from Open Source.

There’s the issue – organisations trying to wing it on their own with Open Source solutions will mean they suffer more pain than if they buy proprietary tools and a service agreement. The best way is to engage a consultancy or specialist individual who can provide the same level of support you’d get buying a support contract for a proprietary tool and that way there’s no difference between proprietary or Open Source solutions. Later on the difference is saving tens of thousands in service contracts as insurance in case something goes wrong, also ridiculous.

Mark Crowther.

Code Coverage with Test Cases?

It hasn't really struck me until now - Why do testers think of coverage in terms of code - why aren't we thinking of coverage in terms of what the system does or should do? i.e Behaviour?

Thinking in terms of what the system should do is why we're testing isn't it? Isn't that what the customer / user wants us to be making sure before they get the software? Isn't focusing on behaviour how we assure that UAT is a success?

Never in my career have I ever really done 'code coverage'. I've played with it, talked about it with developers, even helped define accaptable coverage levels but I've never run 'code coverage tools' and declared my tests cover xx% of code. The developers I've worked with have, I remember this being the way when I was at EA. They put together Unit Tests and Component Integration Tests, ran the tools or did-the-math and declared coverage at a certain %.

What I have done however is declared coverage of Requirements. I've analysed what Test Scenarios exist, things I would do with the software to demonstrate the Requirements had been delivered on (coded the right thing), then Test Cases to exercise the Scenarios (coded the thing right) and find those lovely bugs.

Hmmm.....

It's testing with a focus on Behaviour

Mark.

What are your Selenium challenges?

It seems that when using Selenium natively there are a number of common challenges people encounter. Here’s my list of things I encountered and thought “Hmm... how do I do that then?”
What would you add? What have you struggled with or are struggling with now?

• Dealing with pop-up windows
• Testing dynamic text or content
• How to go about testing Flash
• Capturing screen shots, either to file or in some form of report
• Iteration of the test case, running repeatedly with minor changes
• Data Driven Testing, pre-cooked data or generating on the fly
• Generating useful test status reports
• Setting up Remote Control
• Setting up Grid

Mark.