Sunday, 29 March 2015

Replacing QA with 100% automation

Replacing QA with 100% automation

A question came up on one of the forums recently, that echoes a question I and I’m sure you’ve been asked at some point. That question being along the lines of reaching 100% automation. Groan… yes, that old chestnut.

As experienced testers we already know the answer, for the avoidance of doubt the answer is: No.

Thanks for reading, now on with my day!


OK, OK… we wish the ill-informed question could be dismissed so easily.

However, though we know its asked by the ill-informed, we can’t blame people for asking. To senior management, non-technical, non-testing staff it’s a reasonable question. It’s our job to respond in a way that ensures we leave them informed.

To whit… if asked, here’s a few thoughts on why we can’t achieve 100% coverage via automated testing. (We’ll skip the QA Vs Testing perspective for now, hey they’re ill-informed, just roll with it).

You could also send them this link of Case Studies on Automation, by Dorothy Graham (RTFM…? :)

It isn’t possible to achieve 100% test coverage of the application/system with manual testing, so strictly it can’t be possible to do the same with automation. Though interestingly we might expand coverage…

Technical Limitations
There are aspects of the system you can check and test manually, that can’t be automated ‘technically’. Examples include Accessibility and Usabilty, others include complex end-to-end scenarios that might require exploratory or dynamic testing dependent on variable results.

Automation of what?
The different levels of testing (unit, integration, system, etc.) mean we’d need to have suits of different automation tests, focusing on different aspects of the system / application.

Tools and Team Skills
If we wanted to ‘automate everything’ we’d need a host of tools; web front end, application interface, API testing, webserver/DB/CDN testing. Maybe 100% automation is just one of these?

Assuming we define what 100% means in this context, do we even have the tools in place or the staff who know how to use them? If not, there will be some manual testing going on.

Don’t have time / money
The Return on Investment from automating everything just won’t be there. So, even if we had the tools, time, skills and a limited automation remit – it’s highly unlikely the time and money needed would be worth it. Automating all of a payment system that’ll be there for 10 years is more likely than automating a website that’ll be trashed in 6 months.

What other reasons can you think of?


Enjoyed this post?
Say thanks by sharing, clicking an advert or checking out the links above!
Costs you nothing, means a lot.