QA Survey, Interview Questions, Awareness Campaign?

Some time ago myself and a colleague were asked to brainstorm a list of questions that might be used in an internal survey. The purpose of which was to assess how aware the organisation was of what the test team did.

It was one of those great 1 hour brainstorms that produced a peice of work that surprised both of us.

Having been searching my PC for something entirely different it was a pleasant surprise to rediscover it. So, before I lose it again I thought it might be wise to share it ;)

Enjoy using these and add any in the comments that you think of yourself.
..................

Reporting
1) Do you know what type of reporting QA provide during a development project?
2) Are you aware of any reporting or information that is available when QA are not engaged in a development project?
3) Do you know if QA publish any information such as what they are working on or who is working on it?
4) Are you aware of what could be provided using any reporting systems we might have?
5) Are you aware of any reporting systems QA might have?
6) Have you ever seen reporting for areas such as Bug Quantity found, % complete of the test cycle, % code coverage?
7) Are the above or similar of use to you? How?

Testing
8) Is there a process for requesting testing / test resource from QA?
9) Do you know what type of testing QA provide during a development project?
10) Are you aware of the testing that QA perform when not engaged in a development project?
11) What reasons do you know of for QA selecting a type of testing to deliver?
12) Can you name any dependencies QA have from other teams that help them deliver the testing?
13) How do these dependencies impact QA delivering the testing?
14) Is it clear why testing takes the time it does?
15) When a Test Engineer isn’t testing, do you know what they are doing?
16) Is it clear how early QA should be involved in a project to deliver effective testing?
17) Do you know if and how QA use Requirements and Design documents?
18) Do you know what items or documents QA create when delivering testing?

Bugs and Issues
19) When do QA submit bugs?
20) Is it clear why QA will submit bugs?
21) Against the above do you know of any reason when we wouldn’t submit a bug?
22) Is the process for managing bugs through QA and dev clear?
23) Do you know the process for deferring bugs?
24) And closing them?

QA and Testing Knowledge Share
25) Has it been made clear what the ‘Software Test Life Cycle’ is?
26) Are you aware of any other practices and processes QA have in place?
27) Is so, do you know where they are published, if at all?
28) Do you know what tools QA use to aid testing?
29) Do you know if QA, development and documentation use the same tools?
30) Do you know how the outputs from tools get used by different teams?
31) Do you know how many Test Engineers are in the team?
32) Are you aware of how they are assigned test work?
33) Do you know if QA have relationships with development teams?
34) If QA do, is it clear why?
35) If QA don’t, do you believe QA should?

Contentious…
36) Have you ever attempted to build a relationship with the QA team?
37) If you have, how did this go?
38) What would you suggest is the single most useful thing QA do?
39) What would you suggest is the single most useless thing QA do?
40) Do you believe QA add value to a project?
41) If not, why not?
42) If so, how?
43) Can you name a project that would be in a lesser state of quality without the input of QA?
44) Can you think of a project where QA didn’t help or perhaps were detrimental to the project?
45) If QA could make one change to what we do, what in your opinion should it be?
46) Do you think it’s clear to QA what your team do?
47) Come to think of it, what is it you do? :]
48) Do you believe QA should be proactive in chasing development or development should come to QA?
49) Do you see QA as a potential pool of resource for adding to development, a stepping stone into development?
50) Should QA provide guidance to the developments teams on how to test their own software?

Test Case execution priority

A question put to me recently:

"If you have Test Cases already prepared and you have to just excute those test cases in a very short time then how will you decide which test cases has to be excuted first. Is there any technique ?" - Mrityunjay

If you have more Test Cases than you have time to run then you clearly need to put them in some priority order to ensure you run the 'Most Important Tests'.

I think it's rare that the test team can do this as we don't have a complete view of the importance of all development features that are the Most Important.

Approach 1

As soon as you have them written get together with people that can decide their priority such as the Architect, Product Owner or Project Manager and Walkthrough the Test Cases. Ask them to put them in order for you. Ask them "If I test nothing else, if I show nothing else works - what would you want?". Then when you have that ask them "If that was done, what would be next?" until you're done. This meeting should be 30 minutes tops.

Approach 2

If the test team really have to choose then there's analysis needed to decide the selection of the cases:

• Risk - Does the Test Cases cover an area where there's been a lot of change or where many issues are usually found?
• Importance - Does the Test Case cover an area that absolutely must work and/or will always be used?
• Breadth - Does the Test Case cover functionality broadly and is a high level test over a set of lowere level, alternate Test Cases?

If you have a Test Case that is covering critical functionality in a high risk area then you need to run it. If you have a Test Case that is covering an area of minor change across functionality hidden several menus deep that will be used by a small group, leave it to the last.

Of course now you need to make sure you're reasonably accurate in your estimation of how long test execution will take. This will ensure you can declare what test will be executed and which wont be.

Test Estimation, take a step back

As test professionals we're expected to answer 'off the cuff' how long something will take to test. It must be the single most common dreaded moment for any test lead or manager. The projects starting and we're expected to know ho long all test tasks will take, including the identification, authoring and execution of tests. Sadly, 'How long is a piece of string...' doesn't go down well nor does sarcasm about how it's pointless estimating because after the project owner/customer/developers delay/change/fail to write documents, etc. the estimate will be worth poop. As true as it might be.

So, these 'challenges' aside how can the test professional at least arrive at a reasonable set of estimations for the test tasks? Firstly we need to know what tasks are needed. It's seems right that we should be able to break down a delivery of testing into a set of known or expected tasks. Having those identified we could then estimate how long each would take.

Taking a step back - it's essential that the test manager is clear on what tasks should be performed for a given project. That comes through designing a test methodology or several which can be applied to various project types.

A further step is to model the effort needed to deliver each task. For example, writing the Test Plan. Though it's impossible to estimate with perfect accuracy the test manager cannot make this up every time. They must have 'model' answers that can be refined in context of the project.
When I get folks who are convinced they can't possibly say how long I ask: 'How long does it take to write a test plan?' - 1 month? 1 hour? Well no. 1 week then? Too much. With all reviews, maybe 1 week start to end. The effort for this? By the test manager maybe 2 days tops. Great, so my model answer is a Test Plan takes 1 week to deliver and requires 2 days of effort by the test manager.

Test Case identification and authoring is more difficult as the complexity and number of cases needed is entirely context dependent. There's a greater reliance here on the project being run in a way that delvers such artefacts as good requirements documents. Ah yes, back to that old chestnut.

Next time I get a project I'll think about it's complexity, the competency of staff, etc. and provide a duration/effort in context of the project. But I have my model answer as a start point so I can arrive at reasonable estimation more quickly. Rinse and repeat for every other task in your methodology and you have a general estimation model(s).

Oh monkies, a Survey!

If you're part of the QA & QC Google Group you'll have noticed my posting about analysis techniques and practices I put up there some time ago. Actually, you probably didn't notice as no one replied. But it's OK, I'm not (that) bitter. To help folks I've put together an anonymous survey. Non, no, don't thank me.

A few months ago I read Lesson 32 (chapter 2 generally) and section I-8 of the NRC Fault Tree Handbook and I felt I was getting a real insight into analytical techniques. You know those things you do naturally to some degree, but you do them in a crude fashion and you're not actually sure if they have a name? That was me, heck still is!

Well here's the shimmy, I've been studying the topic of testing requirements analysis for a few months now. Part of what I'm trying to understand is what we 'really' do in this area. This really is one of those areas where we know what we 'should' do but rarely do or get the chance to do.

There's a link to the survey and pop-up on my website. The vocabulary I use in the survey is ambiguous on purpose as I'm trying not to 'lead' people in their answers. I'm no professional survey author so hopefully it wont confuse you too much! It’s only 10 questions but you’ll see you can add as much extra detail as you wish.

Drop me a mail with your critique of the survey too. I'm very keen to see how these can be used as a tool to get more insight on topics of interest.

Remember the survey is anonymous so I can’t thank you individually, you can always email me of course! So, thanks in advance and I'll make sure I share the information here once I have something meaningful together.

Mark "and just another 50 questions" Crowther

The joy of YouTube

I feel it's taken me a while to realise that YouTube is in fact a great resource for study of software testing. Using the Google Group I am part of or Wikipedia and listening to software testing podcasts are a given. But for some reason I hadn't sat down and spent time on YouTube, curious.

Well, that's the great thing about Christmas breaks. Time to eat too much, watch TV and spend time in front of YouTube.

If you're reading the blog looking at getting into testing then have a watch of the Portnov School's six video set that introduces the course they offer. They're in the US but the overview of what basics a new tester should master are the same the world over. I've encountered a fair amount of people that might have ISEB or similar but don't know some of the basics they seem to cover on this course.

A good video by the well renowned James Bach and is called Becoming a Software Testing Expert. Many comments about James have him viewed as arrogant and self promoting. To a degree I think he is but to think that's all would be to miss the essential message he tries to convey. That being there are no 'experts', there are no 'best practices' and maybe no spoons too. Keep this in mind when watching and it's a great hour with this interesting guy.

Another interesting video is on Agile Testing by Elisabeth Hendrickson. She starts by discussing the more traditional test approaches to put Agile in context. The main discussion picks up at around 13.00 minutes onwards. During the video she discusses Agile and XP and how they sit together, which was a useful perspective, in addition to how Scrum fits in.

Elisabeth mentions a number of things that aren't quite complete 'right'. One thing being the independence of the testing function. Stating it's not correct to have the function independent of the Agile team. In fact Agile teams support that independence through the equality of the function within the team.

As part of the Google Tech Talks series there's Scrum at al with Ken Schwaber which is a great follow up to the previous video. He's surprising in that he doesn't really evangelise the use of Scrum despite being the main creator of it. To follow up Jeff Sutherland then has a talk about Scrum implementation.

The interesting thing from both the Agile and Scrum videos is that both talk about the need for documentation. Something that is often assumed can be dropped as a mark of going Agile...

There's five hours of viewing there alone and many, many more videos to watch. So enough blogging, time for a cuppa and some more videos.

The Dreaded Walkthroughs and Reviews

It's always been a source of bafflement (technical term) that within the testing domain Walkthroughs and Peer Reviews are not more widely practiced. I recall being 'invited' to a code review where a developer went through their code line by line in dolby prologic monotone. It was almost as painful as the many walkthroughs of Test Plans I've subjected folks to.

What I took away from the meeting was how incredibly useful and interesting it 'could' have been. Here was the opportunity to have code explained line by line, to be provided a live narration of the thinking, logic and reasoning behind what had been created. What's more our own views and opinions could be incorporated into what he been made.

If this was some contempory artist or author giving a narration of one of thier works we'd be fascinated and consider ourselves fortunate that we might influence the work. But it's just some bloke we work with and it's code, so we fall asleep.

The problem is it can be hard to get excited about code, even harder to get excited about someone talking about it! The reality is most Walkthroughs and Code Reviews are brain numbingly boring.

You've probably heard of and been subjected to Walkthroughs and Code Reviews of various types, the idea that an author of something (code, plans, schedules, etc.) will sit down with an interested group and walk them through, line by line, clause by clause, page by page, explaining what was written and why. Occasionally asking for input, "all ok?", "Seem to make sense?" so that after say 30 minutes or maybe an hour everyone has found they're not interested anymore and are half asleep. Actually, probably asleep. It makes me feel tired just describing it!

Peer Code Reviews
Peer Code Reviews on the other hand are meant to be snappier, more active and energetic. Think more in terms of having produced a usable piece(s) of code, say a set of core functions or an interface onto those APIs your buddy wrote. Then with this small chunk in hand get it reviewed. The review is say 10 minutes long, you're doing a Walkthrough but it's a lively narrative.

To answer your question directly - Best Practices
- Small, manageable review items: not weeks of coding
- 5 or 10 minutes review time: not 1 or 2 hour brain numbing marathons
- Grab reviewers: don't book out meetings, keep it proactive (but don't break folks concentration if they're 'in the zone'!)
- Actively seek review from different colleagues: cross mentoring and knowledge sharing
- Keep responsive to new perspectives: review isn't just bug finding, it's learning tricks and tips too.

Peer Test Reviews
The interesting twist for us in the test domain is that we can apply this approach to our Test Artefacts. When we're in an Agile environment it serves us well to be working in the same spirit as our developer colleagues.

Why not get Peer Test Review of those 20 or so Test Cases you just put together for that module? Incrementally delivering Ready for Execution Test Cases is a great way to help the likes of Project Managers feel relaxed that we're making progress on our planning and prep.

Doing the same with Test Plans, Designs, Breakdowns or whatever other artefacts you produce is also a win. This lightweight approach achieves our objectives but stops us getting bogged down in heavyweight process.

Follow the above Best Practices and keep the event lively, if you really must book out a meeting to coordinate review with several people at the same time, that's OK. Just go a little overboard with your presentation. Print copies in colour if you can or let folks access it on their lap-tops to save trees. Use the projector to make viewing easier, create slides that are already noted up or can be 'written on', what ever keeps the energy levels up.

Mark Crowther
Peering at you!

The idea of applying Peer Code Review to the test domain came from: http://www.jaredrichardson.net/blog/2007/03/14#peer_reviews, thanks Jared.

The Future of Testing?

A message over at QA & QC Group, well made my blood boil slightly. Here's the question(s) and response I gave. Grrr... etc.

"I am working in MNC in India, Recently one of my team mates (PM level) told me that the future of testing is no scope, and he gave reasons."

To answer your question in succinct form, the future for software testing has never been better, the future is brighter than perhaps we suppose, it's brighter than we can suppose. Why?

1) If any software boom is down first the Companies fire Testers.
Wrong. This assumes the situation as it is now or was in the past, as it is in this transitional state, but you're talking about what the future holds. So this statement is wrong. The ones at risk are development, probably more like Technical Authors or Support Desk. The value the business derives from testing is eminently improved the more difficult the economic situation becomes. The logic for this is because there is a need to maintain a share of a shrinking market where the differentiator would be price and quality, yes in the singular because the are so linked. Testers are far cheaper than developers generally and in safeguarding quality are essential. Safeguarding quality ensures the total cost of ownership is reduced meaning a well tested product is a better investment. A product with higher quality means a reduced need for Support Staff there by bringing a greater saving to the business.

2) Developers can do our job + Coding.
Wrong. A simple reflection on the Division of Labour, Specialisation of Work theories will tell you that there will at some point be enough work and enough need for focus of effort by individuals who are particularly experienced in testing, to need people who's specialisation is testing. Turn this around, can testers also code as well as developers? We would always say No to this, we understand their specialisation, experience, etc. lend themselves to being developers who can test, just as it does us being testers who can develop. But the two professions are not fully inclusive.

3) In future comfortable tools are comming.
Wrong. This has been spoken of for many years and even the best of the Record and Playback tools fail at encountering the simplest of issues. This is the same logic as for the robots running their AI cleaning my house and making me a cup of tea as I type... I still don't have one. Even if we accept the suggestion that these tools become so all powerful they are acting like a tester, who's going to configure, run, maintain, mature what they do? Is this person by definition not a tester?

4) No investments on the Testing more?
Wrong. The very act of investing in the above in a desire to eliminate the tester is by its nature investing in testing. If we accept that the paradigm of how we currently define 'tester' will shift then you'll not get rid of testers. Again, what will happen is the boundaries between tester and developer will blur. I maintain it's the developers who are at risk in many areas.

All of the above statements your team mate made are based on the current paradigm of what a tester is. They are looking at what they understand the tester of today to be and in doing so are making statements about testers that were essentially existing yesterday. Being in the profession we're aware that the days of just hitting keys and clicking the mouse are over, it has been over for all but the most junior members of a test team for perhaps more than 10 years!

Today's and tomorrows tester is a much more technically savvy professional. They can develop test harnesses, stubs and drivers written in and interacting with a variety of programmatic languages, author complex data sets, work in an integrated manner with Agile teams on a level that blurs the boundary between tester and developer, use highly complex tool sets testing across the many components of the global system architecture and much more.

Today's and tomorrows tester is a professionally educated, examined and accredited professional, including but beyond that of a general computer science degree and some courses that a developer may typically have and soon potentially a member of a Chartered Institute. Putting them at the same level as Architects, Lawyers or HR professionals.

The key hitting monkey of yester year now uses techniques grounded in complex statistical and analytical mathematics, cognitive psychology and some of the best scientific research covering everything from computer technology to human logic.

As I said, the future for software testing has never been better, the future is brighter than perhaps we suppose, it's brighter than we can suppose.

Mark Crowther