Test Case execution priority

A question put to me recently:

"If you have Test Cases already prepared and you have to just excute those test cases in a very short time then how will you decide which test cases has to be excuted first. Is there any technique ?" - Mrityunjay

If you have more Test Cases than you have time to run then you clearly need to put them in some priority order to ensure you run the 'Most Important Tests'.

I think it's rare that the test team can do this as we don't have a complete view of the importance of all development features that are the Most Important.

Approach 1

As soon as you have them written get together with people that can decide their priority such as the Architect, Product Owner or Project Manager and Walkthrough the Test Cases. Ask them to put them in order for you. Ask them "If I test nothing else, if I show nothing else works - what would you want?". Then when you have that ask them "If that was done, what would be next?" until you're done. This meeting should be 30 minutes tops.

Approach 2

If the test team really have to choose then there's analysis needed to decide the selection of the cases:

• Risk - Does the Test Cases cover an area where there's been a lot of change or where many issues are usually found?
• Importance - Does the Test Case cover an area that absolutely must work and/or will always be used?
• Breadth - Does the Test Case cover functionality broadly and is a high level test over a set of lowere level, alternate Test Cases?

If you have a Test Case that is covering critical functionality in a high risk area then you need to run it. If you have a Test Case that is covering an area of minor change across functionality hidden several menus deep that will be used by a small group, leave it to the last.

Of course now you need to make sure you're reasonably accurate in your estimation of how long test execution will take. This will ensure you can declare what test will be executed and which wont be.

Test Estimation, take a step back

As test professionals we're expected to answer 'off the cuff' how long something will take to test. It must be the single most common dreaded moment for any test lead or manager. The projects starting and we're expected to know ho long all test tasks will take, including the identification, authoring and execution of tests. Sadly, 'How long is a piece of string...' doesn't go down well nor does sarcasm about how it's pointless estimating because after the project owner/customer/developers delay/change/fail to write documents, etc. the estimate will be worth poop. As true as it might be.

So, these 'challenges' aside how can the test professional at least arrive at a reasonable set of estimations for the test tasks? Firstly we need to know what tasks are needed. It's seems right that we should be able to break down a delivery of testing into a set of known or expected tasks. Having those identified we could then estimate how long each would take.

Taking a step back - it's essential that the test manager is clear on what tasks should be performed for a given project. That comes through designing a test methodology or several which can be applied to various project types.

A further step is to model the effort needed to deliver each task. For example, writing the Test Plan. Though it's impossible to estimate with perfect accuracy the test manager cannot make this up every time. They must have 'model' answers that can be refined in context of the project.
When I get folks who are convinced they can't possibly say how long I ask: 'How long does it take to write a test plan?' - 1 month? 1 hour? Well no. 1 week then? Too much. With all reviews, maybe 1 week start to end. The effort for this? By the test manager maybe 2 days tops. Great, so my model answer is a Test Plan takes 1 week to deliver and requires 2 days of effort by the test manager.

Test Case identification and authoring is more difficult as the complexity and number of cases needed is entirely context dependent. There's a greater reliance here on the project being run in a way that delvers such artefacts as good requirements documents. Ah yes, back to that old chestnut.

Next time I get a project I'll think about it's complexity, the competency of staff, etc. and provide a duration/effort in context of the project. But I have my model answer as a start point so I can arrive at reasonable estimation more quickly. Rinse and repeat for every other task in your methodology and you have a general estimation model(s).