Thursday, 18 June 2009

June SIGIST Talk - 'Pragmatic Testing - the Middle Way'

The following text is the transcript of the talk given by Mark Crowther at the June SIGIST in London.

What is Agile/Heavyweight/Testing anyway?
Often we see on forums, blogs and so on discussions that ask about and responses that aim to provide a definition of what Agile testing is in practical terms.

In the same way we might use say Waterfall as a definition of what Heavyweight is, it sometimes seems that we're looking for a diagram or model as a way to define Agile.

Or perhaps by saying that Heavyweight is all about documents and process we can compare and say Agile must therefore be about avoiding documents and process.

The issue is these approaches of simple comparison will always fail to help us clearly define Agile and Heavyweight. The fact is that right now Agile and Heavyweight are at best metaphors or perspectives on how to approach testing practice.

They're not clearly defined paradigms, i.e. there's no robust and complete definition of the practices that make up Heavyweight or Agile test approaches. So in my view we as a profession don't clearly and collectively know what being in the Agile testing mode of practice is but we expect and are expected to deliver as if we do.

We don't have a clear set of practices and approaches that we can point to and say 'that's agile or that's heavyweight'. We have a few random things that sit in each area but mainly a subjective idea of what it means to be Heavyweight or Agile.

Who's read or contributed to discussions on forums where the question is about the tester’s role in an agile team?

It comes up every 4 or 5 weeks on the forums. The other one being 'I been told we're going agile, what does that mean'?!?! You can almost hear the tester’s frightened voice behind the question.

The very fact we're at a SIGIST dedicated to testing 'in practice' tells us there's still a lot of thinking and discussion to do, not just around Agile either. If it was all signed, sealed and delivered we'd be talking about something else, here and within the testing community generally.

I previously worked at AOL and when I was there I had the good fortune to work with Thoughtworks. If you're familiar with them you'll know they could readily be considered thought and practice leaders in the agile development domain.

In fact around 8 years ago, on a project they were in to help deliver, they introduced us to crazy new ideas such as Daily Stand-ups, backlog items and Wikis. They were doing agile development and testing, 8 years ago... and here we are today as a profession still discussing what it means to do Agile testing in practice.

I believe there's a fundamental reason for this and it's that our current way of thinking about what Heavyweight and Agile testing practice is - is deeply flawed.

It's flawed to such a degree that if we continue in the current mindset we'll carry on re-asking the same questions for years and still not arrive at the answers we're looking for.

False Expectations of the Testing Future
My first contention is that we hold a false expectation that the testing profession is undergoing an evolution from a Heavyweight view of the testing world to a more Agile centric one.

In the same way we might consider ourselves to have matured from an ad-hoc / unstructured testing world into the heavyweight.

That in some way we'll become progressively more enlightened about how to realise testing process and practice that's moves us more towards Agile and in so doing moves us beyond the Heavyweight.

It's my view that this destination doesn't exist and that in fact we're already seeing what the future of the testing profession will look like. Consider what happens in practice when you're on projects and planning or actually delivering testing.

You may consider a project to be an essentially heavyweight one, perhaps it's testing in a regulated industry. But how often in those projects are you asked to create all the documents you proposed;
- but could you maybe not write out the test cases with all the steps?

Perhaps the project is lightweight requireing a more agile approach but how often are you then asked to work from iteration to iteration...
- but provide details and durations of all your tasks for the entire project just for budgeting and planning purposes?

- or could you provide a more complete Gannt chart as that's what the customer is expecting to see, instead of just the dashboards you were planning to provide?

Even in Ken Schwaber's book, Agile Project Management with Scrum, he mentions at the end of almost each chapter that he has to step away from 'pure' Scrum and compromise on the way he would like run things, in the ways just described.

The evidence from my experience and what I've heard and read from others is that the reality of testing practice is neither purely traditional/Heavyweight or Lightweight/Agile, but is more of a Hybrid of the two.

I'd suggest that the future of testing practice isn't going to be defined in terms of Heavyweight or Agile as we think about those perspectives at this time, but in the future testing will defined in ways more similar to the Hybrid approach that we're experiencing on most projects today.

I believe that this hybrid approach, driven by the project's constraints and practicalities of the delivery, is more representative of the 'normal' state of testing practice that we experience.

However, while we continue not to realise that 'hybrid is actually normal testing' we'll continue to hold the mistaken belief that we must be purely heavyweight, agile or it's not quite right.

What's more, in so doing we'll continue to try and evolve into Agile from heavyweight at the exclusion of practices and techniques we keep experiencing the need for.

Meaning and Impact of a Testing Paradigm
Adopting this perspective of hybrid testing as normal, we can more easily move towards defining a central, stable paradigm that more accurately describes the core of our profession.

In saying this I define a paradigm; as being 'a set of exemplary practices that define the core principles of a discipline'. This collection of practices come together to give us a definition of normal testing in the same way we have 'normal science'.

Normal science is often referred to as 'thinking inside the box' and represents the day-to-day accepted ways of approaching a scientific discipline.

The idea of a Normal Testing paradigm can be considered in the same way, as representing the exemplary set of practices we'd use in the day to day testing situations we'd expect to find ourselves in.

I'm not talking about defining immutable best practices here by the way, but I do believe it would be acceptable to refer to them as good practices. In many professions this set of practices is collated into a peer reviewed Body of Knowledge (BoK) by perhaps the governing, chartered institute of that profession.

We have the start of that in the ISTQB but it doesn't go far enough as the ISTQB doesn't publish the actual knowledge, just the syllabi for various courses.

Defining the testing knowledge relies instead on accredited commercial organisations designing courses or authors writing books that provide material that can be used to deliver the syllabi.

Not everyone follows ISTQB of course, as such the testing profession's approach to defining the actual knowledge that would deliver our 'normal testing paradigm' is at best commercially biased, at worse fragmented and un-coordinated.

This situation therefore perpetuates the issue of us never being able to 'finally' define what Agile, Heavyweight or a normal testing paradigm might look like.
At least not in a way that is consistent, agreed and accessible to everyone in the profession.

It's this situation that I suggest causes us to remain confused about topics we knew about 8 years ago or to ask the same questions on forums every 4 weeks.

It's also what causes us to perceive the work of testing luminaries, such as James Bach, Elizabeth Hendrickson, Michael Bolton, etc. as the new, emerging paradigm we must follow or get left behind.

It's also what causes us to feel exasperated as to what we should be learning and the approaches we should be taking, and why there seems to be so much disagreement about what should now be; done-and-dusted fundamentals.

It's attempting to make sense of this is why I say that as most of us go through our careers we inevitably sway from Heavyweight to Agile to somewhere else and often feel very confused along the way.

It’s also a barrier for new comers to the profession and limits our standing as a serious bona fide profession in the eyes of the rest of the world.

So what to do given the current situation?

The Middle Way – a Toolbox for Testing
About 12 months or so ago NMQA had a series of Workshops in-house that essentially touched on the points I've made so far.

We realised that even when called in to work on what were suggested to be Heavyweight or Agile projects they were always – ‘Agile, but...’ and ‘Heavyweight, but...’ or something entirely different.

Now as a consultancy we’d responded to these situations and delivered - but drawing on our previous experiences as managers and members of test teams we recognised getting in this situation isn’t limited to just consultancies.

During our careers as test professionals we’re going to encounter differing environments, maybe we’ll work in;
- digital media or the online space where a more ‘agile’ approach is needed
- pharma or military where regulation needs a more ‘heavyweight’ approach

This is the ‘normal testing paradigm’ that’s at the heart of our profession and being entirely focused, schooled in, beguiled by the Agile or Heavyweight perspective is going to limit us.

The practical outcome of this perspective at NMQA was to develop a ‘Testing Toolbox’ that contains all of the core models, practices, processes, documents, etc. that we would reasonably expect to need for most projects.

We used this to define in real terms, objectively the - ‘normal testing paradigm’ as NMQA experience it, a collation of everything stable, accepted, we felt could place ‘inside the box’. This is the same idea as I mentioned for the scientific paradigm earlier on.

But what about testing practices and the work of people such as James Bach, Elizabeth Hendrickson, James Lyndsay, Matt Heusser, Lisa Crispin, Michael Bolton, and many others I could mention?

It’s important that they and the practices they promote, and agile in general, are not mistaken as the sum of the testing profession even if we often fall into talking about them as if they are.

Continuing the analogy from earlier a lot of their work I’d suggest are great examples of ‘thinking outside the box’. These people are thought leaders, practice leaders and they challenge and stretch us by what they think and do.

Like experimental science the work they do, pokes, prods, stresses, refines, re-defines, replaces, improves and tests the accepted body of knowledge. Every now and then from this we get a new or more powerful tool in our testing toolbox.

Exploratory testing is probably the most recent example that springs to mind.

We need to realise that 80% of ‘us’ need what makes up the normal testing paradigm as we have days jobs where we can’t experiment, we don’t have time or opportunity to dally with interesting and amusing experimental practices.

This is how I see these luminaries, these thought and practice leaders, as a vital part of the energy and vibrancy that the profession needs to overcome the inevitable entropy that would occur in maturing the profession in the way I’m discussing.

Getting Involved
There’s two things I don’t have time for in this talk today, Questions and the opportunity to show you the tools in the toolbox.

What’s more this is the first time this perspective has been presented to the testing community. We know what it looks like within NMQA but I’m interested in seeing what it means to the wider testing community.

Please do something for me - what I want to ask you to do is participate and contribute to this discussion. Here’s how;

• Visit and go to the forums. There’s already a post there waiting for your feedback, thoughts, disagreements, alternate perspectives.

• You’ll find a link in that post to a survey about this talk, click the link and complete the survey.

In return - if this talk has got you thinking, if the ideas of moving beyond the ‘agile v heavyweight’ discussion, if the ‘normal testing paradigm’ resonates with you and if building a ‘Testing Toolbox’ for your organisation is of interest...

Drop me an email or call me and I’ll come and visit you and your team and have a Workshop type meet for an hour or two where we can run through some of this again, I can show you some of the practices, template documents, test techniques, where we can think about what it means in the context of your organisation.

It’s my view that we need to fundamentally rethink the way we view our profession. The old perspectives have served us well so far but I don’t see them doing so well in the future.

I’ve enjoyed talking to you this morning and I hope this has been of interest to you.

Thank you.

Mark Crowther