Friday, 25 July 2014

More Reporting - Pitching at the right level

As mentioned before, a key part of any test delivery is the reporting we provide. In my experience, management are often not interested in the majority of what we do. Sure, at a high level they care about whether we know what we're doing, but they're not interested in the low level details as we are. For us, the fun is in the cleverness of analysis, writing great sets of test cases, executing in different ways, maybe adding some automation into the mix and so on. That for us is the important stuff and rightly so, you don't get a delivery without it. However, the management team need mainly just to know what was planned, what happened and where it leaves us. Hence the need for reporting that is informative, easily understood and readily actionable.

When looking at what reporting to provide remember, it's all an 'interpretation' of what happened and you can describe your activities in a way that fits with management expectations. If they're completely on-board with your approach that makes it easier, if not, you might need to get creative. An example that springs to mind was on an engagement where management wanted an approach that followed the typical SDLC phases, weekly steering committee meetings and dashboards produced in Powerpoint slides. Meanwhile the implementation team wanted a more agilistic approach. The desire was to deliver 'something' demonstrable, if not strictly usable, on a weekly basis. That meant following a Scrum like approach to planning and delivery, seemingly opposite to that which management was expecting. Leading this team in a not quite Scrum master role, was a manager who wanted not only weekly reporting updates but end of day updates too.

As the lucky one in charge of environments and testing, it was my job to reconcile the needs of all sides and report out on what the test team were delivering. The technique is to expose essentially the same information but in the way requested. Wow, that was insightful! Let's look at what that means in practice.

The SDLC approach means high level dates on a schedule, spanning key phases, perhaps at best breaking the phase down into key sub-milestones. Phases from this perspective are 'big' - Analysis, Planning, Execution and end of phase Reporting. As these phases proceed a weekly summary of progress for is provided to senior management and execs. This is a sequential, big-up-front approach so senior management expect reporting in that context. Reporting contains data in context of the phase, not so much the activities underneath. For the agilistic mindset, reporting like this can be frustrating, but it needs to be remembered that at some point up the chain, all reporting morphs into this. Board members and Shareholders don't want to know how many test cases in what state there were today. The daily detail needs to be rolled-up into one higher level report.

Where are we through the SDLC phase, are we on target for the end date?
What No. or % of the total are we at and is this matching our plan?
What are the high level common issues being faced?

This provides insight into management concerns which usually revolve around money. Is the implementation of the new software on-track with Product Lifecycle plans? Will the product hit market in time for the marketing activities planned? Is the ROI achievable in the time expected to maintain share price? It's a rare tester that sweats this stuff, it's a rare marketeer that sweats the defects and test automation details. For the team closer to the actual work, this level of reporting is too high - it's not adhering to our rules above of being informative and readily actionable.

Back in the team, the reporting needs to be closer to what we discussed in the End of Day Reporting blog article. With their agilistic mindset, the team need information that shows the heartbeat of the project day by day (half-day by half-day is my preference!). In this way they can slow down or speed up as required. This is informative and actionable for them, but isn't useful by senior management. Rolling the daily reports up into a weekly at the same level is also something the team management can respond to.

Looking in on the project, senior management see the sequential phases, following the SDLC as requested. Underneath, the team can be slicing delivery up into small chunks of daily usefulness.