Test and learn is a popular and ‘agile’ approach to development, but does it work at scale?
What is ‘Test and Learn’?
Test and Learn is an approach followed by many consumer-focused companies to test ideas in a small number of situations to predict impact. It’s designed to assess impact on Key Performance Indicators (KPI’s) across a network or customer base, to see if the impact will be greater on some outlets, channels or customers than others and which components of the idea are actually working. It’s not just about finding out if it worked, it’s about learning and sharing the insight about the test itself, what was measured, what worked, what assumptions were made, what did we discover that’s new, who needs to know about what we’ve learnt and what should we all do as a consequence. Capital One used this approach successfully as far back as 1988 and it’s been common in many companies seeking to reduce investment risk. But it is, or at least has been an evaluation approach not a ‘silver bullet’ for implementing marketing strategy.
The costs of ‘Test and Learn’
If used well, Test and Learn approaches can deliver significant savings especially in reduced ROI and avoidance of sub-optimal investment. But, when poorly implemented Test and Learn can have significant hidden costs, perhaps in some cases prohibitive. On the ‘up-side’, well structured tests, with clear outcomes can inform more sophisticated strategy and pinpoint opportunities without committing too much capital or resource. Shared learning can help avoid repeating mistakes and provide a knowledge base of good practice. There can also be some other beneficial side effects: discovery of new unpredicted opportunities, devolution of decision making, democratisation of knowledge and more realistic real time insight.
However, on the ‘down-side’, poorly designed and executed tests are costly. If the expected outcomes are not clear, the tests themselves are not designed properly or critical assumptions have not been challenged there will be costs. Some of these costs are obvious, wasted resource in carrying out tests for example. But, others are far more significant and often hidden. These include costs of misdirection and bias that may lead to further and perhaps significant ‘wasted investment’. Where learning is ot shared effectively, costs are also incurred. These too, can be significant, and hidden: silo’d repetition of poor practice, sub-optimal corporate memory, random brand damage and increased costs of compliance ‘after the fact’.
Test and Learn as an asset
So what’s new? Collaboration, ideation and co-creation technologies hold some keys to implementing ‘Test and Learn’ at scale. Those who understand these technologies know this and are currently developing significant competitive capability. Those that don’t may struggle for some time to optimise the costs of Test and Learn and realise it’s full potential. But collaboration, ideation and co-creation tools are not the solution to test and learn at scale, they are only enablers. Nothing replaces rigorously design tests and disciplined execution and no one should underestimate the investment in culture and process required to realise the benefits of shared learning throughout an organisation. For those that institutionalise the disciplines of Test and Learn using collaboration, ideation and co-creation technologies there are significant benefits. We know this, and it’s part of a fast moving and new frontier in Enterprise Capability Building.
We also know that it’s not the only approach and that Test and Learn is just that; testing and learning. Other, more sophisticated strategic and tactical approaches are also required to enable the ‘modern marketing machine’, engineering approaches and disciplines that, like Test and Learn are perhaps borrowed from other industries or disciplines: agile and networking methods from technology for example. Even so, new opportunities to build new forms of Enterprise Capability are without doubt available.