The first step we always talk about is, “What was the purpose of the experiment and were we very clear with the management team about what we were trying to prove or disprove?” A lot of times at the start of a project we communicate, “We’re going to do an experiment on set-based design to see its benefits,” and we leave it very open-ended. And like anything that you leave open-ended, that leads to different expectations from different people in the organization – those who are more familiar with LPPD obviously will have a more realistic expectation of what will come out of set-based design; senior managers will certainly be more interested in the end result, which may happen in the short term or, in accordance with product development, may happen years later.
So from their perspective, if they’re not seeing instant results, they may see the experiment as a failure when the jury may still be out. Or this may be a failure in that there was an experiment conducted with a specific target outcome, and we didn’t achieve that.
In the latter case, it’s important to remember that there is ALWAYS a silver lining that occurs. Very rarely do we have a complete and utter lack of learning in an experiment. If we do, we probably didn’t do something or just stuck with the status quo.
The first thing to usually do is do a good reflection with the experiment team to analyze what went well and what did not.