Deep in the Weeds: When Testing is Slow and Expensive

Controlled testing in modern systems is fairly straight-forward. There are many tools to handle statistical analysis, random population sampling, data collection, etc. With the number of web visitors routinely numbering in the millions, the statistical techniques are also greatly simplified because of the preponderance of data. What do we do though when data is very costly?

In this post, we'll take an overly simplified model and solve it using response surface modeling to find an approximate optimum with very little data. At the end of the post we'll discuss some possible caveats and some ideas for getting around those caveats.