Cloud App Security — Performance vs. Correctness

Way back when I was a kid, my Dad used to take my brother and I to a car show once in awhile and the two of us would always be wowed by all of the exotic cars on display (back then when most cars looked like this, it wasn’t hard to be wowed by even moderately exotic cars). But beyond how the cars looked, we’d always come away wondering, “which one is the fastest?” For us, the bottom line was which car could do zero to sixty in the shortest amount of time. Usually, such speculation would lead to endless arguments (there was no Wikipedia at the time to settle such arguments).

When it comes to technology solutions, we often quickly descend into the same singular means of evaluation. We may couch it other terms (what’s the download speed, how long will a reboot take, how long does it take to print, etc.), but at the end of the day we often care about little else other than how fast does it go? And in many ways this makes sense—we only have so many hours each day and we’d like to minimize how much time is spent waiting on our technology to do its thing. But I’d like to suggest that when evaluating technologies, especially Internet/cloud-related technologies and cloud app security, there’s more than just performance to consider. And to push the point further, I’d even assert that there are some things that are more important than sheer performance; namely, correctness, reliability, and security. In this post, we’ll start with correctness.

Way back in the spring of 1993 when the Megahertz Myth was widely embraced by technology consumers, Intel brought out their Pentium (“P5”) processor as the successor to their 80486 (“486”) processor. It was quite a CPU and represented a significant leap forward in performance. One of the notable improvements over the 486 was the enormous gains in floating point calculations (the part of the CPU that handles mathematical operations with numbers that have decimal places). In some cases, the performance was as much as 15x higher. But by the fall of 1994, it was discovered that the P5 processor had a flaw in that same speedy floating point unit: under certain rare calculations, it would return slightly inaccurate results. For the vast majority of people using a P5-based machine, this would have essentially zero impact. But as word spread (even CNN ran a story on it) and the flaw was blown out of proportion, Intel ultimately caved to public pressure and offered to recall all of the processors that had been sold. Ultimately, this cost the company a cool $475 million. In the end, it became clear that the heralded performance of the P5 meant nothing when compared to lack of correctness.

The same need for correctness applies to Internet technologies—especially those that are business critical. But how do you evaluate correctness? One way is transparency: if you are relying on complex cloud technologies to run your business and those cloud technologies rely on sophisticated analytics (and with big data showing up everywhere, who isn’t?), you want to make sure that you are given a clear explanation of how these analytics work. You can’t just assume that the “black box” is working because someone told you to “just trust us—it works.” This level of transparency has been the cornerstone—and success—of open source software and there’s no reason why you shouldn’t expect a certain level of transparency with your closed source cloud applications. And once you know how something works, you can better evaluate the results you get from it.

So does performance matter? Of course it does; a wrong algorithm choice can mean the difference between arriving at a solution in a reasonable amount of time and never arriving at a solution at all. And history has shown that some algorithms may not even be useful from a practical standpoint until an efficient solution is found (one of the most significant being the development of the fast Fourier transform as a means of efficiently computing the otherwise sluggish discrete Fourier transform). But if the algorithm is just wrong to begin with, no amount of performance is going to make up for it.

===

References:

http://en.wikipedia.org/wiki/AMC_Pacer

http://en.wikipedia.org/wiki/Pentium_FDIV_bug

http://en.wikipedia.org/wiki/Floating_point_unit

http://en.wikipedia.org/wiki/P5_%28microarchitecture%29

http://en.wikipedia.org/wiki/80486

http://en.wikipedia.org/wiki/Megahertz_myth

http://en.wikipedia.org/wiki/Fast_Fourier_transform

http://jeremykun.com/2012/07/18/the-fast-fourier-transform/