It wasn't that long ago that we couldn't comprehend how to store and manage the vast volumes of data we create every day, never mind how to analyze it to reap its hidden value. Academia took care of detailed data analysis while businesses relied instead on good old-fashioned gut instinct.
Back then, data analysis was complicated and required experts with hard to find skills to own processes to ensure data was of a high enough quality and proper analytics were applied. If something in the process were to go awry, or if the data collected couldn't be validated, entire projects would have been at risk of being scrapped.
No sooner than computers became financially and widely accessible did the real business value of big data analytics become known. The gut instinct business had relied on previously were still relevant skills, but this new age of analytics set forth a "trust, but verify" business mentality and with it an exponential explosion in data volumes.
Over time, the value of data analysis became more apparent, and expertise from academia started bleeding into the business world. Costs to store data and to perform analysis came down too, making it more practical to do so. Soon we became more cognizant how being data-driven in our thinking and processes could reap significant benefits.
We could now apply new analysis techniques like regression and generalized linear models to large bodies of data. Iterative optimization techniques and maximization simulations -- techniques that were once thought to be only theoretical possibilities -- became everyday realities.
Then, with the advent of the Cloud, these innovations in data analytics steamrolled forward at breathtaking speed. Businesses no longer needed to shell out massive sums of money to build huge data warehouses to store and process data. Instead, they could send data to the Cloud at a fraction of the cost which had the added benefits of more directly justifying more investment in technology.
The Cloud also made it possible to scale storage and computing needs based on immediate business needs. Businesses didn't have to add, upgrade, or decommission clumsy servers; they could click an icon to spin up an instance instead.
So then vast bodies of data and the Cloud-based muscle needed to manage and process it became plentiful, and all was good.
But was it?
The specter of the human intuition needed to decide which data had value still lingered. It also created a bottleneck of sorts because, as volumes of data continued to grow larger and more complex, the pool of talent with the natural gut instincts and analytics skills dried up. And those that remained often lacked knowledge of the latest, bleeding edge technologies and techniques needed to process big data correctly.
Necessity, though, is often the mother of invention. So as data-oriented industries rode the wave of rapidly declining computing costs, they also came up with a new way of thinking. If it's a lack of human talent that's creating the bottleneck, why not train our machines to do a human's work for us? Doing so would not only eliminate a resources roadblock but also reduce the overhead associated with finding and retaining hard-to-find talent.
Necessity, meet "Artificial Intelligence" or as it's more oft referred to, "AI." With AI we can systematically automate data analysis using a virtual brain that explores networks to mine information and to train itself how to identify virtually any threat -- no matter how insignificant.
"AI technology" is a broad term that covers things like machine learning, deep learning, predictive and prescriptive analytics, automated written reporting and communications, and voice recognition and response.
Whatever it is called, an exciting new era in which machines autonomously structure data and extract information is upon us. It's an era where the complexity in data fades away, and analytics no longer overwhelms the business landscape. Investors are buying into this potential. Recent research shows that VC funding in private AI companies reached an $1B in Q2 2016, an all-time high. But for businesses, deciphering what's true and what's hype can be challenging, even from the mouths of experts. If you are in the 62% of enterprises that are expected to use at least some AI technologies by the end of this year, how can you make sure you made the correct choice?
Don't let the words fool you. Step back for a moment, and you'll realize that data analysis isn't all that much different than it was back in the day:
Making sure you've collected quality data, and knowing how methods of data collection can affect or even bias data sets.
Building information-enriched data sets and analytically tractable model features, then applying analytical and statistical techniques that are based on known data collection structures and inherent biases.
Checking models to make sure errors that might call into question the validity of the data are found and, when needed, cross-validating the performance of data to ensure predictive accuracy.
Monitoring ongoing model performance (including the learning path for models that are updated continuously) so that you understand the evolution of the model over time, and to ensure stability and performance of the model.
All of these concepts are relevant in coming up with analytic results that are accurate for machines as they are for human beings. Coming to that realization gives you a strong basis on which to try and untangle the buzzword salad that permeates data-oriented businesses today.
Machine learning is not going to magically solve all your data analysis and modeling problems, though. And it's not going to eliminate the need for human structuring, design or supervision of tasks in InfoSec. The role of these experts may transition into a higher-level function set, like what types of features to look for in the data you collect, the construction of algorithms and how they reason about data, and recognizing the constraints of these algorithms and guiding their application in threat detection constructs.
These tasks are indispensable and unavoidable, and machines can't overshadow the need to have experts on hand as a result. Organizations that find a way to mix human intuition with the power of machines will be the ones that reap the rewards of deeper data insights and significant scale.
Next blog post