How do we define this mindset and turn it into something teachable to other organizations?
LetÕs start here:
Are you into consuming security products or exploring data? Do you feel that you need a security appliance for everything? Do you say Ògive me the dataÓ or Ògive me out-of-the-box content, canned rules, signaturesÓ? Do you just want to be shown Òwhat you need to knowÓ or are you willing to figure what you need to know from the data you have? Would you rather learn Òwhat your data is trying to tell youÓ or Òwhat latest stuff the vendors have on saleÓ? At this point, if you just want Òa boxÓ, the path of big data analytics is not for you. Analytic mindset seems to determine the success of a big data initiative for security more than anything else. Those organizations that succeed with using big data for security are all subscribers to this view. They all state that for the foreseeable future, there will be no Òboxed security big data analyticsÓ products (except for some narrow and specific problems solved by specific tools).
Along the same line, somebody asked me one of those days ÒDo I need to toss out my SIEM and buy Òa big data productÓ? Ð NO, SILLY!!! You need to try using your SIEM to actually analyze the data inside itÉ. If you analyze the data inside your SIEM to its maximum potential, then you may need to look beyond that into other tools and approaches. But start from data exploration, not from tool replacement!
Therefore, the best analytics Òstarter packÓ is the one you can do on the data and tools you have. If you have RDBMS full of logs, flows or context data Ð start there. Leverage the data you have collected to make better decisions; use traditional BI tools on that database to see what emerges (some of the current Ôbig data for securityÕ champions started like that). In fact, if all you have is Excel and bunch of exported reports Ð well, start exploring there!
The evolution then continues like this: ask questions of the data you have -> get a useful answer Ð> become more data driven Ð> gather more data Ð> ask more useful questions.
Organization then start to naturally Òthink data firstÓ: new threat pops up? LetÕs go into our data and see what is up, then create new analytic approaches to detect and investigate it Ð rather than start whining Òwhat tool do I buy next?Ó No amount of Hadoop will give you big data analytics without a mindset. As I found out, this mindset and data curiosity is most important; by the way, mindset importance is also well-established for doing indicator hunting and anomaly detection, such as using network forensics and ETDR tools (also see Alert-driven vs Exploration-driven Security Analysis).
So, go and build your own data analytic discipline! Build analytic-centric and data-centric mindset Ð rather than buy or download any particular big data technology. Start data driven Ð not tool-driven (and, yes, Hadoop is a tool too Ð and the one often hard to implement, operate and utilize, especially in the absence of clarity of purpose or your goals). You cannot solve a mindset problem by buying technology; you need a mindset for leveraging data differently.
The only path is to shift the thinking, learn to be data-centric and data-driven and then solve problems that call for bigger data. Such culture change has to happen for the big data approaches to become pervasive across the industry. And yes, this includes willingness to explore, follow leads, and occasionally arrive at dead ends and algorithms that donÕt work.
In fact, most of my questions about the particular algorithms aimed at those few (REALLY few!) organization that do advanced analytics on large-scale security data resulted in no single list of Òtop useful algorithms.Ó Machine learning (ML), Bayesian, clustering, various data mining and text mining methods were mentioned, but none were highlighted as Òmust use.Ó What was a must? Again, it was a mindset and willingness to dip into a toolbox of algorithms to throw at dataÉ
Finally, some quick tips:
Got a SIEM? Go beyond vendor reports, run those queries direct to backend, extract and visualize. Got a little other data relevant to security? Try open source mining tools, write scripts to analyze and profile data, and look at the data and see what it is trying to tell you É To summarize, while conceptually, security is becoming a big data analytics problem, practically, it wonÕt become that for you if you keep investing in prevention and buying boxes.
There you have it! Now, GO EXPLORE YOUR DATA!
Datum: 18-11-2013 Anton Chuvakin Bron: Gartner Key word: online analytical processing Content: about what is Big Data