Skip to main content

Big Security Data to Big Security Intelligence

The concept of 'Big Data' is not new and was generally used to discuss the vast amounts of information resulting from the processing in areas such as astromony, weather or meteorological planning calculations.  The resulting processes produced petabytes of data, either in the form calculated results, or via the collection of  raw observations.  Approaches to store, manage and query the data vary, but many utilise concepts such as distributed and grid computing with vast amounts of physical storage, often directly attached (DAS), based on solid state or high speed SATA devices in order to allow rapid query execution.  Massively Parallel Processing was then applied in front of the data in order to execute rapid relational queries.

In recent years, networking, consumer and and social media applications have started to produce vast amounts of log, user and alerting information that needs to be stored and analysed.  For example, Walmart handles more than 1 million customer transactions every hour.  That equates to around 2.5 petabytes of data.  Those transactions not only need to processed accurately, but they will also need storing for accounting, management reporting and compliance mandates.  Social networking is another area requiring the storage of huge user data, such as news feeds, photo objects, pointers, tags and user relations such as followings, friendships and associations.

The main issues with storing such vast amounts of data are generally around being able to index, query and analyse the under lying data.  End users require search results in near real time response.  Analytics are expected to be contextual, with detailed and flexible trend, history and projection capabilities, that can be easily and simply expanded and developed.

Another producer of this big data concept is that of security appliances, devices and software.  Intrusion Protection Systems, firewalls and networking equipment will produce huge amounts of verbose log data that needs to interpreted and acted upon.  Security Information and Event Management (SIEM) solutions over the last 10 years, have developed to a level of maturity where log and alerting data is quickly centralised, correlated, normalised and indexed, providing a solid platform where queries can be quickly interpreted and results delivered with context and insight.

But as security data continues to increase, simply having the ability to execute a query with a rapid response is not enough.  The first assumption to this is that the query that needs to be run, is actually a known query.  That is, a signature based approach.  A set criteria is known (perhaps a control or threat scenario) which is simply wrapped within a policy engine, that is compared against the underlying data.

As security data starts to develop further and include identity, business and threat intelligence data, a known query may not exist.  The concept of the 'unknown unknowns' makes it difficult to be able to traverse vast amounts of data without knowing what trends, threats, exceptions or incidents really need attention or more detailed analysis.  The classic needle-in-a-haystack scenario, but this time needle is of an unknown, colour, size and style.

A simple example is analysing which entitlements a user should or should not have.  If an organisation has 100,000 employees, each with twelve key applications, with each application containing 50 access control entries, the numbers alone require significant processing and interpretation.  If the compliance mandate quickly requires the reporting and approval of 'who has access to what' within the organisation, a more intelligent approach is required.

This intelligence is in the form of having a more adaptable, contextual based approach to analysing the large volumes of data.  It simply wouldn't be effective to perform static queries.  A dynamic approach would include being able to automatically analyse just the exceptions held within a large data set, with the ability to 'learn' or adapt to new exceptions and deviations.

As attack vectors continue to increase, utilising internal and external avenues, security intelligence will become a key component of the information assurance counter measure tool kit, resulting in a more effective and pin pointed approach.

(Simon Moffatt)

Popular posts from this blog

Top 5 Security Predictions for 2016

It's that time of year again, when the retrospective and predictive blogs come out of the closet, just before the Christmas festivities begin.  This time last year, the 2015 predictions were an interesting selection of both consumer and enterprise challenges, with a focus on:

Customer Identity ManagementThe start of IoT security awarenessReduced Passwords on MobileConsumer PrivacyCloud Single Sign On
In retrospect, a pretty accurate and ongoing list.  Consumer related identity (cIAM) is hot on most organisation's lips, and whilst the password hasn't died (and probably never will) there are more people using things like swipe login and finger print authentication than ever before.

But what will 2016 bring?

Mobile Payments to be Default for Consumers

2015 has seen the rise in things like Apple Pay and Samsung Pay hitting the consumer high street with venom.  Many retail outlets now provide the ability to "tap and pay" using a mobile device, with many banks also offer…

Customer Data: Convenience versus Security

Organisations in both the public and private sector are initiating programmes of work to convert previously physical or offline services, into more digital, on line and automated offerings.  This could include things like automated car tax purchase, through to insurance policy management and electricity meter reading submission and reporting.

Digitization versus Security

This move towards a more on line user experience, brings together several differing forces.  Firstly the driver for end user convenience and service improvement, against the requirements of data security and privacy.  Which should win?  There clearly needs to be a balance of security against service improvement.  Excessive and prohibitive security controls would result in a complex and often poor user experience, ultimately resulting in fewer users.  On the other hand, poorly defined security architectures, lead to data loss, with the impact for personal exposure and brand damage.

Online-ification: The Role of Identity

The Wikipedia entry for Digital Transformation, "refers to the changes associated with the application of digital technology in all aspects of human society".  That is a pretty broad statement.

An increased digital presence however, is being felt across all lines of both public and private sector initiatives, reaching everything from being able to pay your car tax on line, through to being able to order a taxi based on your current location.  This increased focus on the 'online-ification' of services and content, drives a need for a loosely coupled and strong view of an individual or thing based digital identity.