Governments do not typically send military personnel to war without first knowing the challenges they are facing. This much is logical. And by that logic, neither our intelligence services nor police forces seek to prevent issues of national security without first attaining a comprehensive understanding of the threats they face.
One of the challenges these institutions now contend with, however, is that they are dealing with an overwhelming amount of data. Getting that data out of the siloes they have long been stored in to transform it into actionable intelligence is challenging. Once that data has been analysed and made sense of, sharing it across the intelligence community has proven to be even more difficult. Both of these challenges affect issues of national security needlessly.
My colleague Jon Williams and I were privileged to be able to discuss this complex and evolving threat intelligence environment with members of the intelligence and policing community, last fall, at the National Security Summit 2017. Below is an overview of this presentation.
Threats to Critical Infrastructure Evolving and Becoming More Dangerous
Today, the nature of threats to critical infrastructure and democratic institutions is ever-changing. Historically, sovereign countries with state armies would wage war to determine the ownership of territory. Today, ‘Script Kiddies’ – or teenage hackers – can quickly and easily create malware to disrupt or destroy IT systems across the enterprise and public sector, including critical national infrastructure.
One of the most effective examples of this was the WannaCry ransomware, which took down a number of NHS systems briefly in May of this year. The attack hit over 200,000 victims and 300,000 computers and left NHS staff resorting to using pen and paper to record essential patient data. It also delayed critical surgical procedures that patients were waiting on.
However there is evidence that “script kiddy” actions have evolved into attacks by nation states. In the last two years we saw the Russian adversary group, Cozy Bear, infiltrate the unclassified networks of the White House, State Department, and US Joint Chiefs of Staff. And Fancy Bear, another Russian adversary group is blamed for the hacking of the Democratic National Convention (DNC) to leak emails from the then presidential candidate, Hillary Clinton. The firm Crowdstrike, hired by the DNC, alleges that Cozy Bear has ties to Russian Intelligence. If true, security becomes the forefront to national security.
Threats In the Shadows
Many of these threats organize and take shape via social media, or through encrypted services and the dark web, which aids hackers in their bids to hide in the shadows, away from the eyes of our national security services. In this environment, it becomes increasingly difficult to manage these threats. These non-state actors know what they are doing, they are professional and, therefore, where previously we knew where the threat was coming from, now it is far less clear.
What we do know is that, where there are commonalities in the techniques and technologies used by these perpetrators, there are commonalities in the techniques intelligence services, and police forces must use to stop them – this is tradecraft. Take safeguarding, for example. The techniques used in process to exploit vulnerable children are in essence the same techniques used for radicalization.
To identify this tradecraft and counteract these perpetrators, good data governance within the intelligence and policing communities is critical. This is an incredibly complex environment, which requires intelligence experts to work with data of different types (structured, unstructured), with different origins (open-source and private), of varying quality and sometimes questionable provenance, very quickly to determine what is real and what is fake, in order to act with the best information available.
Improving Tradecraft for Investigators and Intelligence Communities
At MarkLogic, we have worked with the intelligence and policing community to improve tradecraft for investigators. We have provided better tools and solutions that bring data together quicker, to link it to other data and enable intelligence workers to achieve better insights, faster. We have also worked with these communities to put processes in place for best practice where we’ve seen organisations that know how to use these data processes effectively to share that knowledge.
At one police force in the UK, we are bringing all of their intelligence data into one place, making it easily searchable and discoverable. In providing this 360-degree view of this force’s data, the data moves from being data to information and then becomes actionable intelligence.
For example, in relation to safeguarding, data concerning an emergency call can now automatically be linked via a single point of entry to all of the force’s data on recent missing persons, as well mapping and addresses and any intelligence attained via social media. Not only does this save time by shortcutting and automating data collection and analysis, but it also saves costs and enables more accurate earlier intervention. In this instance, this police force is projected to save eight per cent of its annual budget.
But the reality for most police forces and intelligence communities across the world is that they have an immense amount of data growing at an exponential rate, which remains siloed, and there is a reluctance to share it. These institutions have traditional data, big data, and they have dark data – which is data that we don’t know an awful lot about. Left in siloes, this data is not being used as efficiently as it should, but if we can bring this data together, fuse it and start linking it with other pieces of data that would not typically have been used, we can suddenly unlock previously unknown insights.
Making Data Accessible and Relevant
The institutions that protect our national security and public safety operate a large number of siloed applications and systems. The rigidity of the relational model does not allow these systems to be easily changed, thus proliferating the creation of siloed applications to deal with specific isolated data records (for example 911/command and control calls, arrests, case and custody management, missing persons reports and intelligence). But the problem they are dealing with has evolved.
Likewise, the stakeholders within these institutions have not changed in their outlook towards the data they have at their disposal. There is no incentive to share that data so that it can be linked with other data to unlock an insight. Instead, knowledge is deemed to be power, and so many of these institutions withhold data for their own use.
To ensure that our intelligence communities and police can both respond to and predict events before they happen, accurate data needs to be joined up and linked, providing single view of a suspect, victim, witness or vulnerable citizen, a history of their interactions with these agencies and their relationships to other people, locations and events. Threat management systems require good data governance systems, which need buy-in from senior stakeholders to ensure data is shared effectively. But beyond our shores, we need a commitment to further research and investment from governments across the world.
For More Information
Understanding Data Governance. This 22-page report from O’Reilly Media looks at best practices for creating a framework to meet the security and intelligence needs.