November 21, 2016

Data sharing systems used within the Five Eyes partnership

(Updated: November 28, 2016)

From the Snowden revelations, the general public learned about the Five Eyes partnership between the signals intelligence agencies of the United States, the United Kingdom, Canada, Australia and New Zealand, but details about this cooperation remained shrouded in secrecy.

Now, a batch of internal newsletters of the NSA's Signals Intelligence Directorate (SID), published last August by the website The Intercept, provides new information about various systems for sharing information, metadata, content and reports among the Five Eyes partners.

- From BRUSA to Five Eyes
- Joint Executive for SIGINT Interoperability (JESI)
- Information sharing: IWS
- Interoperable access control: PKI
- Sharing metadata: MAINWAY
- Federated metadata queries: GLOBALREACH
- Sharing content: TICKETWINDOW
- Sharing end reports: CATAPULT
- SIDtoday newsletters


From BRUSA to Five Eyes

The Five Eyes community grew out of the cooperation between Britain and the United States during World War II. On March 5, 1946 both countries signed the BRUSA (now known as UKUSA) Agreement on communications intelligence cooperation. This is not only about collecting signals intelligence, but also about security measures, like the use of codewords to restrict access to highly sensitive sources and reports.*

In June 1948 the UKUSA Agreement was established, which Canada, Australia and New Zealand signed on along with the UK as "Second Parties". A separate agreement between Canada and the USA (CANUSA) was signed in November 1949, followed by one with Australia in September 1953.*

Finally, in May 1954, the BRUSA Agreement was renamed UKUSA, which became also the name for the complex network created by these often overlapping agreements, appendices and memoranda of understanding.* Australia acted on behalf of New Zealand until the latter became a full member in 1955 or 1977.

The (signals) intelligence agencies that have less close bilateral relationships with NSA are called Third Party partners. Currently, there are over 30 Third Party partners, see: NSA's Foreign Partnerships

When the term Five Eyes (for classification purposes abbreviated as FVEY) came in use is not clear, but the SIDtoday newsletter from August 5, 2003 confirms that "Five Eyes" is derived "from the "US/UK/CAN/AUS/NZ EYES ONLY" caveat that limits the distribution of SIGINT reports to the listed Second Party countries."

The initial network of bilateral relationships between the five partner countries was eventually transformed into a "group partnership" in 1993 - as was revealed in a newsletter from August 25, 2003. It's not explained what this means, but it's sounds like a shift to a more multilateral framework for cooperation among eachother.


The British-U.S. Communication Intelligence Agreement from 1946
(the full text as pdf - click to enlarge)


Joint Executive for SIGINT Interoperability (JESI)

In 1998, the agencies of the Five Eyes group established the Joint Executive for SIGINT Interoperability (JESI, pronouncesd as "jessy"). In the newsletter from August 25, 2003, JESI is described as a "multi-national executive body responsible for ensuring continued interaction and interoperability among the five SIGINT partners". JESI doesn't have its own staff, it's just a collaboration platform.

Officials from the Five Eyes agencies also meet at an annual JESI conference. In July 2003 this meeting was held in the Australian capital Canberra and was focused on the mission objectives of the partner agencies and how they relate to the 5-EYES SIGINT Partnership Business Vision, which was published earlier that year. They addressed the following topics:
- Mission collaboration and knowledge sharing
- Enabling SIGINT operations through information assurance
- Exchange of finished intelligence
- Maintaining business continuity



For a more efficient cooperation among the Five Eyes partners, the following systems were created, most of them initiated by JESI in 2002-2003, as described in the SIDtoday newsletter from August 25, 2003:

Information sharing: IWS

A collaboration tool called InfoWorkSpace (IWS) was created to exchange information between NSA, the US military and partner countries during Operation Enduring Freedom in Afghanistan.

IWS is a software tool that provides chat communications as well as audio and video conferencing, file sharing, virtual whiteboards, and shared desktop views through desktop computers connected to a secure network.* As within the Five Eyes it's about signals intelligence, IWS most likely ran, and maybe still runs on NSANet.

According to a SIDtoday newsletter from September 10, 2003 IWS was already used by over 4000 NSA and their Second Party counterparts at the working levels. They collaborated on topics like Operation Enduring Freedom, international terrorism, real-time collection coordination, SIGINT development and multi-intelligence tasking.

This succesful use of IWS led JESI decide that the system should also be used at leadership-level. As of 2003, the SIGINT directors of the Five Eyes partners would use IWS to enhance their collaboration on subjects ranging from current intelligence objectives to future collection planning. They would get access to one of the IWS servers managed by NSA, codenamed VOTEDOOR.


InfoWorkSpace, here being used during the Joint
Expeditionary Force Experiment (JEFX) 2006
(photo: CHIPS Magazine)

In another newsletter from December 19, 2003, it is said that not long before, the SIGINT directors of NSA, the Canadian CSE, the Australian DSD and New Zealand's GCSB held their first virtual meeting using the InfoWorkSpace tool. However, their counterpart at "GCHQ was unable to attend due to a computer failure."

According to the newsletter, this first meeting lasted over an hour and was mainly about "efforts against terrorism, especially ways to extend cooperation across the SIGINT community, and to include the HUMINT [Human Intelligence] community". A next virtual meeting using IWS was scheduled for the middle of January 2004.

The tech website Motherboard found the following video presentation of the InfoWorkSpace (IWS) tool, which was developed by ezenia!, a small company from Salem, New Hampshire:





Interoperable access control: PKI

In order to give Second Party employees access to joint collaboration systems, JESI pushed the partner agencies to deploy interoparable Public Key Infrastructure (PKI). The NSA's PKI is a comprehensive encryption system to protect classified information against:

- Unauthorized disclosure and modification through digital signing
- Unauthorized access through access controls and authorization services
- False user idenfications

An SIDtoday newsletter from July 8, 2003 explains that the new PKI system would replace the ICARUS e-mail encryption system by October 2003. A valid PKI certificate was also needed to use applications like Peoplesoft and CONCERTO. The latter is NSA's internal personnel system, which has separate parts for human resource and security clearance information.

The new PKI certificates were first issued to NSA employees who were US citizens and held a blue, green, or gold badge. Later, PKI certificates would also be issued to employees of Second Party agencies and to non-US citizens. This PKI system seems to be a software solution without two-factor authentication with a token like the CAC-smartcard of the US military.


Sharing metadata: MAINWAY

Since 2006 it was thought that MAINWAY was a repository just for telephone metadata, but based upon recently leaked and declassified documents, it was explained on this weblog that MAINWAY also contains internet metadata as well as the domestic phone records NSA previously collected under the authority of Section 215 of the USA PATRIOT Act.

Rather unexpected, the SIDtoday newsletter from August 25, 2003 now also reveals that "MAINWAY, a system that uses phone call contact chaining to identify targets of interest, was provided to each of our partners. The partners now supply additional contact information to the database to enhance the joint ability to identify targets".

So MAINWAY is not only fed with the domestic US telephone records and the foreign telephone and internet metadata collected by NSA, but also with foreign metadata provided by GCHQ, CSE, DSD and GCSB. According to the quid pro quo rule for intelligence cooperation, all Five Eyes partners can apparently also query the MAINWAY database for their national security interests.

However, Second Party analysts have no access to the domestic US phone records, but so far there are no documents that mention this explicitly (recently published dataflow diagrams show that MAINWAY has separate BRF [BR FISA or Section 215 records] partitions though).

A GCHQ presentation from 2010, which was published earlier, shows the user interface of the IMMINGLE tool with check boxes for direct access to various metadata repositories, including MAINWAY II:


User interface of GCHQ's IMMINGLE tool for access to various metadata repositories
(the full presentation as pdf - click to enlarge)



Federated metadata queries: GLOBALREACH

Besides direct access to the metadata contained in MAINWAY, analysts from the Five Eyes partners can also use the GLOBALREACH system. In documents that were published earlier, this system is described as a "federated query service via accounts and access verified by PKI certificates" which probably runs on NSANet.

As a federated service, GLOBALREACH can be used to query multiple metadata databases with one single login. A 2005 document says that for example CIA would provide metadata "from non-SIGINT sources for inclusion in the dataset searched by GLOBALREACH" and it's likely that it can also search the foreign metadata from MAINWAY.

A pilot for a similar federated query tool codenamed ICREACH for the US Intelligence Community (IC) was started in 2007. After NSA "persuaded other US IC agencies to make almost 100 bn previously NOFORN records shareable with the 5-eyes via GLOBAL REACH", agreements were reached with the Second Party agencies, whereafter they started to provide ICREACH with telephony metadata, making them accessible to over 1000 analysts across 23 US intelligence agencies.

After establishing ICREACH, these analysts got access to more communication modes (including landline, mobile, satellite and VoiP call records), the types of metadata increased from 5 fields to 33 fields and the total volume rose from 50 billion to over 850 billion records - ca. 126 billion of which from Second Party partners. 1-2 billion records were said to be added daily, so by now, ICREACH may provide access to over 5 trillion metadata records.


Architecture of the ICREACH federated query system
(the full presentation as pdf - click to enlarge)



Sharing content: TICKETWINDOW

An older collaboration system for the Five Eyes partners is described in a SIDtoday newsletter from November 7, 2003: TICKETWINDOW. This system was established in 1999 by the NSA's Data Acquisition division to enable reciprocal data sharing with Second Party parters - without revealing sensitive sources and collection methods, which often restricted data sharing. Within TICKETWINDOW, NSA shares most data, but the other partners also contribute from their own collection.

In 2003, TICKETWINDOW was regarded a success story: new sources from the partner countries helped NSA to be more productive, while for the Australian DSD, more than 40% of their product reporting was from TICKETWINDOW collection, particularly from NSA collection. Both the British GCHQ and the Canadian CSE had doubled their output of TICKETWINDOW reports in 2002. Maybe this system is somehow related to the mysterious SIGADs starting with DS, which seem to denote collection by Second Party countries.

A similar data sharing system for the SIGINT Seniors Europe (SSEUR) group of Third Party partners is the Signals Intelligence Data System (SIGDASYS).


Sharing end reports: CATAPULT

Finally, there's also a system for sharing intelligence reports among the Five Eyes partners. According to a newsletter from May 8, 2003, NSA and the Canadian CSE set up a prototype portal to exchange SIGINT products between NSA and its Second Party partners under the codename CATAPULT.

The CATAPULT portal "contains all 2nd party viewable product shared with CSE to include multimedia reporting, CRITICOMM released product, and SIGINT on Demand (SOD) items", all of which is accessible from NSANet through a browser interface. CATAPULT is based on CSE's SLINGSHOT system, which delivers SIGINT reports to Canadian "customers" like policy and decision makers.

CATAPULT was brought under the JOURNEYMAN umbrella program for modernizing the way SIGINT analysts can write and disseminate their reports. As CATAPULT started as a prototype, it may have been replaced by a system that includes all Five Eyes partners.


Besides the systems described above, JESI also initiated the creation of several protected websites to allow employees of the Second Party agencies to securely share data within specific communities of interest.

As close as the cooperation between these agencies may have become, the sharing mechanisms are still meant to support each member's foreign intelligence tasks. The Five Eyes are not a body of its own with its own goals or targets, like for example a rather ridiculous target list on Wikipedia suggests.

Also, the data sharing system TICKETWINDOW isn't the successor of ECHELON, as Wayne Madsen wrote on the website Intrepid Report. ECHELON was (and under the name FORNSAT still is) a worldwide network of satellite intercept stations to provide in the information needs of each of the Second Party countries.


SIDtoday newsletters

In May 2016, The Intercept started publishing large batches of documents from the Snowden archive, to begin with the SIDtoday newsletters from 2003, all the way to the most recent available ones from 2012. A second batch came in August 2016 and so far, a total number of 429 SIDtoday newsletters have been published, from March 2003 to July 2005.

These newsletters are an interesting source for historical research as they add or confirm many details about NSA. Although some of them are about operations that could be controversial, taking away full nine years of SIDtoday newsletters isn't proportionate and forms an example of where Snowden wasn't very selective.



Links and sources
- The Intercept: All published editions of SIDtoday
- About Canada and the Five Eyes Intelligence Community (pdf)
- Martin Ruder: Hunters and Gatherers: The Intelligence Coalition Against Islamic Terrorism
- NSA: UKUSA Agreement Release 1940-1956

October 19, 2016

With NSA contractor Martin arrested, other leakers may still be at large

(Latest UPDATE: October 31, 2016)

Earlier this month we learned the name of a second person who stole top secret documents from the US National Security Agency (NSA). After Edward Snowden admitted doing so publicly in June 2013, the FBI has now arrested the 51-year old Harold T. Martin III at his home in Maryland.

Martin hoarded lots of classified documents, not only from NSA but also from a number of other military and intelligence agencies. The FBI is still comparing them with those from the recent Shadow Brokers leak and a range of other NSA leaks from the past few years, but given what's known now, it seems likely that at least one other leaker is still at large.



The house of Harold T. Martin III in Glen Burnie, Maryland
(photo: Jose Luis Magana/The Associated Press)


The New York Times reported that when the FBI raided Martin's house on August 27, they found paper documents and many terabytes of highly classified information, even going back the 1990s. At least six documents were from 2014. It was reported that Martin first took the classified documents on paper, later on CDs and more recently on thumb drives.

The reason why Harold Martin brought home and stored such large numbers of top secret documents isn't yet clarified. One suggestion is that he may have used them for research for his dissertation about "new methods for remote analysis of heterogeneous & cloud computing architectures", which he was working on at the University of Maryland.


Documents from multiple agencies

It should be noted that not everything Martin stole comes from NSA. In the official charges there are no names of the agencies where the documents come from, they are only described as highly classified, including ones that are marked as Top Secret and Sensitive Compartmented Information (SCI).

With the documents going back to the 1990s, he may well have started hoarding them from the places where he worked in those days. From 1987 to 2000, Martin served at the US Navy, achieving the rank of lieutenant, but he left active duty in 1992.

As the Washington Post found out, he then took a variety of tech jobs with government contractors, like at Computer Sciences Corp. (CSC) somewhere in the 1990s and later, until 2009, at Tenacity Solutions, for which he worked at the Office of the Director of National Intelligence (ODNI). Over the course of 18 years, Martin worked for a total of 8 different defense contractors.

In 2009, Harold Martin started to work for Booz Allen Hamilton, for which he was a contractor at NSA from 2012 to 2015, when Booz transferred him to the Pentagon’s Office of Acquisition, Technology and Logistics (AT&L), which is responsible for often highly sensitive and classified procurement programs. There he stayed until the moment of his arrest last August, after which he was also fired by Booz.

Officials have meanwhile said that Martin took classified documents not only from NSA, but also from his other workplaces, including ODNI and AT&L.

It's interesting as well that in the charges against Martin, a whole paragraph is dedicated to the at least six documents from 2014, which are described as being produced "through sensitive government sources, methods, and capabilities". As signals intelligence is traditionally seen as the most sensitive capability, maybe just these six documents are from NSA.



The building of the Office of the Director of National Intelligence (ODNI)
where Harold Martin worked as a contractor before 2009
(photo: Microsoft, via Cryptome.org - click to enlarge)


Shadow Brokers investigation

After the "Shadow Brokers" disclosed a large set of secret NSA hacking tools last August, the FBI began investigating this leak. At the same time there was a lot of speculation: was NSA hacked from the outside? Had an NSA hacker been sloppy? Were the tools leaked by an insider? Maybe the same insider responsible for earlier leaks that hadn't been attributed to Snowden?


On September 22, it was reported that during the FBI investigation, NSA officials had said that a former agency operative had carelessly left the hacking tool files available on a remote computer, where Russian hackers found them. If that's correct, then it seems likely that the FBI traced Harold Martin when they were looking for that careless NSA hacker. It has not yet been confirmed that Martin was that person though.

Harold Martin was working at NSA's hacking division TAO around the time when the tools were considered to be left exposed, somewhere after October 18, 2013, but a former TAO hacker told the Washington Post that Martin "worked in the unit’s front office carrying out support roles such as setting up accounts, not conducting actual operations."

Even if Martin was the man who left the hacking tools exposed, then we still don't know who found them and published them under the name Shadow Brokers. It's not very likely that this was done by Martin himself, as Shadow Brokers published additional messages on August 28, October 1, October 15, and October 31, when he was already in custody. The actual publication can therefore be the work of for example Russian, Iranian or North Korean hackers or even independent hacktivists.


Other sources?

Could Harold Martin also be the source of earlier leaks, that were not attributed to Edward Snowden? In theory he could have been that "second source" next to Snowden: none of these other leaked documents (like the TAO catalog, XKEYSCORE code, tasking lists and end reports) are newer than 2015, when Martin left NSA. Contrary to this Martin is described as very patriotic, which doesn't fit the fact that these particular leaks were clearly meant to harm and embarrass the US and NSA.


Also, Martin hasn't (yet) been charged with espionage or the attempt to provide classified information to a third party or a foreign government - which doesn't seem something the US government would leave out or keep secret after the recent and unprecedented statement in which the Office of the Director of National Intelligence accused Russia of hacking the Democratic National Committee (DNC) and other political organizations.

Should the FBI investigation confirm that Harold Martin was only responsible for leaking the NSA hacking tools (after which unknown others published them) and that none of his documents were provided to foreign intelligence agencies or showed up in the earlier revelations, then there's most likely yet another leaker from inside NSA.

The Shadow Brokers leak standing alone and not related to the earlier non-Snowden leaks is of some importance, because only among the stuff published by the Shadow Brokers there are files with a date (October 18, 2013) after the day that Snowden left NSA (May 20, 2013).

This means that when Harold Martin is the initial source of the Shadow Brokers files, we can no longer exclude the possibility that the earlier leaks do come from the Snowden trove. If that would be the case, then someone with access to them went rogue and had them published on his own account. But it should also be noted that both Glenn Greenwald and Bruce Schneier explicitly said that some of these leaked documents did not come from Snowden.

The more likely option is therefore that there's still another leaker at large, someone with a more evil intent than Harold Martin and Edward Snowden - a conclusion which is not very comforting and which also raises questions about NSA's internal security...



Some NSA buildings at the Friendship Annex (FANX) complex near Baltimore
(photo: live.com, via Cryptome.org - click to enlarge)


NSA's internal security measures

The NSA's hacking division TAO, where Harold Martin worked for some time, is apparently not located in the well-known NSA headquarters building at Fort Meade, but in one or more leased office buildings outside, one of them at an office complex called Friendship Annex (FANX) near Baltimore. TAO also has units at NSA's four Cryptologic Centers across the US.

Entrance to the highly secured TAO headquarters building is strictly controlled: one has to go through an imposing steel door, protected by armed guards, and entrance is only possible after entering a six-digit code and passing a retinal scanner to ensure that only specially cleared individuals are allowed in.

Such security measures are more aimed at keeping outsiders out, than at insiders in. And when it comes to finding inside moles of hostile foreign intelligence agencies, the NSA is also said to have a rather bad track record. The Manning and Snowden leaks made NSA painfully aware of this and so preventive insider-threat detection programs were put in place.

It's not clear whether these new systems failed in the case of Harold Martin, or that they simply weren't yet implemented at the TAO location where he worked - anti-leak software that was designed by Raytheon to "spot attempts by unauthorized people to access or download data" was also not yet installed at the NSA facility in Hawaii when Snowden was working there.

Tracking what employees are doing inside is one thing, checking what they take out is another. But according to The Washington Post, the NSA (like other agencies) does not impose universal checks of personnel and their belongings as they enter and leave agency buildings. Security guards only conduct random checks and use their discretion in order to keep en build the trust of the employees.

"If you have a bag full of stuff, you’re probably going to get stopped" said a former TAO operator to the Post, but, in general, "Disneyland has more physical security checks than we had". This was confirmed by two other former NSA employees, saying that "nobody does pocket checks" and that "Anything that could fit in a pocket could go out undetected".

It would also take hours to screen every person leaving NSA buildings, and because the vast majority of employees go through extensive vetting, so there's an inherent amount of faith in staff at the agency. Besides checks, NSA facilities will also have detection gates, but it seems that it was easier for Snowden to walk out with his thousands of documents than many would have thought.

As former NSA general counsel Rajesh De explained, it is unlikely "you’re going to be able to stop every incident of somebody taking documents if they’re determined to do so. But the real question is how quickly can you detect it, how quickly can you mitigate the harm of any such incident."



An old sign inside the NSA headquarters building
showing what kind of items are not allowed in.
(screenshot from a documentary about NSA)


Conclusion

Harold Martin stole a lot of classified documents from multiple military and intelligence agencies where he worked over the past 20 years, with maybe just a small number from NSA. The still ongoing FBI investigation has to make clear whether Martin was responsible for exposing the TAO hacking tools.

If not, then there has to be yet another careless NSA employee, but then it's also still possible that the hacking tools came from a source responsible for a range of earlier leaks. So far it seems that Martin isn't the source of those earlier leaks, which means that the so-called "second source" is still at large.

The case of Harold Martin also made clear that security measures at NSA, and other US agencies, were not as strict and tight as outsiders would have expected: even for someone without a strong ideological or financial drive like Martin it was apparently not that difficult to regularly walk out with top secret documents.

Many things have not yet been confirmed or clarified, but at least the Shadow Brokers leak and the subsquent arrest of Harold Martin created more awareness among the American public of the fact that there have been more leaks than just those from Snowden.

In August 2014, Bruce Schneier was probably one of the first who identified a second and a third leaker besides Snowden. Many more similar leaks followed and a full list of them was compiled on this weblog in December 2015 (still being updated). As an excerpt of this listing, a short overview of the most important non-Snowden leaks was published in The New York Times last week.

Update #1:

Shortly after this blog posting was published, The New York Times came with a new report saying that the volume of classified documents Harold Martin had in his possesion seems larger than those stolen by Edward Snowden and even than those of the Panama Papers from 2015.
FBI investigators apparently also found that the TAO hacking tools were among Martin's documents, but because he is not very cooperative, it is still not clear how they came in the hands of the mysterious Shadow Brokers, who subsequently published them. So far there's no evidence that Martin was hacked or that he sold information.
He seems to have hoarded all these documents in order to get better at his job, as he is described as someone who imagined himself a top spy and an important player in the world of digital espionage.

Update #2:

On October 20, it was reported that the FBI had found the huge amount of 50 terabytes of data at Martin's home, but it is not yet clear how much of that is actually classified. Also found were "hard-copy documents that were seized from various locations during the search that comprise six full bankers’ boxes worth of documents" with many of the documents marked Secret and Top Secret.
One document was marked Top Secret/SCI and had this additional caveat at the top of the document: "THIS CONOP [Concept of Operation] CONTAINS INFORMATION CONCERNING EXTREMELY SENSITIVE U.S. PLANNING AND OPERATIONS THAT WILL BE DISCUSSED AND DISSEMINATED ONLY ON AN ABSOLUTE NEED TO KNOW BASIS. EXTREME OPSEC [Operational Security] PRECAUTIONS MUST BE TAKEN" - Martin had no need to know for this operation.

Update #3:

Harold Martin appeared in court for the first time on Friday, October 21. There, his lawyer said that things like an "unlocked garden shed, stuffed with more classified documents than the contractor [...] could ever read, might be a symptom of a mental disorder" - and also that keeping top secret material in plain view in his home and car was not the conduct of a spy or a political activist.
Although he was charged with the relatively minor criminal offenses of theft of government property and unauthorized retention of classified material, Martin had to stay in jail because he could be a threat to national security as investigators couldn't rule out that he might have hidden classified information in other, yet undisclosed locations.
Even after seven weeks of investigation, the FBI was still not able to show whether Martin gave any of his documents to anyone else, nor could they link him to the Shadow Brokers.

Update #4:

A legal document filed by federal prosecutors on October 27 says that the information stolen by Harold Martin included numerous names of intelligence officials working under cover outside the United States. It is not clear whether these officials were from NSA or from other US intelligence agencies where Martin had worked as a contractor.

We also don't know how many documents Harold Martin actually stole from NSA: everything that matters within that agency is classified under the SCI compartment SI (Special Intelligence), but so far, the FBI investigation only mentioned very few documents that were classified as Top Secret/SCI.

To be continued...


Links and Sources
- New York Times: N.S.A. Appears to Have Missed ‘Big Red Flags’ in Suspect’s Behavior
- John Schindler: It’s Time to Rename NSA the National INsecurity Agency
- The Washington Post: NSA contractor thought to have taken classified material the old-fashioned way
- Daily Beast: Democrats Say WikiLeaks Is a Russian Front, U.S. Intelligence Isn’t So Sure
- Defense One: Data-Theft Arrest Shows that Insider Threat Remains Despite Post-Snowden Security Improvements
- John Schindler: Has the Russian Mole inside NSA finally been arrested?
- New York Times: N.S.A. Suspect Is a Hoarder. But a Leaker? Investigators Aren’t Sure.
- The Cipher Brief: First on The Cipher Brief: Snowden's Boss Shares Lessons Learned

September 22, 2016

Secret report reveals: German BND also uses XKEYSCORE for data collection

(Updated: November 12, 2016)

Over the past few years we learned a lot about Germany's foreign intelligence service BND, although not from leaks, but from the public hearings of the parliamentary commission that investigates NSA spying operations and its cooperation with German agencies.

Recently however a secret government report was leaked to German media, which not only identifies violations of the data protection act but also reveals the codenames for several BND systems and the fact that BND uses the American XKEYSCORE system not only for analysis, but also for collection purposes.

Here, the new information from the secret report is combined with things we know from earlier sources and reportings.

- A secret report
- The SUSLAG liaison office
- Selectors provided by NSA: TND and SCRABBLE
      - BND's selector database: PBDB
- Operations SMARAGD and ZABBO
- Metadata analysis: VERAS
- Analysis and collection: XKEYSCORE
- Integrated analysis: MIRA 4
- Legal defects


The BND satellite intercept station at Bad Aibling, Germany
(Photo: AFP/Getty Images)


A secret report

The report that now has been published goes back to September 2013, when the then federal data protection commissioner Peter Schaar ordered a thorough inspection of the BND satellite intercept station in Bad Aibling, which took place on December 3 and 4 of that year.

In October 2014, Schaar's successor Andrea Voßhoff conducted a second visit to Bad Aibling, which in July 2015 resulted in an extensive and detailed report (German: Sachstandsbericht) about all the systems used at this BND station. This report was (and still is) classified as Top Secret.

Additionally, Voßhoff made a legal assessment based upon the Sachstandsbericht. This was finished in March 2016 and sent to then BND president Schindler and the federal chancellery. It was classified as Secret, but was leaked to regional broadcasters NDR and WDR and a transcription of the full document was published by the digital rights platform Netzpolitik.org on September 1.

Both reports are about the cooperation between BND and NSA, which goes back to 2004, when the Americans turned their satellite intercept station Bad Aibling (codenamed GARLICK) over to German intelligence. In return, BND had to share the results from its satellite collection with the NSA, for which the latter provided selectors, like e-mail addresses, phone numbers, etc. of the targets they were interested in.



Google Maps view of the Mangfall Barracks in Bad Aibling, Germany.
The building at the very top seems to be the BND facility,
the one nearby with the white roof NSA's "Tin Can".


The SUSLAG liaison office

After taking over the Bad Aibling satellite station, BND seems to have moved the control facility to the nearby Mangfall Barracks, which were taken over from the German armed forces (Bundeswehr) in 2002. For the Special US Liaison Activity Germany (SUSLAG), which is the liaison office of NSA for Germany, a new highly secure container building was built on the Mangfall Barracks premises in 2003 (nicknamed "tin can" or Blechdose).

According to the commissioner's report, the SUSLAG building and the building with BND servers and equipment are connected through a 100 MBit/s fiber optic cable. SUSLAG also has a technical data link to the NSA's primary communications hub in Europe, the European Technical Center (ETC) in the Mainz-Kastel district of the city of Wiesbaden.

Cooperation between the US and Germany in the Joint SIGINT Activity (JSA, 2004-2012) took place inside the BND building, for which NSA personnel had access permissions. After the JSA was terminated, SUSLAG personnel kept their entrance rights for the BND building, but it has separate rooms for highly sensitive information to which none of the Americans have access.

A letter from BND from October 15, 2015 says that at that moment, 10 people from NSA worked at SUSLAG, with following access rights:
- 2 have access to building 7 (SUSLAG) only
- 4 have access to building 7 and building 4 (Administration)
- 4 have access to building 7 and building 8 (BND)

The SUSLAG building is only used by NSA personnel and BND claims that the data protection commissioner has no jurisdiction over the SUSLAG, but she disputes that and says the SUSLAG building is simply part of the BND complex. She also regrets that SUSLAG doesn't recognize her oversight authority.




Selectors provided by NSA: TND and SCRABBLE

For the satellite interception in Bad Aibling, some 4 out of 5 selectors come from NSA, the rest from BND. According to Süddeutsche Zeitung, NSA provided BND with roughly 690.000 phone numbers and 7,8 million internet identifiers between 2002 and 2013. That is an average of something like 60.000 phone numbers and 700.000 internet identifiers a year, or 164 phone numbers and over 1900 internet identifiers each day.

From the parliamentary hearings we already knew that BND personnel pulls the American selectors from an NSA server, and the commissioner's report now reveals that this server is in NSA's ETC in Wiesbaden. On this server BND puts back any results for these selectors. These data transfers from and to ETC go through the SUSLAG facility, but BND is able to get direct access to the NSA server in Wiesbaden through an FTP-gateway (a "BACOM system").

Selector databases

From an earlier parliamentary hearing we know that BND stores the selectors from NSA in two databases: one for IP selectors (from NSA only), and one for telephone selectors (from both NSA and BND). Each agency had access to its own IP database; the phone database was managed jointly, but BND could only approve or disapprove NSA selectors, and NSA could only do so with those from BND.

The names of these databases were not known until now, but the commissioner's report mentions them, along with some additional details:
- Target Number Database (TND), which exists since 2008 and holds the telephone selectors from both NSA and BND. The latter either come from BND's own tasking database PBDB or are provided by domestic security services.

- SCRABBLE, which only holds selectors for packet-switched (internet) communications provided by NSA, after their format has been converted. These selectors initially had no description (Deutung, like a justification for the target). Because of this, BND temporarily stopped using them as of May 2015, and for the commissioner any results from them are unlawful because BND was not able to determine whether they are necessary for its mission.

Their names indicate that these database systems were provided by NSA, and together with the fact that they also contain NSA-provided selectors, this is likely the reason why these names were never mentioned during the parliamentary hearings - unlike those of BND's own systems.
Updates:

It was noticed that TND and SCRABBLE were actually mentioned once during the parliamentary hearings, when former BND president Schindler said that "the US has [its own] databases TND and SCRABBLE".

- PBDB - During a parliamentary hearing on November 9, 2016 it came out that BND's own tasking database PBDB (PersonenBezogene DatenBestände) became operational in the Summer of 2014, after a test period that started late 2012. Both in this system and in the previous system, it is/was logged when for example a selector was deactivated. An even older system had no such logging capability. The PBDB is maintained by the T2-branch from BND headquarters. Analysts can enter any selectors (often multiple ones for a particular target) into PBDB that they assume useful for foreign intelligence purposes. Newly entered selectors are checked (through the DAFIS system) at BND headquarters to make sure they don't pull in German communications. Results generated by approved and activated selectors seem also to be stored in the PBDB database, where they can be accessed by groups of 4 to 5 analysts working on the particular topic. After it came out that BND itself also used selectors related to partner countries, those selectors were moved to a separate partition (called Gruppenliste) of the PBDB database in October 2013, so they couldn't be tasked anymore.

Approval

Before being stored in the SCRABBLE and TND databases, both the telephone and internet selectors have to pass the DAFIS filtering system, which checks whether they belong to German citizens or companies or may otherwise contradict German interests. Accordingly, the selectors are marked as "allowed" or "protected".

Those marked "allowed" are subsequently being activated ("tasked") on the actual data collection systems. The report says that for this, hard selectors like phone numbers and e-mail addresses can be freely combined with content search terms (Inhaltssuchbegriffe) like key words, which could refer to the GENESIS language used for more complex XKEYSCORE searches.

According to the report, selectors marked as "protected" are send back to NSA and are also deactivated in the TND and SCRABBLE databases - to make sure that they won't get activated when NSA provides them a second time (this confirms that there's no separate database (Ablehnungsdatei) with rejected selectors as was suggested during the earlier parliamentary commission hearings).

BND refused the data protection commissioner access to TND and SCRABBLE, so she wasn't able to check the individual selectors. She regarded that as a massive restriction of her supervision authority.



Operations SMARAGD and ZABBO

Selectors that have been approved are send to the systems that filter out communications that match those selectors. Some of these systems are in Germany, others are abroad. The report of commissioner Voßhoff for the first time discloses two specific data collection operations and their codewords:

- SMARAGD, a cable tapping operation somewhere outside Europe and in cooperation with another foreign intelligence agency.

- ZABBO, collection in Bad Aibling of satellite communications from Afghanistan.

There's no explanation for why the commissioner only mentions these two operations. The satellite antennas in Bad Aibling undoubtedly collect from many more countries, but maybe these are the only operations from which, during the investigation period, data were shared with NSA.

SMARAGD = WHARPDRIVE ?

The way SMARAGD is described perfectly fits a certain type of operations in which a 3rd Party partner of NSA like in this case BND, cooperates with yet another country that secretly provides access to data traffic, which is then also shared with NSA. According to the book Der NSA Komplex, BND and NSA conducted about half a dozen of such operations in recent years.

In its english version of the news report about this issue, the website Netzpolitik.org points to an NSA document that was published earlier by Der Spiegel. In it, we see EMERALD mentioned as an alternate codename for the NSA operation WHARPDRIVE, which is exactly such a trilateral program in which a third secret service participates.

WHARPDRIVE was still active in 2013, but in the Spring of that year, employees of the private company that operated the communication cables, accidently discovered the clandestine BND/NSA equipment, but the operation was rescued by providing a plausible cover story.*

The NSA report from April 2013 however said that "WHARPDRIVE has been identified for possible termination due to fiscal constraints", but this may have been coincided with the exposure of the program in the book Der NSA Komplex in March 2014.

It should also be noted that Netzpolitik.org came up with this identification by translating the German codename SMARAGD into its English equivalent EMERALD. It is possible that the Americans also translated the German codeword SMARAGD into EMERALD, but just as likely is that it's a different program (maybe as a successor with the same set-up).

Update:
During a parliamentary hearing on November 9, 2016, member of parliament Renner said that SMARAGD is identical with EMERALD and that the operation was deactivated after Snowden, because it was mentioned in documents. BND-employee R.U. said that a cable access which terminates in Bad Aibling (likely the one from the SMARAGD operation), provided just a minimal data stream, by fault of the foreign intelligence service (probably the 3rd partner involved).

Operation Eikonal

But there's another codeword connection: from 2004 till 2008, NSA cooperated with BND in operation EIKONAL in order to get access to fiber optic cables from Deutsche Telekom in Frankfurt.

From the parliamentary hearings we know that operation EIKONAL had GRANAT as its internal BND codename. And with GRANAT being German for garnet, and SMARAGD for emerald, we see that both operations are actually named after a gemstone, which often indicates some kind of similarity.

In October 2014, the Danish paper Information reported that the WHARPDRIVE access was opened in February 2013 and had the same size as EIKANOL. This operation EIKANOL or EIKONAL was a typical example of the way NSA cooperates with 3rd Party partner agencies under its RAMPART-A program, but unlike the SMARAGD/WHARPDRIVE operations with the cable access point being inside Germany:


 
Left: bilateral cable access operation (RAMPART-A) - Right: trilateral cable access operation
In the cases discussed here, Germany would be "Country X"
(click to enlarge)


It is tempting to identify SMARAGD and ZABBO as the two collection programs (SIGADs US-987LA and US-987LB) from the BOUNDLESSINFORMANT chart for Germany that was published in July 2013. For both facilities together, more than 552 million metadata records were counted between December 10, 2012 and January 8, 2013.

Provided that this chart shows the only data shared by BND, it's very well possible that the satellite collection program ZABBO is one of them. For the cable access SMARAGD this is less certain and depends on when this program started and whether it is identical with WHARPDRIVE (which started in February 2013).



BOUNDLESSINFORMANT screenshot showing metadata provided by BND
(click to enlarge)

Data transfer

The report of the data protection commissioner also provides an impression of the BND networks through which collected data are brought back to headquarters.

Data collected abroad are send back to Germany over the operational network ISNoVPN (apparently something that goes "over VPN" for secure tunneling) and then arrives at a dedicated demilitarized zone (DMZ) network for data collection (Datenabholungs-DMZ).

In this DMZ network there's a virtual machine (VM) that acts as a host for data that come in from each collection facility (Erfassungsansatz). The report mentions the virtual machines "Import VM SMARAGD" and "Import VM ZABBO" for the operations SMARAGD and ZABBO respectively.

In these virtual machines, the metadata go through an Application Level Gateway (ALG), which is a security components combined with a firewall. Such an ALG is able to detect, filter and when necessary, delete data from an incoming data stream. Again, there's an ALG for each collection facility: for example SMARAGD-ALG for data from the SMARAGD collection effort.

Finally, the collected data arrive at a network called NG-Netz, which is the back-end in Bad Aibling of the transfer system that pulls in data collected at a front-end access point (Erfassungskopf) somewhere abroad.



(click to enlarge)


Metadata analysis: VERAS

The system that BND uses for analysing bulk metadata from circuit-switched communications is called VERAS, which stands for Verkehrs-Analyse-System or Traffic Analysis System. VERAS stores metadata only for up to 90 days and according to the commissioner's report they are derived from two sources:

- Metadata that come with communications collected after matching with specific selectors (the related content goes to the INBE database)

- All the metadata from selected communication links (satellite frequencies and fiber optic channels) that are regarded useful for intelligence purposes, but only after passing the DAFIS filter.

According to the manual for VERAS version 4.3.x from 2010, the system has a topology mode, in which connections can be created level after level, similar to the "hops" we know from the NSA's contact chaining method. There's no limitation to the number of levels that can be added and analysts can also focus on specific targets to create patterns-of-life (Bewegungsprofile) for them.

This kind of contact-chaining and metadata analysis inevitably involves metadata from innocent people. BND distinguished between directly and indirectly relevant. Directly relevant are metadata related to people who are already known or suspected for being relevant for intelligence purposes.

Indirectly relevant are metadata related to people who have some kind of connection to directly relevant people, or when such metadata are being stored from a "geographical point of view", which apparently refers to metadata of people being somewhere near a target without having been in direct contact.

The report says that metadata connected on such a geographical basis results in much more people being involved than when using call or connection chaining. Data related to indirectly relevant people are also used by BND, for example as new selectors.

VERAS was introduced in 2002 and recently, VERAS 4 has been replaced by VERAS version 6, which was developed by the German armed forces (Bundeswehr) as part of the VERBA (VERkehrs-Beziehungs-Analyse) project.

For VERAS 6 there's not yet a database establishing order (see below), but in February 2015 BND sent the commissioner a draft version, which she already considers illegal because BND admits that it is technically impossible to prevent that data of innocent people are being used in the VERAS system.



Analysis and collection: XKEYSCORE

Already in July 2013, Der Spiegel reported that BND president Schindler had informed the parliamentary intelligence oversight commission (PKGr) that his agency was using NSA's XKEYSCORE system since 2007, but only for analysis, not for data collection. This was confirmed by W. K., a sub-division manager in the BND's Signals Intelligence division, during a parliamentary hearing.

But now, the report of the data protection commissioner says that BND uses XKEYSCORE not just for analysis, but also for the collection of both metadata and content.

The report explains that in its data collection, or front-end function, XKEYSCORE uses selectors, single ones or freely combined ones in the form of fingerprints, to search for matches in IP traffic of both public and privat networks, and stores anything that matches these selectors.

Remarkably enough, the commissioner writes that XKEYSCORE searches all internet traffic worldwide ("weltweit den gesamten Internetverkehr"), which seems to be a copy/paste from sensationalistic press reports, as XKEYSCORE can only search data which are collected at some physical access points and not even NSA has access to all the world's communications traffic, let alone BND.



Slide from an NSA presentation about the XKEYSCORE system


Besides picking out and storing communications that match specific selectors, XKEYSCORE is also able to store a so-called "full take", a temporary rolling buffer of all data from a particular link. This in order to find files which aren't directly associated with specific selectors - which was heralded as its unique capability.

The commissioner's report only mentions this buffer function when it cites a BND response calling XKEYSCORE "a local and temporary buffering of data" which in their opinion doesn't make it a database. The commissioner disagrees and says it's a database, because even when it's just for a short time, the data are available for usage. This means a there should have been a database establishing order for XKEYSCORE (see below).

Front-end and back-end

The report doesn't explain what XKEYSCORE actually does in its function as a back-end analysis tool. But maybe instead of distinguishing between collection and analysis, we should look at the difference between the front-end and the back-end functions of the system, which is explained in a manual for its so-called Deepdive version.

This learns us that the back-end performs high-speed filtering and selection using both strong selectors (like e-mail addresses) and soft selectors (like key words), and also uses various plug-ins to extract and index the metadata, which are also used for the rolling buffer-functionality of XKEYSCORE:



Diagram showing the dataflow for the DeepDive version of XKEYSCORE


The front-end is where the intercepted data streams come in, which are first reassembled by the METTLESOME and xFip components. Then, only the most useful streams are forwarded based upon rules using country codes, keywords and such. Finally, the Defrag component conducts full sessionizing, which means that the separate IP packets that travel over the internet are reassembled into their original readable form again.

The commissioner's report says that initially the sessionizing of data from a particular communications link was conducted by another NSA system codenamed WEALTHYCLUSTER (WC, which is for lower data rates), but that this kind of processing was more and more taken over by XKEYSCORE (XKS).

So, if the distinction between collection and analysing corresponds to that between front-end and back-end, that means that the new thing we learned from the commissioner's report is that BND apparently also uses XKEYSCORE for sessionizing the data they collect, and not only for filtering and analysing them.

This sessionizing might seem rather obvious, but real-time filtering and sessionizing at data rates as high as 10 Mbit/s requires very fast, specialized and expensive equipment. Well-known manufacturers are Narus and Verint, and it seems likely that their equipment is used for XKEYSCORE too.

As XKEYSCORE is only used for internet communications, the NSA selectors are derived from the SCRABBLE database. The results of the collection are transferred to NSA, after having been filtered by DAFIS to get rid of data related to Germans.



Integrated analysis: MIRA 4

Besides all the systems mentioned before, BND also uses MIRA 4, which stands for Modulare Integrierte Ressourcen Architektur or Modular Integrated Ressource Architecture, version 4. According to a letter from BND from February 2015, this system is used to store all the content, whether from e-mail, voice, fax or teletype messages, within a certain BND station and apparently also enables software to process and select raw data in order to create intelligence reports (Meldungen).

This was however contradicted by a letter from BND from December 2015 which said that MIRA 4 is only used to store just those Meldungen. The commissioner replied that she would be thankful when BND could clarify this discrepancy.

Apparently not noticed by the commissioner is an NSA report from 2006, which was published by earlier Der Spiegel, and which says that German analytic tool suites like MIRA 4:
"integrate multiple database analytic functions (such as viewing voice and listening to fax [sic]), much like NSA Headquarters has UIS (User Integrated Services). In some ways, these tools have features that surpass US SIGINT capabilities. Among a series of interesting items, NSA analysts noted that BND analysts could seamlessly move from VERAS (call-chaining software) to the associated voice cuts."

Later on, the 2006 NSA report says: "The BND responded positively to NSA's request for a copy of MIRA4 and VERAS software, and made several requests from NSA concerning target and tool development and data".

During a parliamentary hearing in October 2014, BND's own data protection officer Ms. H. F. said that in 2010, MIRA 4 was replaced by INBE as a system that apparently not only stores the content of communications, but also makes it available for analysis.

The 2016 commissioner's report says that data stored in MIRA 4 were not migrated to INBE, when the latter system became operational in 2011. Data in MIRA 4 seem to have been automatically "aged off" after 90 days and the last backup of the system was destroyed in the Summer of 2014.



Legal defects

The purpose of the secret report by federal data protection commissioner Andrea Voßhoff was to determine the legality of the data collection, processing, storing and analysing systems at the BND field station in Bad Aibling. The two main problems she identified are about necessity and the lack of database establishing orders.

Necessity

According to the German data protection law, BND is only allowed to receive, store, process and analyse personal data after checking that they are necessary and relevant for its foreign intelligence mission as authorized by German law. In various cases, especially when it comes to bulk collection of metadata and receiving the selectors from NSA, the agency doesn't or cannot check the necessity for each piece of data. This makes it unlawful for BND to posess and use those data.

The problem behind this is that when such laws were made, there was no awareness of secret services using large sets of metadata, which also includes those of innocent people. Also in this particular case, almost all data collected in Bad Aibling and shared with NSA will be collected from crisis zones like Afghanistan, which makes them more relevant for BND's mission and less likely of containing German communications.

Database establishing orders

Another major legal defect the commissioner found was that for the BND databases VERAS 4, VERAS 6, XKEYSCORE, TND, SCRABBLE, INBE, and DAFIS there was no database establishing order (Dateianordnung) and that they were also set up without prior approval by the commissioner. This makes the existance of these databases unlawful, which means the data they contain should be deleted immediatly until an establishing order is provided.

BND argued that the absence of a database establishing order is just a formal defect and doesn't affect the legal status of a database and its content. The commissioner doesn't agree with that and says that one of the functions of an establishing order is to determine the purpose of a database, which limits and restricts the use of the personal data in it. The lack of such an order also means that there are no rules for when approvals by oversight bodies are required, thus making the use of these databases both unlawful and uncontrolled.

In response

Meanwhile, on September 7, the German interior ministry released a draft for a new data protection act, in which it is proposed that in the future, the data protection commissioner will not have the authority anymore to impose sanctions or fines on the secret services - so restricting the commissioner's authority rather than strenghten it.

Finally, on September 15, Edward Snowden also mentioned the commissioner's report on Twitter, saying that it "confirms mass surveillance". Apparently he didn't read the report, as it is actually about the lack of specific legal restrictions, not about the scope of BND's collection efforts.




Links and Sources
- Rolf Weber: Der geleakte BND-Bericht der BfDI Voßhoff -- wie gewohnt bei näherem Hinsehen wenig skandalträchtig
- Netzpolitik: Secret Report: German Federal Intelligence Service BND Violates Laws And Constitution By The Dozen
- Der Spiegel: NSA-Standorte in Deutschland: Wiesbaden
- Wikipedia: Operation Eikonal

August 20, 2016

Is the Shadow Brokers leak the latest in a series?

(Latest UPDATE: November 20, 2016)

Earlier this week, a group or an individual called the Shadow Brokers published a large set of files containing the computer code for hacking tools. They were said to be from the Equation Group, which is considered part of the NSA's hacking division TAO.

The leak got quite some media attention, but so far it was not related to some earlier leaks of highly sensitive NSA documents. These show interesting similarities with the Shadow Brokers files, which were also not attributed to Edward Snowden, but seem to come from an unknown second source.



Screenshot of some computer code with instructions
from the Shadow Brokers archive from August 2016
(click to enlarge)


The Shadow Brokers files

Since August 13, Shadow Brokers posted a manifesto and two large encrypted files on Pastebin, on GitHub, on Tumblr and on DropBox (all of them closed or deleted meanwhile).

One of the encrypted files could be decrypted into a 301 MB archive containing a large number of computer codes for server side utility scripts and exploits for a variety of targets like firewalls from Cisco, Juniper, Fortinet and TOPSEC. The files also include different versions of several implants and instructions on how to use them, so they're not just the malware that could have been found on the internet, but also files that were only used internally.

A full and detailed list of the exploits in this archive can be found here.

Security experts as well as former NSA employees considered the files to be authentic, and earlier today the website The Intercept came with some unpublished Snowden documents that confirm the Shadow Brokers files are real.

Besides the accessible archive, Shadow Brokers also posted a file that is still encrypted, and for which the key would only be provided to the highest bidder in an auction. Would the auction raise 1 million bitcoins (more than 500 million US dollars), then Shadow Brokers said they would release more files to the public. This auction however is likely just meant to attract attention.

Updates:

Shadow Brokers, or people posing like them, posted an short announcement on Pastebin on August 28, and a third, long message including a "self-interview" on Medium.com on October 1. On October 15, a fourth message was published on Medium, saying that the auction was cancelled.

On October 31, Shadow Brokers came with a "Halloween message" on Medium, this time including a new file, which contains "configuration data for an as-yet-undisclosed toolkit for a variety of UNIX platforms" and also a list of 352 IP addresses and 306 domain names the NSA's hacking team Equation Group may have used for their operations. These addresses include timestamps from August 22, 2000, to August 18, 2010. The 10 most impacted countries are China, Japan, Korea, Spain, Germany, India, Taiwan, Mexico, Italy and Russia.




Screenshot of a file tree from the Shadow Brokers archive from August 2016
(click to enlarge)


From the Snowden documents?

According to security experts Bruce Schneier and Nicholas Weaver the new files aren't from the Snowden trove. Like most people, they apparently assume that Snowden took mostly powerpoint presentations and internal reports and newsletters, but that's not the whole picture. The Snowden documents also include various kinds of operational data, but this rarely became public.

Most notable was a large set of raw communications content collected by NSA under FISA and FAA authority, which also included incidentally collected data from Americans, as was reported by The Washington Post on July 5, 2014. The Snowden documents also include technical reports, which are often very difficult to understand and rarely provide a newsworthy story on their own.

Someone reminded me as well that in January 2015, the German magazine Der Spiegel published the full computer code of a keylogger implant codenamed QWERTY, which was a component of the NSA's WARRIORPRIDE malware framework. So with the Snowden trove containing this one piece of computer code, there's no reason why it should not contain more.

Contradicting the option that the Shadow Brokers files could come from Snowden is the fact that some of the files have timestamps as late as October 18, 2013, which is five months after Snowden left NSA. Timestamps are easy to modify, but if they are authentic, then these files have to be from another source.


A second source?

This brings us to a number of leaks that occured in recent years and which were also not attributed to Snowden. These leaks involved highly sensitive NSA files and were often more embarrassing than stuff from the Snowden documents - for example the catalog of hacking tools and techniques, the fact that chancellor Merkel was targeted and intelligence reports proving that NSA was actually successful at that.


It is assumed that these and some other documents came from at least one other leaker, a "second source" besides Snowden, which is something that still not many people are aware of. The files that can be attributed to this second source have some interesting similarities with the Shadow Brokers leak. Like the ANT catalog published in December 2013, they are about hacking tools and like the XKEYSCORE rules published in 2014 and 2015 they are internal NSA computer code.

This alone doesn't say much, but it's the choice of the kind of files that makes these leaks look very similar: no fancy presentations, but plain technical data sets that make it possible to identify specific operations and individual targets - the kind of documents many people are most eager to see, but which were rarely provided through the Snowden reporting.

As mainstream media became more cautious in publishing such files, it is possible that someone who also had access to the Snowden cache went rogue and started leaking documents just for harming NSA and the US - without attributing these leaks to Snowden because he would probably not approve them, and also to suggest that more people followed Snowden's example.

Of course the Shadow Brokers leak can still be unrelated to the earlier ones. In that case it could have been that an NSA hacker mistakenly uploaded his whole toolkit to a server outside the NSA's secure networks (also called a "staging server" or "redirector" to mask his true location) and that someone was able to grab the files from there - an option favored by for example Edward Snowden and security researcher the grugq.



Diagram showing the various stages and networks involved
in botnet hacking operations by NSA's TAO division
(source - click to enlarge)


An insider?

Meanwhile, several former NSA employees have said that the current Shadow Brokers leak might not be the result of a hack from the outside, but that it's more likely that the files come from an insider, who stole them like Snowden did earlier.

Of course it's easier for an insider to grab these files than for a foreign intelligence agency, let alone an ordinary hacker, to steal them from the outside. But if that's the case, it would mean that this insider would still be able to exfiltrate files from NSA premises (something that shouldn't be possible anymore after Snowden), and that this insider has the intent to embarrass and harm the NSA (Snowden at least said he just wanted to expose serious wrongdoings).

Here we should keep in mind that such an insider is not necessarily just a frustrated individual, but can also be a mole from a hostile foreign intelligence agency.

Update:
On August 21, NSA expert James Bamford also confirmed that TAO's ANT catalog wasn't included in the Snowden documents (Snowden didn't want to talk about it publicly though). Bamford favors the option of a second insider, who may have leaked the documents through Jacob Appelbaum and Julian Assange.


Russian intelligence?

On Twitter, Edward Snowden said that "Circumstantial evidence and conventional wisdom indicates Russian responsibility", but it's not clear what that evidence should be. It seems he sees this leak as a kind of warning from the Russians not to take revenge for the hack of the Democratic National Committee (DNC) e-mails, which was attributed to Russian intelligence.

This was also what led Bruce Schneier to think it might be the Russians, because who other than a state actor would steal so much data and wait three years before publishing? Not mentioned by Schneier is that this also applies to the documents that can be attributed to the second source: they also pre-date June 2013.

A related point of speculation is the text that accompanied the Shadow Brokers files, which is in bad English, as if it was written by a Russian or some other non-western individual. This is probably distraction, as it looks much more like a fluent American/English speaker who tried to imitate unexperienced English.

The text also holds accusations against "Elites", in a style which very much resembles the language used by anarchist hacker groups, but that can also be faked to distract from the real source (it was also noticed that the e-mail address used by Shadow Brokers (userll6gcwaknz@tutanota.com) seems to refer to the manga Code Geass in which an exiled prince takes revenge against the "Britannian Empire").



Screenshot of some file folders from the Shadow Brokers archive
(click to enlarge)


Conclusion

With the authenticity of the Shadow Brokers files being confirmed, the biggest question is: who leaked them? There's a small chance that it was a stupid accident in which an NSA hacker uploaded his whole toolkit to a non-secure server and someone (Russians?) found it there.

Somewhat more likely seems the option that they came from an insider, and in that case, this leak doesn't stand alone, but fits into a series of leaks in which, since October 2013, highly sensitive NSA data sets were published.

So almost unnoticed by the mainstream media and the general public, someone was piggybacking on the Snowden-revelations with leaks that were often more embarrassing for NSA than many reportings based upon the documents from Snowden.

Again, obtaining such documents through hacking into highly secured NSA servers seems less likely than the chance that someone from inside the agency took them. If that person was Edward Snowden, then probably someone with access to his documents could have started his own crusade against NSA.

If that person wasn't Snowden, then it's either another NSA employee who was disgruntled and frustrated, or a mole for a hostile foreign intelligence agency. But for an individual without the protection of the public opinion like Snowden, it must be much harder and riskier to conduct these leaks than for a foreign state actor.

Former NSA counterintelligence officer John Schindler also thinks there could have been a (Russian) mole, as the agency has a rather bad track record in finding such spies. If this scenario is true, then it would be almost an even bigger scandal than that of the Snowden-leaks.

Update #1:
During an FBI-led investigation of the ShadowBrokers leak, NSA officials reportedly said that a former agency operative carelessly left the hacking tool files available on a remote computer, where Russian hackers found them. After this was discovered, NSA tuned its sensors to detect use of any of the tools by other parties, like China and Russia. But as that wasn't the case, NSA did not feel obligated to warn the US manufacturers.

Update #2:
On October 6, 2016, The New York Times reported that on August 27, 2016, the FBI arrested 51-year old Harold T. Martin III, who worked at NSA as a contractor for Booz Allen Hamilton. In his home in Glen Burnie, Maryland, "many terabytes" of highly classified information was found, from the 1990s until 2014. Hal Martin was described as a hoarder, but so far, investigators are not sure he was also responsible for the various leaks that could not be attributed to Snowden.

Update #3:
On November 19, it was reported by the Washington Post that there had been yet another, previously undisclosed breach of cybertools, which was discovered in the summer of 2015. This was also carried out by a TAO employee, who had also been arrested, but his case was not made public. An official said that it is not believed that this individual shared the material with another country.



Links and Sources
- TheWeek.com: How the NSA got hacked
- EmptyWheel.com: Where Are NSA’s Overseers on the Shadow Brokers Release?
- Observer.com: NSA ‘Shadow Brokers’ Hack Shows SpyWar With Kremlin Is Turning Hot
- TechCrunch.com: Everything you need to know about the NSA hack (but were afraid to Google)
- WashingtonPost.com: Powerful NSA hacking tools have been revealed online
- NYTimes.com: ‘Shadow Brokers’ Leak Raises Alarming Question: Was the N.S.A. Hacked?
- LawfareBlog.com: NSA and the No Good, Very Bad Monday