I maintain that there is an inverse correlation between the technical abilities of the government to harvest data and their competence to use it for anything. – DerLeader(aka king klusterfuck)
Well perhaps you don’t get it, .. while that MAY have been true 20+ years ago, but in the last 10 years BIG shit has happened, in english its called ‘data mining’ ( https://blogthetruthandrun.wordpress.com/data-mining-like-the-nsa-on-the-cheap/ ),
With this new math, and new fast computers, you can ask any rhetorical question and get instant response. Essentially JIM (DL) the HAL(2001 space) is here. Super Computer’s that do what humans can’t do, … take millions of facts and deduce facts in a nanosecond.
This link ( https://blogthetruthandrun.wordpress.com/data-mining-like-the-nsa-on-the-cheap/ ), talks about the algorithms that are being used. So at least if you want to learn you can use the right terms to find shit.
Take the BIG company that’s ISRAEL and behind PRISM, the company is called “PALANTIR” or something like that, they’re a MACHINE-LEARNING company,, this new technology sift’s through the NSA database ( data on all humans on earth ), and allows anybody with access to the system, to know anything, anytime, …
SO JIM your premise that all this data and they can’t use it is BOGUS, now I’m going to assume you made the statement cuz you don’t have a clue, given your not a mathematician or computer scientist.
But large progress has been made in the last 5 years in the area of ‘machine learning’, e.g. Teaching computers to SIMULATE the human mind, so we know have computers that do what people do, but humans can only deal with 5-13 facts at a time, the new algorithms support millions of facts, so now even though the NSA has a quadrillion pieces of data, they can sift that data for a correlation in a nanosecond.
Like the VERIZON argument about ‘metadata’, what’s the big deal, well that metadata can allow a user of the ‘PRISM’ system to say, … where now are all the GUN-LOVER’s, … or PEAK-OIL NERDS, … where are they now? Then on a MAP world wide it shows where they’re all at, … why? Cuz the NSA doesn’t care about people, it cares about hot-spots, the governments fear social movement of like minded mass.
Amid the torrent of stories about the shocking new revelations about the National Security Agency, few have bothered to ask a central question. Who’s actually doing the work of analyzing all the data, metadata and personal information pouring into the agency from Verizon and nine key Internet service providers for its ever-expanding surveillance of American citizens?
Well, on Sunday we got part of the answer: Booz Allen Hamilton. In a stunning development in the NSA saga, Guardian reporter Glenn Greenwald revealed that the source for his blockbuster stories on the NSA is Edward Snowden, “a 29-year-old former technical assistant for the CIA and current employee of the defense contractor Booz Allen Hamilton.” Snowden, it turns out, has been working at NSA for the last four years as a contract employee, including stints for Booz and the computer-services firm Dell.
The revelation is not that surprising. With about 70 percent of our national intelligence budgets being spent on the private sector – a discovery I made in 2007 and first reported in Salon – contractors have become essential to the spying and surveillance operations of the NSA.
From Narus, the Israeli-born Boeing subsidiary that makes NSA’s high-speed interception software, to CSC, the “systems integrator” that runs NSA’s internal IT system, defense and intelligence, contractors are making millions of dollars selling technology and services that help the world’s largest surveillance system spy on you. If the 70 percent figure is applied to the NSA’s estimated budget of $8 billion a year (the largest in the intelligence community), NSA contracting could reach as high as $6 billion every year.
But it’s probably much more than that.
“The largest concentration of cyber power on the planet is the intersection of the Baltimore Parkway and Maryland Route 32,” says Michael V. Hayden, who oversaw the privatization effort as NSA director from 1999 to 2005. He was referring not to the NSA itself but to the business park about a mile down the road from the giant black edifice that houses NSA’s headquarters in Fort Meade, Md. There, all of NSA’s major contractors, from Booz to SAIC to Northrop Grumman, carry out their surveillance and intelligence work for the agency.
With many of these contractors now focused on cyber-security, Hayden has even coined a new term — “Digital Blackwater” – for the industry. “I use that for the concept of the private sector in cyber,” he told a recent conference in Washington, in an odd reference to the notorious mercenary army. “I saw this in government and saw it a lot over the last four years. The private sector has really moved forward in terms of providing security,” he said. Hayden himself has cashed out too: He is now a principal with the Chertoff Group, the intelligence advisory company led by Michael Chertoff, the former secretary of Homeland Security.
One of NSA’s most important contractors may be Narus, a subsidiary of Boeing that makes a key telecommunications software that allows government agencies and corporations to monitor huge amounts of data flowing over fiber-optic cables. According to Bill Binney, one of four NSA whistle-blowers who’ve been warning about NSA’s immense powers, one Narus device can analyze 1,250,000 1,000-character emails every second. That comes to over 100 billion emails a day.
“Narus is the one thing that makes it all possible,” Binney told me over the weekend, of the Verizon surveillance program unveiled by the Guardian. “They probably pick up 60 to 80 percent of the data going over the [U.S.] network.” The Narus technology, he added, “reconstructs everything on the line and then passes it off to NSA for storage” and later analysis. That includes everything, he said, including email, cellphone calls, and voice over Internet protocol calls such as those made on Skype.
NSA’s use of the Narus technology first came to attention in 2006. That was when an AT&T technician named Mark Klein went public with his discovery that NSA had hooked Narus devices to AT&T’s incoming telecom stream in San Francisco and set up a secret room that allowed NSA to divert AT&T’s entire stream to its own databases. Binney believes the equipment was hooked up to as many as 15 sites around the country.
The Narus devices can’t pick up everything, however, because large amounts of traffic (such as domestic calls and Internet messages) don’t go through the switches. That’s why NSA apparently decided in 2006 to create the PRISM program to tap into the databases of the Internet service providers such as Yahoo and Google, Binney says. “Even though there’s so many Narus devices collecting on the Net, they don’t get it all,” he explained. “So if they go to the ISPs with a court order, they fill in the gaps from the collection on Narus.”
But once the data is downloaded, it has to be analyzed. And that’s where Booz and the other contractors that surround the NSA come in.
Booz Allen Hamilton is one of the NSA’s most important and trusted contractors. It’s involved in virtually every aspect of intelligence and surveillance, from advising top officials on how to integrate the 16 U.S. spy agencies to detailed analysis of signals intelligence, imagery and other critical collections technologies. I first introduced Booz’s intelligence business in a 2007 profile in Salon when President Bush appointed Michael McConnell, a Booz veteran and former NSA director, to be director of national intelligence (he’s now back at Booz).
Among other secret projects, Booz was deeply involved in “Total Information Awareness,” the controversial data-mining project run for the Bush administration by former National Security Adviser John Poindexter that was outlawed by Congress in 2003.
Another major presence at NSA’s Business Park is SAIC. Like Booz, it stands like a private colossus across the whole intelligence industry. Of its 42,000 employees, more than 20,000 hold U.S. government security clearances, making it one of the largest private intelligence services in the world. “SAIC provides a full suite of intelligence, surveillance and reconnaissance (ISR) and cybersecurity solutions across a broad spectrum of national security programs,” it claims on its website.
Despite its grandiose claims, however, SAIC is also known for several spectacular intelligence failures, including NSA’s ill-fated Trailblazer project to privatize its analysis of signals intelligence. Other companies acting as pillars of NSA’s SIGINT analysis team include Northrop Grumman, Raytheon, CACI International, and hundreds of smaller companies scattered around the Washington Beltway (you can read detailed explanations of what they do for NSA in my book “Spies for Hire”). They, in turn, are surrounded by a small army of “big data” companies that are hired by NSA to sift through data for suspicious patterns and map the creation of “illicit networks” that can be followed or investigated.
In April, I wrote about one of those companies, Palantir Technologies Inc., in Salon. It sells a powerful line of data-mining and analysis software that maps out human social networks that would be extremely useful to NSA analysts trying to make sense of all the telephone and Internet data downloaded from Verizon and nine Internet companies that was described in the latest blockbuster stories in the Guardian and the Post.
“Their bread and butter is mapping disparate networks in real time,” a former military intelligence officer who has used Palantir software told me. “It creates a spatial understanding that can be easily used by analysts.” (See the detailed profile of Palantir I posted on my website last Friday.)
But how did NSA, long considered the crown jewel of U.S. intelligence, become so privatized in the first place?
In the late 1990s, faced with a telecommunications and technological revolution that threatened to make the NSA’s telephonic and radar-based surveillance skills obsolete, the agency decided to turn to private corporations for many of its technical needs.
The outsourcing plan was finalized in 2000 by a special NSA Advisory Board set up to determine the agency’s future and codified in a secret report written by a then-obscure intelligence officer named James Clapper. “Clapper did a one-man study for the NSA Advisory Board,” recalls Ed Loomis, a 40-year NSA veteran who, along with Binney and two others, blew the whistle on corporate corruption at the NSA.
“His recommendation was that NSA acquire its Internet capabilities from the private sector. The idea was, the private sector had the capability and we at NSA didn’t need to reinvent the wheel.”
Hayden, who was the NSA director at the time, “put a lot of trust in the private sector, and a lot of trust in Clapper, because Clapper was his mentor,” added Loomis. And once he got approval, “he was hell-bent on privatization and nothing was going to derail that.” Clapper is now President Obama’s director of national intelligence, and has denounced the Guardian leaks as “reprehensible.”
Hayden was relentless in shifting NSA from an agency that relied on in-house experts for its technology to one of the most privatized agencies in government today. His first action, a project known as Groundbreaker, outsourced all of NSA’s internal communications system. In one fell swoop, hundreds of longtime NSA employees left their government jobs one day and walked in the next morning wearing their green badges from CSC and its many subcontractors.
“To this day, the IT at Fort Meade is owned by a private sector company,” Hayden boasted recently. “That worked. That was a really good idea.” CSC remains the head of the “Eagle Alliance” consortium, and is now one of NSA’s biggest suppliers of cybersecurity services.
But Hayden’s master project, the grandiose Trailblazer project to private NSA’s analysis of signals intelligence flowing over the Internet, didn’t fare so well. Managed by SAIC in a consortium that included Northrop Grumman and Booz Allen Hamilton, it burned through over $5 billion without producing any actionable intelligence, and was canceled in 2005.
Despite the scandals and massive amount of money spent on private intelligence contractors, however, the mainstream media has been slow to report on the topic. It took until 2010, years after the spending spree began, for the Washington Post to highlight intelligence outsourcing in its famous series on “Top Secret America.” The paper, despite its work on the PRISM story, is still behind the curve
**** FIGHT BACK … Here’s HOW
BIG BROTHER really is watching you. A series of revelations over the past week has revealed the extent of the US government’s snooping. But there are ways that the average citizen can avoid the prying eyes of the state.
Last week, whistleblower Edward Snowden – a former contractor with the National Security Agency (NSA) – told UK newspaper The Guardian that the NSA not only has details of phone calls made by millions of Verizon customers, it also has some form of access to its citizens’ internet activity as part of a programme named Prism.
The details of exactly how the NSA accesses personal data held by US internet companies are still unclear (see “Split the difference“). Access to Verizon’s call metadata was obtained using a secret court order forcing the firm to hand over information including call duration, number and cellphone tower details.
But the main cause for concern is that network science today means governments can glean remarkable insights from the vast amount of data they compile about their citizens’ every move. So what exactly can they find out about us – and how can we opt out?
Tanya Berger-Wolf at the University of Illinois in Chicago, who studies methods of extracting information from large data sets, says combining data from sources such as Google, Facebook and Verizon can tell you a lot. “You can put together a very good, composite dossier of a person,” she says.
Phone calls alone can provide plentiful information. Yves-Alexandre de Montjoye of the Massachusetts Institute of Technology and colleagues analysed 1.5 million anonymised call records from a Western cell carrier. They showed that it takes just four calls or text messages, each made at a different time and place, to distinguish one person’s movements from everyone else’s (Nature Scientific Reports, doi.org/msd).
An experiment by German politician Malte Spitz shows what happens when you fuse such data with online activity. Spitz sued German telecoms giant Deutsche Telekom to get it to hand over six months of his own phone data. Then, working with German newspaper Die Zeit, Spitz melded that data with social network and other web information about him to create a map that tracked his movements and activities. It showed where Spitz was at any given time, what he was doing, how many calls he made and how long he was connected to the internet. The NSA’s supercomputers would make light work of creating an even more detailed portrait of anyone it was interested in.
Chris Clifton, who works on data privacy at Purdue University in Indiana, says the NSA will be using software to sort the records into groups by similarity – people who make lots of calls, for example, or people who never call abroad. Patterns in time could be useful, too. If one call appears to spark a flurry of others, that might mean the first phone number belongs to an authority figure in a criminal organisation, for instance.
But for citizens who want to guard their privacy, there are a number of options. Apps like Silent Circle and RedPhone can already encrypt your calls and send them over a data connection or Wi-Fi instead of through your carrier’s voice network. They also stop carriers from logging end phone numbers. Downloads have exploded since The Guardian‘s revelations – but such apps do not give you full anonymity because they cannot prevent your movements between phone masts being tracked.
A new standard for communication known as WebRTC (Web Real-Time Communication) could enable users to make calls over the internet without leaving any traces at all. That’s because it doesn’t rely on centralised servers but rather sends traffic directly between individual computers.
Combined with an encrypted connection using the anonymising Tor network, which sends data via volunteer networks of computers, WebRTC could keep your internet communications invisible to prying eyes.
An organisation called Tor Servers is aiming to bolster traffic speeds across Tor exit nodes – the points at which traffic from Tor enters the real internet. Its mission statement is to “make the Tor network more stable, faster and more anonymous for everyone”.
There are even efforts afoot to build an entirely new internet, one free from control by large corporations and, by extension, governments. Project Meshnet aims to have its own router hardware, and for this to communicate without using the infrastructure of large telecoms companies. That is still some way off, but for now you can use the software version, called cjdns, which runs on existing infrastructure. Physical Meshnets are already up and running in Maryland, Seattle and New York.
And there are ways to protect people’s privacy while still obtaining information. An MIT project called openPDS works by only allowing third parties to ask questions of a data set, without allowing them to get their hands on the raw data. This, combined with legal systems that notify individuals when their data has been searched, could change the privacy debate. “Such a ‘mixed approach’ to privacy is the way forward,” de Montjoye says.