How the CIA Partnered With Amazon and Changed Intelligence
How the CIA Partnered With Amazon and Changed
Intelligence
By Frank Konkel July 11, 2014
The intelligence community is about to get the equivalent
of an adrenaline shot to the chest. This summer, a $600 million computing cloud
developed by Amazon Web Services for the Central Intelligence Agency over the
past year will begin servicing all 17 agencies that make up the intelligence
community. If the technology plays out as officials envision, it will usher in
a new era of cooperation and coordination, allowing agencies to share
information and services much more easily and avoid the kind of intelligence
gaps that preceded the Sept. 11, 2001, terrorist attacks.
For the first time, agencies within the IC will be able
to order a variety of on-demand computing and analytic services from the CIA
and National Security Agency. What’s more, they’ll only pay for what they use.
The vision was first outlined in the IC Information
Technology Enterprise plan championed by Director of National Intelligence
James Clapper and IC Chief Information Officer Al Tarasiuk almost three years
ago. Cloud computing is one of the core components of the strategy to help the
IC discover, access and share critical information in an era of seemingly
infinite data.
For the risk-averse intelligence community, the decision
to go with a commercial cloud vendor is a radical departure from business as
usual.
In 2011, while private companies were consolidating data
centers in favor of the cloud and some civilian agencies began flirting with
cloud variants like email as a service, a sometimes contentious debate among
the intelligence community’s leadership took place.
As one former intelligence official with knowledge of the
Amazon deal told Government Executive, “It took a lot of wrangling, but it was
easy to see the vision if you laid it all out.”
The critical question was would the IC, led by the CIA,
attempt to do cloud computing from within, or would it buy innovation?
Money was a factor, according to the intelligence
official, but not the leading one. The
government was spending more money on information technology within the IC than
ever before. IT spending reached $8 billion in 2013, according to budget
documents leaked by former NSA contractor Edward Snowden. The CIA and other
agencies feasibly could have spent billions of dollars standing up their own
cloud infrastructure without raising many eyebrows in Congress, but the
decision to purchase a single commercial solution came down primarily to two
factors.
“What we were really looking at was time to mission and
innovation,” the former intelligence official said. “The goal was, ‘Can we act
like a large enterprise in the corporate world and buy the thing that we don’t
have, can we catch up to the commercial cycle? Anybody can build a data center,
but could we purchase something more?
“We decided we
needed to buy innovation,” the former intelligence official said.
A Groundbreaking Deal
The CIA’s first request for proposals from industry in
mid-2012 was met with bid protests to the Government Accountability Office from
Microsoft and AT&T, two early contenders for the contract. Those protests
focused on the narrow specifications called for by the RFP. GAO did not issue a
decision in either protest because the CIA reworked its request to address the
companies’ complaint.
In early 2013, after weighing bids from Amazon Web
Services, IBM and an unnamed third vendor, the CIA awarded a contract to AWS
worth up to $600 million over a period of up to 10 years. The deal, handled in
secret, was first reported by FCW in March 2013, sending ripples through the
tech industry.
A month after the deal became public, IBM filed a bid
protest with GAO that the watchdog eventually upheld in June, forcing the CIA
to reopen bids to both companies for the contract. A legal struggle between
Amazon and Big Blue ensued, and AWS filed a lawsuit against the federal
government in July 2013, claiming the GAO sustainment was a “flawed” decision.
In October, U.S. Court of Federal Claims Judge Thomas
Wheeler sided with Amazon and overturned GAO’s decision to force the CIA to
rebid the contract. Big Blue went home, AWS claimed victory under the deal’s
original financial specs, and nearly 18 months after the procurement was first
released, the CIA and Amazon went to work.
It is difficult to underestimate the cloud contract’s
importance. In a recent public appearance, CIA Chief Information Officer
Douglas Wolfe called it “one of the most important technology procurements in recent
history,” with ramifications far outside the realm of technology.
“It’s going to take a few months to bring this online in
a robust way, but it’s coming,” Wolfe said.
“And I think it’s going to make a big difference for national security.”
Securing New Capabilities
The Amazon-built cloud will operate behind the IC’s
firewall, or more simply: It’s a public cloud built on private premises.
Intelligence agencies will be able to host applications
or order a variety of on-demand services like storage, computing and analytics.
True to the National Institute of Standards and Technology definition of cloud
computing, the IC cloud scales up or down to meet the need.
In that regard, customers will pay only for services they
actually use, which is expected to generate massive savings for the IC.
“We see this as a tremendous opportunity to sharpen our
focus and to be very efficient,” Wolfe told an audience at AWS’ annual
nonprofit and government symposium in Washington. “We hope to get speed and scale
out of the cloud, and a tremendous amount of efficiency in terms of folks
traditionally using IT now using it in a cost-recovery way.”
Many agencies within the IC already have identified
applications to move to the cloud. In a recent report, National Reconnaissance
Office Chief Information Officer Donna Hansen said her agency had picked five
applications, including its enterprise resource planning software, to migrate
to the IC cloud.
As with public clouds, the IC cloud will maximize
automation and require standardized information, which will be shared through
application programming interfaces, known as APIs. Amazon engineers will
oversee the hardware because AWS owns the hardware and is responsible for
maintaining it just as they do in the company’s public data centers.
Whenever Amazon introduces a new innovation or
improvement in cloud services, the IC cloud will evolve. Company officials say
AWS made more than 200 such incremental improvements last year, ensuring a sort
of built-in innovation to the IC cloud that will help the intelligence
community keep pace with commercial advances. Wolfe said AWS’ capacity to bring
commercial innovation from places like Silicon Valley to the IC is one of the
contract’s greatest benefits. Whenever AWS introduces new products, the CIA
will be able to implement them.
“The biggest thing we were trying to do—the visionary
folks a couple years ago—was answer the question, ‘How do we keep up?’” Wolfe
said. “The mission we have is important. The pace and complexity is really not
[diminishing], in fact, it may be increasing. We feel it is very important to
deliver the best IT and best products and services we can to our customers in
the IC.”
What of the data, though? Intelligence agencies are
drowning in it, collecting and analyzing an amalgamation of information from
sensors, satellites, surveillance efforts, open data repositories and human
intelligence, among other sources. Is that data really secure in the cloud?
The CIA is convinced it is.
The IC cloud “will be accredited and compliant with IC
standards,” says a senior CIA official familiar with the IC cloud. It will, for
example, be able to handle Sensitive Compartmented Information, a type of
classified information. “Security in the IC cloud will be as safe as or safer
than security on our current data centers,” the senior CIA official says.
Because the IC cloud will serve multiple tenants—the 17
agencies that comprise the IC—administrators will be able to restrict access to
information based on the identity of the individual seeking it. The idea is to
foster collaboration without compromising security. Visually, the IC cloud can
be thought of as a workspace hanging off the IC’s shared network—a place where
data can be loaded for a variety of tasks like computing or sharing. The IC
cloud gives agencies additional means to share information in an environment
where automated security isn’t a barrier to the sharing itself. This could
prove vital in situations reminiscent of 9/11, in which national security is an
immediate concern.
Cloud vendors, including Amazon, have argued that cloud
infrastructures can be more secure than traditional data centers because there
are fewer points of entry, but the leaks by Snowden illustrate the potential
threat from inside an organization. Snowden was able to access and download
classified information intelligence officials said he shouldn’t have been able
to access.
To access information within the IC cloud, analysts must
have the proper permissions. In addition, the standardized environment and
automation means all activity within the cloud is logged and can be analyzed in
near real-time.
Some government officials view cloud computing as
inherently less secure than computing on locally controlled servers, but the
CIA’s acceptance of commercially developed cloud technology “has been a wake-up
call” to those who balk at it, according to John Pirc, a former CIA cybersecurity
researcher who is now chief technology officer at NSS Labs, a security research
firm.
“You hear so many people on the fence about cloud, and
then to see the CIA gobble it up and do something so highly disruptive, it’s
kind of cool,” says Pirc. “To me, this removes the clouded judgment that cloud
isn’t secure. Their moving forward with this should send a message to the rest
of the industry that cloud is something you shouldn’t be afraid of.”
Pirc is no stranger to disruptive technologies. At the CIA’s
research labs in the early 2000s, he recalls virtualization—a technology that
allows multiple operating systems to run simultaneously on the same servers,
allowing for far more efficient computing—before it became an integral
component of many IT enterprises.
Intelligence agencies use commercial off-the-shelf technology all the
time, but to Pirc, the importance of the cloud capabilities the CIA gets
through leveraging Amazon Web Services’ horsepower is best exemplified in
computing intelligence data. Scalable computing is critical for fostering
shared services and enhanced collaboration between disparate intelligence
agencies.
“What it allows them to do is spin up servers and add
more [computing power] fast, and when you’re computing intelligence data, the
more compute power you have, the faster you can react,” Pirc says. “In the
private sector, compute is all about money and profit, but from my viewpoint
when I worked for the agency, you’re working with extremely time-sensitive
information. Being able to have that compute power, something that might have
taken a couple of hours might instead take a few seconds. Profits aren’t lost
when you make mistakes in the intelligence community—people die when you make
mistakes.”
A test scenario described by GAO in its June 2013 bid
protest opinion suggests the CIA sought to compare how the solutions presented
by IBM and Amazon Web Services could crunch massive data sets, commonly
referred to as big data.
Solutions had to provide a “hosting environment for applications
which process vast amounts of information in parallel on large clusters
(thousands of nodes) of commodity hardware” using a platform called MapReduce.
Through MapReduce, clusters were provisioned for computation and segmentation.
Test runs assumed clusters were large enough to process 100 terabytes of raw
input data. AWS’ solution received superior marks from CIA procurement
officials, according to GAO documentation, and was one of the chief reasons the
agency selected Amazon.
Limited Details
The CIA declined to comment when Government Executive
asked about the extent of the IC cloud’s capabilities or that of the National
Security Agency’s cloud. Amazon also declined to describe the IC cloud’s
technical capabilities.
It is a good bet, though, that the AWS-built cloud for
the IC will have capabilities at least equal to existing capabilities Amazon
has already implemented across government.
For example, the company provides the cloud bandwidth for
the Securities and Exchange Commission’s collection of more than 1 billion
trade records and more than a terabyte of new data per day through its Market
Information Data Analytics System. This example may be prescient given that
now-public surveillance efforts indicate the IC collects billions and perhaps
trillions of pieces of metadata, phone and Internet records, and other various
bits of information on an annual basis. The potential exists for the CIA to
become one of AWS’ largest customers.
Within the intelligence community, examples abound where the
cloud’s capabilities could significantly boost the mission.
As the geospatial hub of the community, the National
Geospatial-Intelligence Agency ingests, analyzes, metatags and reports all
geo-intelligence and multisource content in its flagship program called Map of
the World. Geospatial data’s importance to the IC has increased in recent
years, as evidenced by NGA’s nearly $5 billion budget and its staff nearly
doubling in size since 2004. For intensive applications like ingesting or
analyzing geospatial data, scalable computing could have a significant impact
on mission performance. The cloud also could improve the way the agency shares
its large data sets.
What the IC has done with cloud is not easily replicable,
according to American Council for Technology President Rick Holgate, but it is
worth paying attention to.
“The IC has a model other agencies should look to and
aspire to in terms of transforming the way they think about delivering services
across a large enterprise,” Holgate says. “They are looking to common platforms
and service delivery models across an entire enterprise, and not just gaining
cost efficiencies, but to provide foundational capabilities to really allow it
to operate.”
Whether or not the IC cloud serves as an example for the
rest of government, the CIA’s quest to buy innovation will loom large for years
to come.
Comments
Post a Comment