OpenStack Security

OpenStack Security Notes, and how they help you the Operator

Fri, 08 Sep 2017 00:00:00 +0000

For this post I will explain what OpenStack Security notes are, and how they benefit operators in securing an OpenStack Cloud.

OpenStack Security Notes (OSSN’s) are solely to notify operators of a discovered risk, that are often not directly addressed by a code patch.

OSSN’s can be in the form of a deployment architecture recommendation, configuration value or a file permission.

Consider the meme ‘If you do this, you’re going to have a bad time’ to get an idea of what OSSN’s are about.

Some examples of recent OSSN’s would be:

The end to end process of an OSSN, starts when a member of the security project, a project core, or a VMT member, mark a launchpad bug by adding the ‘OpenStack Security Note’ group. An author will then assign themselves to the bug, and will commit to authoring the OSSN. Public notes may be worked on by anyone, whereas embargoed notes are only handled by the security project core members.

Once the author has a draft in place, they will submit a patch to the security-docs repo, where other members of the security project and cores from the related project of the original launchpad bug, can review the note content.

After the patch has received two +2 reviews from security project core members, and a +1 from a core within the concerned project, the OSSN is merged into the security-docs repository.

Once merged, the reviewed text will be posted to the OpenStack Wiki , and a GPG signed email will be sent to the openstack & openstack-dev mailing lists.

The OpenStack Security Project welcomes anyone who wants to help Author or review OSSN’s. Security Notes are often a path to the election of core members of the OpenStack security project. OSSN authorship was how I personally found myself elected almost two years back.

Anyone new to the security project offering to help author a Security Note, will be given lots of support on creating their first OSSN from other Security Project members.

We also welcome feedback from operators on how valuable you find OSSN’s, and ways you feel may improve the process. After all, the process is there to benefit you the operator.

For anyone with an interest in OpenStack Security, the OpenStack Security Project can be found on the irc-channel #openstack-security and we meet weekly on #openstack-meeting-alt every Thursday @ 17:00 UTC time.

You can also email the security project on the OpenStack developer mailing list, by using a [security] tag in the subject line.

Luke Hinds (Security Project PTL)

Syntribos team recap for Newton

Wed, 26 Oct 2016 00:00:00 +0000

Our team set out to accomplish several tasks during the Newton cycle:

Improve Syntribos to the point that it was reliable and useful for testing the 6 projects OSIC has chosen to focus on for Newton (keystone, neutron, glance, nova, cinder, and swift)
Test those 6 key projects and report our results to upstream developers
Based on our results, determine future action items to further improve Syntribos

We succeeded in making Syntribos more configurable, easier to use, and less prone to false positives. We released several new features, removed cruft in the codebase, added function/class comments, and wrote unit tests, all with the goal of making Syntribos a more effective tool for testers, and easier to contribute to as a developer. We were able to test all 6 key projects, and reported several bugs in their Launchpads.

We started off the cycle by making improvements aimed at easing further development. This included cleaning up the codebase, creating documentation with sphinx, fixing bugs, writing unit tests, and adding docstrings, among other changes. We also removed our dependency on OpenCAFE at the request of some in the community, which took several weeks. This leaves us with a pretty small dependency base, which should make future maintenance / modification more manageable. Once we were more confident in the core codebase, we started focusing on how to improve the accuracy and depth of tests conducted by Syntribos. We used a special vulnerable API created by Matt Valdes from Rackspace to validate our improvements, and ensure that our tests were detecting the issues we introduced into the API.

We spent a significant amount of time creating templates and extensions for each project. This typically took at least 1 or 2 days per project, significantly reducing the amount of time we spent testing. However, much of the heavy legwork for testing these projects is now out of the way, and future testing should be significantly easier. Our team is also more experienced with the OpenStack projects we tested, and with security testing in general in some cases.

Overall, we believe we were successful in meeting our goals for this cycle, though we believe that more work is required to get Syntribos ready for production use by others in the community.

Our Key Accomplishments

Worked to improve Syntribos tool from April 5th through September 30th
Participated in the OpenStack Security Project’s midcycle, where we wrote several OSSNs, contributed to the barbican threat analysis process, and discussed Syntribos with others in the community
Tested 6 OSIC key projects from August 29th through September 30th
Found 4 defects during our testing, and submitted them in projects’ Launchpads

Metrics

Syntribos

Bugs reported in Launchpad: 17 [3]
Bugs resolved: 15 [3]
Unit test coverage at start: 9%
Unit test coverage at end: 63%

OSIC Key Projects

Request templates created: 611 [4]
Bugs reported in Launchpad: 4 (see Reported Bugs below)

Reported Bugs

String “..%c0%af” causes 500 errors in multiple locations
- Affects: keystone, cinder, neutron, glance
- Launchpad: https://bugs.launchpad.net/keystone/+bug/1613901
[Duplicate] Stored XSS in glance image names
- Affects: horizon
- Launchpad: https://bugs.launchpad.net/horizon/+bug/1623735
Authenticated “billion laughs” memory exhaustion / DoS in ovf_process.py
- Affects: glance
- Launchpad: https://bugs.launchpad.net/glance/+bug/1625402
One embargoed issue that is still being triaged

Challenges

Removing OpenCAFE took several weeks, and while it removed a large dependency, it cut down on time for other improvements.
Our short testing schedule (1 month of testing for 6 projects) didn’t give us much time to learn the intricacies of each project, and test them at a deeper, domain-specific level. Some projects offered significantly more endpoints than others, and in some cases we had to move on before fully evaluating every component. However, we were able to at least do some basic testing on every offered endpoint for every tested service.
- The significant time investment required to create templates for each project further limited the time we had to test these projects.
Lack of unit tests meant that many changes introduced bugs/crashes into master and required fix-ups. This happened less often as our coverage improved.
Documentation was lacking or inaccurate at the outset, and required significant effort to improve.
Our team’s relative inexperience with the OpenStack projects under test, and security testing in general in some cases, made testing more challenging.

Syntribos Improvements / Future Plans

As we performed our one-month engagement testing various OpenStack services, several members of the team took notes about features, bugs, and general improvements to be made in Syntribos.

Planned Changes for Ocata

Cutting a stable Syntribos release on PyPI to reflect the many updates since it was last released
Exploring multithreading for performance/time improvement in test runs, to make them more viable for gate jobs or similar
Enabling Syntribos to understand context beyond a single request (i.e. enable tests to create, then read, modify, and delete a resource)
Rethinking request templates by using a less cluttered/repetitive format, and giving more information to Syntribos for improved testing accuracy
Stretch goal: Adding more formatters for results output (e.g. HTML, human-readable text)
Further improving test reliability & confidence, and reducing false positives

Members of Syntribos Team

Aastha Dixit (Github) - Intel
Charles Neill (Github) - Rackspace
Khanak Nangia (Github) - Intel
Matt Valdes(Github) - Rackspace
Michael Dong (Github) - Rackspace
Michael Xin (Github) - Rackspace
Rahul Nair (Github) - Intel
Vinay Potluri (Github) - Intel

Secure Development in Python

Mon, 26 Sep 2016 00:00:00 +0000

OpenStack is one of the largest Python projects, both in code size and number of contributors. Like any development language, Python has a set of best (and worst) security practices that developers should be aware of to avoid common security pitfalls. One mission of the OpenStack Security Project is to help developers write Python code as securely and easily as possible, so we created two resources to help.

Secure Development Guidelines

The Secure Development Guidelines were created with the goal to make it quick and easy for a developer to learn:

What is the best practice
An example of the incorrect (insecure!) way of accomplishing a task
An example of the correct way of accomplishing a task
Consequences for not following best practices
Links for further Reference

As developers ourselves we’re guilty of more than the occasional copy-paste. The Correct section of the Secure Development Guidelines are a perfect source to jump in and get the best practice code snippet you need.

Bandit

Bandit was built to find common insecure coding practices in Python code. Developed for the OpenStack community by the OSSP, it is the best Python static analysis tool available (in our biased opinion). Like all OSSP resources and tools, Bandit is open source and we encourage people to use it, extend it, and provide feedback.

If you’re new to Bandit a good way to get started is by watching this:

Also check out our wiki.

If you have any questions please contact us on the OpenStack Developer Mailing list (using the [Security] tag), or visit us on IRC in #openstack-security on Freenode.

Maturing the Security Project

Thu, 22 Sep 2016 00:00:00 +0000

This blog article is intended to address the recent discussions on the openstack-dev mailing list, following the suggestion by Thierry on behalf of the TC that the OpenStack Security Project “should be removed from the Big Tent” because the security team failed to nominate and elect a project ream lead (PTL) for the next release cycle. This process is required for all active project teams and is seen by the TC as a failure in community engagement that the OSSP has missed this deadline, again.

Back in the early days of the Security Project being in the big tent I missed the election deadline for my nomination. Pure oversight on my part, I was new to the role of PTL having been grandfathered in from the working group and I simply didn’t realise what was required for elections. ‘Missing a nomination once is bad, so missing the most recent nomination window is obviously very bad and raises questions over the level of engagement we have in the community, particularly as everyone in the OSSP also missed the email sent to highlight the closing nomination window (Its the one on the 19th)…

Unfortunately during the nomination window I was temporarily distracted dealing with some local issues. I’ve discussed these with a member of the TC who recognises that it was a temporary thing that’s unlikely to happen in the future, however the bell has been rung and we must decide how to proceed.

Maturing the Security Project

Missing two nominations reflects badly on a project team and leads to several understandable questions being asked: Who are these people? Are they an active team? Should they be moved outside of the big tent?

These are understandable questions, I feel that my on-thread response addressed them for the most part. What I want to focus on is the things that we need to do to be a better part of the community and ensure that project teams and the TC are both aware of what we do and how we help improve security in OpenStack.

We know from the feedback we’ve had from downstream OpenStack consumers that our work is valued, we need to better demonstrate that value within the OpenStack community. I think a good place to start is to look at the Project Team Guide examine what we are already doing and where we fall short. Of course this doesn’t include the good things we do like providing CI tooling for security, threat analysis etc but it is the minimum boxes that we should be ticking off as a project team and that I should be driving as PTL.

I want to be clear, I think that the Security Project is doing great things to enhance security in OpenStack. We need to become a better community player though, through doing so I expect new opportunities to innovate on security and create new ways to make OpenStack more secure.

Score Card

I’m proposing a score card for the security project, to ensure we’re doing all that we should be doing and identify those areas where we need to improve. I’ve based this on the Project Team Guide

Requirement	Status	Notes
Open Code	Achieved	All code in git and licensed appropriately
Open Design	Achieved	All design is open to the public, conducted at summits etc
Open Development	Achieved	We follow standard OpenStack best practice
Open Community	Needs Improvement	We have a gap around the mailing lists that we need to address
Public Meetings on IRC	Achieved	1700UTC Thursdays #openstack-meeting-alt
Project IRC channel	Achieved	#openstack-security
Community Support Channels	Mostly Achieved	We are strong on Launchpad and IRC which is where 90% of our workload comes from however we need to pay more attention to the ML and ask.openstack.org
Planet OpenStack	Achieved	This security blog posts to planet openstack
Participate in Design Summits	Achieved	Regular, very well attended sessions
Release Management	Achieved	We have a number of software projects that we created to support or enhance security in OpenStack. As they’re not directly consumed by OpenStack Operators they’ve not been part of the normal release cycle. Instead we follow the Independent release model.
Support Phases	Needs Improvement	Traditionally we have not followed the normal support phases for our projects because they have not been directly consumed by downstream OpenStack users. However there’s a clear opportunity to get more in line with the rest of the OpenStack community here. This should make things like rolling Bandit changes out through CI easier.
Testing	Achieved++	All of our software and documentation efforts have appropriate gate tests in place. Functional and Unit tests are in place where appropriate. We’ve also built tooling that other teams are using in their projects for Security gate tests. We’re not just testing, we’re also testing our integration with the projects that have adopted us.
Vulnerability Management	Achieved	Our software projects don’t have the vulnerability managed tag, however as the OSSP we do triage any security issues in our own software following standard processes, this is best demonstrated with the recent XSS issue in Bandit https://bugs.launchpad.net/bandit/+bug/1612988
Documentation	Achieved	We have a lot of documentation out there for customers and consumers of openstack OSSNs, security.openstack.org, the security guide as well as developer documentation such as that for Anchor and Bandit

The Four Opens

To paraphrase from the OpenStack documentation it’s important that any project participating in the big tent adopt and practice the “four opens”. Open Source, Open Design, Open Development and Open Community.

For the most part we have done a good job of following these, all of our code is developed under the appropriate Apache Licenses and all of our documentation efforts like the security guide, security notes, threat analysis etc are all conducted openly and use the same peer review tools as our code projects. We develop new ideas in the open, attend design summits and encourage new contributions.

Where we have not done such a good job is with the Open Community goal. Of course our team is open to new ideas and new contributions but we have not been as big of a participant in the larger community as we could have been. Our work with the VMT typically means that teams are driven toward us when they require our assistance.

I’d like to expand a little bit more on what Open Community means and where we can improve. OpenStack has some very good documentation on this topic but again I’ll paraphrase here.

Public Meetings on IRC: This is something that the security project has always done. We can be found on #openstack-meeting-alt at 1700UTC every Thursday. Our meetings are public and logged we have a standing public agenda that any developer is welcome to contribute to if they want to participate in the meeting, we also welcome people dropping by with questions, comments etc.

Mailing Lists: When the Security Project first formed we were a working group, we had a separate mailing list that didn’t get used for many things but for legacy reasons that I can’t remember (we’ve been doing security for OpenStack since Essex) we had a private list. As I said it didn’t get used much in our day-to-day and I think that’s a bad practice that we carried across to our big tent operations.

Largely I think this disconnect from the mailing list has arisen because it was not our experience that we needed to use it. Most of our work has always come from teams reaching out directly to us, typically via IRC. I think it will always be the case that teams will be more active on one communication medium than another but I fully accept that to meet our obligations under the four opens we must find a way to work more effectively on the mailing lists.

Community Support Channels: We manage all of our bugs on LaunchPad, that’s the primary way we interact with the VMT. Our IRC channel is reasonably active but as we’ve described above we certainly need to do better on the mailing lists.

Impact of removing Security from the big-tent

Although I think it’s been addressed a number of times on the mailing-list thread I’d like to reiterate two themes from the responses regarding concerns of removing Security from the OpenStack big-tent.

Legitimacy: As can be gleaned from this blog, we haven’t done the best job in making the wider OpenStack community aware of what it is that we do, probably even some teams who are running Bandit in their gate might not realise that it’s a tool that we created for OpenStack to be more secure. However even with teams that haven’t heard of us, we are able to quickly gain traction when they see that we are a ‘proper’ OpenStack project - the truth of the matter is that how most people see OpenStack, you’re either in the tent or you’re largely an irrelevance. We know this because we started outside of the tent and found it much harder to engage with teams where we could see there were obvious security issues. Being outside of the big-tent will make it very difficult for us to act as an authority for signing off that a project has taken reasonable security steps before applying for a vulnerability managed tag, a relatively recent change.

Investment: Running any OpenStack project requires investment, very few projects succeed based only on people working on them in their spare time. For the most part investment here means giving people time to contribute to Security as part of their working week, to provide funding for spaces for meetings and mid-cycles and to cover the time and expenses of contributors travelling to design summits etc. It’s no secret from looking around OpenStack that some historically big contributors have been scaling back the number of people they send to summits, the numbers of active contributor they maintain etc. Having been in the position of lobbying various corporations for support in these areas I cannot imagine a scenario where we could leave the big tent and continue to dedicate time to the efforts we have in place.

Without the legitimacy we have from being part of the big-tent we will not get the investment required to deliver and enhance security within OpenStack.

Moving Forward

I think it’s clear by now that I want the Security Project to have the opportunity to stay within the big-tent. I’d like to continue on as PTL at least through a period of maturing the Security Project to ensure that our baseline operations are aligned with what the wider community expects of any big-tent project.

I want the opportunity to improve the score card above and have us achieving everything on that list. I see no reason why we can’t begin acting on these things now and that our status can easily be judged on this basis during the next election cycle.

Clearing the Air about Vulnerabilities in OpenStack

Thu, 05 May 2016 00:00:00 +0000

Recently, there have been a few talks around “vulnerabilities” within the OpenStack project that introduced some undue concern.

Some of them call out important concepts such as CIA and the CVE database. Unfortunately, they all attempt to highlight vulnerabilities or attack vectors within OpenStack that have either been addressed years ago, or are not able to be addressed by the upstream community and are the responsibility of the group deploying and maintaining the cloud.

To understand how OpenStack handles vulnerabilities securely, it is important to briefly introduce the OpenStack vulnerability management process.

Vulnerability Management in OpenStack

A vulnerability in OpenStack usually begins life as a bug filed against a project with the “security” tag. These bugs are marked private and sent directly to the project’s security team and the Vulnerability Management Team (VMT). An initial triage is performed to understand whether the bug represents a legitimate security issue and if so what the impact is. If the issue is confirmed, an advisory and patch are prepared and validated privately. Once the the advisory and fix are available, OpenStack stakeholders are given two weeks early notice to patch their systems before public disclosure. The reason the two week notice is important is because it is expected that a responsible OpenStack provider will respond to security advisories in a timely manner.

Vulnerability Management Outside of OpenStack

The Vulnerability Management Team (VMT) only manages issues for OpenStack components (with the “vulnerability:managed” tag) inside the OpenStack ecosystem. A portion of recently discussed “vulnerabilities” have to do with third party applications deployed on an OpenStack cloud. The responsibility for securing third party applications is shared between the third party developer to produce timely patches to secure products, distributions to make sure there are sane defaults baked-in to the product, deployers to ensure their configurations are tuned to their environment and applications as well as patching methods, and the OpenStack community. To that end, the OpenStack Security Project maintains the OpenStack Security Guide for help architecting secure environments, and Security Notes to address common deployment-specific issues that have been found.

Conclusion - We’re Here To Help

The OpenStack community is very concerned about security, and actively engages the upstream community, deployers, and operators to help increase the overall security posture of every OpenStack deployment. The Security Project has released many tools and references to assist everyone with the knowledge to securely configure and maintain their OpenStack cloud.

Finally, if you have found a security issue in OpenStack, please disclose it responsibly by marking “security” in the LaunchPad bug report, or by contacting the involved project team or VMT team members directly. The Security Project welcomes everyone wishing to discuss, learn, or contribute to the security of the OpenStack project and can be found on freenode in the #openstack-security room, or on the OpenStack developers mailinglist with the [security] tag.

Applying threat analysis to Anchor

Thu, 28 Apr 2016 00:00:00 +0000

As a followup to my previous post on Threat Analysis I started working through a simple TA process for Anchor with a view to seeing how long the process takes as well as trying to understand how we should document the steps that are required. I think we need to end up with a point by point guide to TA, some simple process that is repeatable and somewhat deterministic.

A good measure of the quality of this documentation would be to have two groups of developers from the same project attempt to perform TA in parallel and compare the results.

We are still working on the process as can been seen from this review.

Reference architectures

One of the problems when trying to work out how to create threat analysis documentation for OpenStack services is that they can be configured in so many different ways. Anchor is probably one of the least complicated services in the ecosystem, capable of being deployed in a completely stand alone, single service, single host configuration. However, this is not how it’s intended to be used ‘in production’. The expectation is that developers use best judgement on what the best practice architecture should look like, what components should be present and what should be optional or recommended. You can see this in the architecture diagram for Anchor below. I decided to represent the system in an HA configuration but with LDAP and the audit queue as optional components. The uses, threats to and protections for the optional components will be included in the TA but the notation will highlight that Anchor does not explicitly require these services to run.

Anchor components

In this diagram you can see that a typical Anchor deployment typically consists of just a load balancer, a couple of Anchor instances and the Anchor configuration file which is stored on disk.

The interface list in this diagram doesn’t contain a lot of information but it’s really just there for a quick reference.

Security Requirements

For each component we like to run through a list of basic security considerations these are at a minimum Confidentiality, Integrity and Availability (C.I.A) - bonus points are awarded for including Authorization, Authentication and Auditability.

Artefact - Generic term for a given component, interface or asset.

Confidentiality: Does this artefact include or access information that should be kept secret - could this information be used to compromise the security or integrity of the system. In the context of the Anchor project, the on disk config, validators and AuthN information stored on-disk (c4) and read through interface (5) contain confidential information.
Integrity: Does this artefact include or access information that it’s critical remains correct? Often it is the case that components that have strong confidentiality requirements will also have strong integrity requirements but there are times where strong integrity requirements can exist without confidentiality. Consider a time signal that a service relies on for synchronisation - the time of day is in no way confidential or sensitive however it’s critical to the process that the integrity of the time signal is preserved.
Availability: Will the system overall fail if this component, interface or asset is momentarily unavailable - what is the impact of a potential failure?
Authentication: Does the artefact involve or require authentication for some or all operations - what is the impact of the authentication system failing.
Authorization: Once an entity has authenticated (the system knows the identity of the user) - is it authorized to perform an action. What happens if an authorization failure allows all users to perform actions?
Auditability: Sometimes also considered as non-repudiation (I wont go into the difference here) is the ability of the system to maintain a immutable log of operations and events that occurred with a level of granularity that allows an investigator to reconstruct any given series of events.

Discussion / Dissection

At this point in the review it’s time to talk about the block diagram. What data travels between components and what happens to that data within each component. Reviews should consider and capture:

What a reference architecture looks like and what parts are optional
The interfaces between components, what data travels over those interfaces
The security requirements for each interface
The protocols used

The content of this discussion should be used to inform the component and interface lists as shown below. Consideration should be given to

Reference Architecture Validation

Does the presented architecture make sense as a reference for future deployer? Yes - the block diagram uses dotted line objects to denote optional components. These components should be present in a strong and robust deployment of Anchor but are not required for a basic / test deployment.

Component list

A component list is particularly useful for large projects as it helps keep track of all the different parts of a system under review. The component list describes the entities in the system and how they might process and persist data.

ID	Name	Purpose	Persists Sensitive Data	Exposed Protocols
c1	Client System	Any server or service that requires a certificate for operations.	Yes - stores certificates from Anchor as well as it’s own private keys.	TLS / 443
c2	Load Balancer	Not strictly required for Anchor but strongly advisable. This component balances traffic between two or more instances of Anchor	No - Passes data between Anchor instances and clients.	TLS / 443
c3	Anchor Instance	To validate certificate requests and generate certificates based on request data.	Yes - Anchor reads configuration data from disk but does not store anything locally other than log caches when the audit stream isn’t available	TLS / 443
c4	Configuration File	To store configuration information	Anchor never writes to this file. It reads lots of sensitive data from the file including validation rules and credentials	Filesystem DAC
c5	Audit Queue	To receive, process and forward log data from Anchor instances.	Anchor emits no sensitive log data. Audit data contains: which host was issued a certificate, the type of authentication used and the validation rules that were met. Depending on configuration the audit stream data may be logged in it’s pre-processed form. Alternatively the representation within a target log management or SEIM application my be persisted.	Unsure
c6	LDAP Server	Anchor can use LDAP to simply authenticate a request or can use group membership as part of validation rules. I.E only a user in the “Nova Engineering” group is allowed to generate certificates matching the “*.compute.cloud” schema	LDAP undoubtedly stores sensitive data however no sensitive data is persisted in LDAP by Anchor or by any side affect of Anchor running.	Unsure

Interface List

The interface list describes how these components communicate with each other.

ID	Name	Purpose	Protocol(s)	Confidentiality	Integrity	Availability	Boundaries
1-2	Client to Load Balancer	The client connection to Anchor. Although this goes to a load balancer first, the client perception is that they are connecting directly with an Anchor instance	REST or CMC - both over TLS	Access credentials are passed over this connection. The credentials are not encrypted but the connection is.	Integrity of requests is important and protected by TLS.	Availability of 2 is important - this is why a LB is used in front of Anchor. Potentially two LB could be deployed with a DNS round-robin configuration	Public / Edge Network -> Control Plane
3-4	Load Balancer to Anchor Instance	This is the main communication channel for Anchor operations. The data from the client is passed to anchor over TLS from the load balancer.	REST or CMC - both over TLS (whatever was provided over 1-2)	See 1-2	See 1-2	See 1-2	Public Facing / Edge Network -> Internal Network / Control Plane
5-6	Disk Read	Anchor reads configuration data from disk. This data can contain AD credentials, validation rules etc. Appropriate DAC and MAC should be set	FS Reads	High	High	Low/Med - Anchor requires access to this file only when the service starts	Entirely within control plane
5-7	Audit Stream	To log events from Anchor	CADF	Low - no sensitive data in logs although in aggregate logs could provide an attacker with an understanding on the layout of the infrastructure	High	Medium	Within control plane
5-8	LDAP connection	To authenticate users and verify that the group they reside in matches what’s required in validation rules	LDAP over TLS	High	High	High	Internal Network / Control Plane -> External corporate network

The next step in the process is to generate sequence diagrams for a selection of common or important operations within the system under review. This is again where the process relies on the best judgement of the reviewers - to generate enough sequence diagrams to map out major functionality or to ensure that the various interfaces in the reference architecture are explored.

Sequence Diagrams

As a brief reminder to the reviewer, sometimes it’s useful to include a simplified diagram that explains the general principles of a system such as the one below.

Obviously, not much progress can be made with this diagram alone but for more complex systems it can sometimes be useful. This diagram might be a simple one that shows the different parts of a system that handle various API calls for example.

The tool that I chose to draw the sequence diagrams is available online at https://www.websequencediagrams.com it allows you to simply describe a sequence using text that is then parsed and turned into a diagram. The syntax is pretty trivial and comes from the js-sequence-diagrams project: https://bramp.github.io/js-sequence-diagrams/ . Below is the text that was used to generate the simple diagram above:

title Simplified Certificate Request Flow
Client->Anchor: HTTPS [ Certificate Signing Request & Credentials ]
Anchor-->Anchor: Validate Credentials
Anchor-->Anchor: Validate Request
Anchor->Client: [ Certificate | Error ]

The threat analysis project does not place any requirements on what tools should be used to generate diagrams but in our experience the js-sequence-diagrams syntax is the easiest to use. The diagram below runs through the process for Anchor to sign (or refuse to sign) a certificate signing request.

In this diagram we’ve used a few annotations to help the reader more easily understand the communication taking place. First lets take a look at the complete source code for this diagram:

title Detailed Certificate Request Flow
Client System->Load Balancer: [1-2] HTTPS [GET /sign? & Certificate Signing Request & Credentials ]
Load Balancer->Anchor Instance: [3-4] HTTP [GET /sign? & Certificate Signing Request & Credentials ]
Anchor Instance->LDAP: [5-8] Get group for user credentials
LDAP-->LDAP: Lookup user
LDAP->Anchor Instance: [5-8] Group membership info
Anchor Instance-->Anchor Instance: Validate group matches requested certificate
Anchor Instance-->Anchor Instance: Apply all validation rules from on disk config
Anchor Instance->Audit Queue: [5-7] CADF [ Certificate Signing Request & Decision & User ]
Anchor Instance->Load Balancer: [3-4] HTTP [ Certificate | Error]
Load Balancer->Client System: [1-2] HTTPS [ Certificate | Error ]

The communication between the client system and the load balancer takes place over interface [1-2] (see the interface table above to check what that interface is and the security requirements around it) the protocol is HTTPS, the http verb is ‘GET’ and the resource is ‘sign’. The first option passed to this is the ‘Certificate Signing Request’, this is separated from other parameters by the ‘&’ character.

Client System->Load Balancer: [1-2] HTTPS [GET /sign? & Certificate Signing Request & Credentials ]

Sometimes we want notation that describes an either/or relationship rather than an inclusive list, in this case we use the ‘|’ character:

Anchor Instance->Load Balancer: [3-4] HTTP [ Certificate | Error ]

These diagrams help reviewers to understand how components communicate and how date flows throughout a system. For each operation reviews can use the interface ID to check what security considerations have been given to that interface and if they map well to the operations described in the sequence diagram.

Lightweight Threat Analysis Process for OpenStack

Tue, 26 Apr 2016 00:00:00 +0000

Lightweight Threat Analysis Process for OpenStack

Following on from our previous posts on Threat analysis and Applying threat analysis to Anchor, we have been working on defining a lightweight process for threat analysis which can be applied to OpenStack projects. This blog post gives a first look at the draft process, the final location of which is to be decided, but it is likely to be in a wiki page, or possibly in the security docs project.

The materials are currently formatted in RST, due to their location as part of the security docs project, they can be cloned with:

git clone git://git.openstack.org/openstack/security-doc
git review -d 220712

We focus on four stages of the threat analysis process:

Preparing artifacts for review
Verifying readiness for a threat analysis review
Running the threat analysis review
Follow-up from the threat analysis review

Preparing artifacts for review

Complete the architecture page. The architecture page describes the purpose of the service, and captures the information that is required for an effective threat analysis review. A template for the architecture page is provided here and there is guidance on diagraming here. If further help or advice is needed, please reach out to the Security Project via the [email protected] mailing list, tagging your email [security].
The architecture page should describe a best practice deployment. If a reference architecture is available this may be a good example to use, otherwise the page should describe a best practice deployment, rather than the simplest possible deployment. Where reference architectures do not exist, it is possible that the architecture drawn for the threat analysis process can be used as a reference architecture.
The following information is required in the architecture page for review:
1. A brief description of the service, its purpose and intended usage.
2. A list of components in the architecture, their purpose, any sensitive data they persist and protocols they expose.
3. External dependancies and security assumptions made about them.
4. An architecture block diagram.
5. Either a sequence diagram or a data flow diagram, describing common operations of the service

Before the review

Verify that the service’s architecture page contains all the sections listed in the Architecture Page Template.
The architecture page should include diagrams as specified in the Architecture diagram guidance.
Send an email to the [email protected] mailing list with a [security] tag to announce the up-coming threat analysis review.
Prepare a threat analysis review etherpad, using this template .
Print the architecture page as a PDF, to be checked in along with the review notes, as evidence of what was reviewed.

Running the threat analysis review

Identify the “scribe” role, who will record the discussion and any findings in the etherpad.
Ask the project architect to briefly describe the purpose of the service, typical uses cases, who will use it and how it will be deployed. Identify the data assets that might be at risk, eg peoples photos, cat videos, databases. Assets in flight and at rest.
Briefly consider potential abuse cases, what might an attacker want to use this service for? Could an attacker use this service as a stepping stone to attack other services? Do not spend too long on this section, as abuse cases will come up as the architecture is discussed.
Ask the project architect to summarize the architecture by stepping through the architecture block diagram.

While reviewing the architecture, perform the following steps:

For each interface between components, consider the confidentiality, integrity and availability requirements for that interface. Is sensitive data protected effectively to prevent information disclosure (loss of confidentiality) or tampering (loss of integrity)? Is there a requirement for availability which should be documented and added to reference deployments? In addition to considering the authenticity of the data in transit, consider how the authenticity of the sending and receiving nodes is assured.
Consider the protocols used to pass data between interfaces. Is this an appropriate protocol, is it a current protocol, does it have documented vulnerabilities, is the implementation in use maintained? Is this protocol used as a security control to provide confidentiality, integrity or availability?
Can this interface be used as an entry point to the system, can an attacker use it to attack a potentially vulnerable service? If so, consider what additional controls should be applied to limit the exposure.
If an attacker was able to compromise a given component, what would that enable them to do? Could they stepping-stone through the OpenStack cloud?
How is the service administered? Is this a secure path, with appropriate authentication and authorization controls?

Once the reviewers are familiar with the service, re-consider abuse cases, are there any other cases which should be considered and mitigated?
Step through sequence or dataflow diagrams for typical use-cases. Again consider if sensitive data is appropriately protected. Where an entry point is identified, consider how risks of malicious input data can be mitigated.
If any potential vulnerabilities are identified, they should be discussed with the project team, if they agree that it is an issue then a note should be made in the findings section of the etherpad, with a short title and summary of the issue, including a note of who found it. If the project team disagree, then the note should be made under the further investigation section.

Follow-up from the threat analysis review

Create a separate bug for each of the security findings listed in the TA Review notes.
Update the Threat Analysis Review Etherpad each of the new launchpad bug numbers.
Paste the contents of the Threat Analysis Review Etherpad into a text file in security-docs/security-ta/notes and push it to the security-review repo using gerit.
Distribute the Threat Analysis Review Notes via email to all who were present at the threat analysis. If anyone discovers errors or omissions in the notes, then make corrections.
On the threat analysis reviews wiki page create a new row in the reviews table, include a link to the master bug, the date of the review, the PTL and reviewers.

Glance Signed Image Validation

Sat, 26 Mar 2016 00:00:00 +0000

Glance Signed Image Validation

A new addition to the OpenStack Security Guide is Signed Image Validation in the Glance service. This will now allow boot-time assurance that an image has not been tampered with before it is booted. The steps for doing this are

A signature of the image is created
A Keystone service context is created
The image signature is encoded and uploaded to Castellan
The image is uploaded to the Glance service
The verify_glance_signatures is set to True in the /etc/nova/nova.conf file

A detailed list of the specific actions for each step is located at Adding Signed Images to Glance.

Once the configuration details above have been taken, when an image with a signature hash in its metadata is referenced as the boot image, the Nova service will securely copy the image from Glance, and compare a hash of the copied image against the signature in from the metadata. If this hash matches the image will boot, giving the user assurance it has not been tampered with.

Pragmatic Security: When Your PoC Goes into Production

Fri, 18 Mar 2016 00:00:00 +0000

Creating a Cloud Security Program

A common anecdote when looking at OpenStack cloud deployments is that when an issue arises - such as with timetables or migrating an application to a cloud-native architecture - that security in one of the first items to slip from the schedule. So when your Proof-of-Concept cloud is ready to be deployed in Production, there are occasional gaps in the overall security and assurance of the environment.

This will be a one size fits all post, considering how each environment will vary on mandatory security requirements. It will be up to the reader to adapt the recommended Security Software Lifcycle Management program to their environment.

Teams

Teams will be referenced in these posts with as much clarity as possible, and our assumption is that they are roughly assigned as service teams - which would take care of an individual OpenStack service such as Nova - and development teams - that are in charge of the applications that run on the deployment. In your situation, this may be able to be directly mapped to individual teams, a single team, individuals responsible for a given service, or even a single individual. It will be up to you to determine what team is best to engage for a given issue.

Each team should be responsible to a master architect - a single technical resource that is tasked with overseeing the integration details for each service and application to ensure they are all able to integrate and develop together for future integrations.

Additionally, there should be a dedicated security architect who will work with both the service teams, development teams, and master architect to ensure the security of the OpenStack deployment. The Security architect should not be responsible to the master architect, but rather a peer that is able to influence priority and build features.

The Gate

It is also assumed that you are utilizing some type of version-tracking software such as git or SVN. Both your services, configurations, orchestration, and application teams are utilizing these version-tracking applications for code check-in and to then trigger gate and build jobs.

This article will assume that Git is used for versioning, and Jenkins for check-in jobs like pep8.

The repositories that approved check-ins are merged into can be considered the ‘source of truth’ for code and configuration information in a running environment.

Security Activities

The security team will work to build security and assurance through each step of the software lifecycle. Traditionally, this is encompassed by host hardening, code review, and firewall/network ACLs. In a cloud native environment, there are a few additional recommended steps. Additionally, the Security Team will also interpret, maintain, and enforce the Security Policy.

Secure Architecture

The Security team works with each service team to identify the best practice approach for a given service, the threat landscape, and secure methods to any identified threat vectors. These are captured in network diagrams, wiki pages, and configuration management databases for reference by the teams required to implement the devices.

Threat Analysis

Once a secure architecture has been documented, the security team and major stakeholders would be invited to a meeting that critically analyzes each service in context where the security architect who influenced the architecture introduces it to the rest of the security team and stakeholders to get addtional points of view of exposure and possible exploit vectors for review.

Static Analysis

On a per-application basis, the appropriate static code analysis tool can be selected and included in the gate so that “low hanging fruit” can be identified and fixed by a developer on check-in, and not depend on other developers or an offical code review by an external party.

Continuous Assurance

The virtual machines used to host the applications will also need to be hardened, and can be through an orchestration tool such as Chef or Ansible. Once a secure baseline has been determined, the orchestration tools can be used to deploy the proper versions of packages and utilities and bring each host up into a known hardened state.

Regular Review

Additionally, the policies, configurations, and results of the above should be validated on a regular cadence. Scheduling regular reviews for each - such as starting with an annual review of each, and then increasing as needed - will ensure the controls are accurately applied for the current environment.

Security Coverage: Adding the Vulnerability:Managed Tag

Fri, 18 Mar 2016 00:00:00 +0000

The Vulnerability Management Team

The OpenStack Vulnerability Management Team (VMT) provides a point of contact for individuals or groups wanting to report a security issue in OpenStack. They do the initial triage and response to any reported vulnerabilities for the projects that have the vulnerability:managed tag. This tag provides a clear indication that a project has coverage from the VMT, and allows OpenStack to have a reasonable security baseline - an assurance passed on to implementors and operators of OpenStack clouds.

Requirements

To create this assurance for a given service, the VMT has a list of requirements to be fulfilled before requesting the vulnterability:managed tag. There are six requirements the VMT looks at when a service requests inclusion.

The vulnerability:managed tag applies to all repos within a given deliverable.
The deliverable must have a dedicated point of contact for security issues.
The PTL for the deliverable is also a point of contact, or delegates one.
A defect/bug tracker for the deliverable is configured to initially only allow access to the VMT, which will then bring in deliverable liaisons as needed.
The deliverable repository is audited for security by a third party.
Automated testing is in place to cover the main features of the deliverable, and are lightweight enough to run locally, but are also in the OpenStack CI infrastructure.

The complete description of each of the requirements is on the Security.OpenStack.org VMT ‘Vulnerability Managed’ page.

Using the OpenStack tracker and CI infrastructure will allow for requirement number four - secure defect/bug tracking, and allow easy extension for requirement six - automated testing.

Requesting The Tag

Once the above requirements are met, a thread describing the request should be created on the openstack-dev mailinglist. Once the request is responded to by the VMT, the tag can be requested through a change to the OpenStack Governance repository. An example request can be seen for the Ironic project

Vulnerability, Managed

Once the above change is merged, the VMT will be able to receive secure bug defect reports, be able to analyze them to determine if they are legitimate or not, develop a patch to remediate the issue, have the deliverable’s point of contact review the patch for impact on the service, and responsibly disclose the defect, impact, and patch to downstream stakeholders.