AWS Fine-Grained Access Control
I wrote up a short wishlist for AWS a while ago which also included Fine-Grained Access Control for the API. I am very excited that Amazon announced the AWS Identity and Access Management which tackles this problem. It is a preview beta and not yet fully integrated into the management console, but still a good move from Amazon from a security perspective.
ACM CCSW 2010 preprint and Master's thesis
I have uploaded my Master’s thesis with the title: Automated Security Analysis of Infrastructure Clouds
We also derived a paper from it which got accepted at the ACM Cloud Computing Security Workshop:
Security Audits of Multi-tier Virtual Infrastructures in Public Infrastructure Clouds
Abstract:
Cloud computing has gained remarkable popularity in the recent years by a wide spectrum of consumers, ranging from small start-ups to governments. However, its benefits in terms of flexibility, scalability, and low upfront investments, are shadowed by security challenges which inhibit its adoption. Managed through a web-services interface, users can configure highly flexible but complex cloud computing environments. Furthermore, users misconfiguring such cloud services poses a severe security risk that can lead to security incidents, e.g., erroneous exposure of services due to faulty network security configurations.
In this article we present a novel approach in the security assessment of the end-user configuration of multi-tier architectures deployed on infrastructure clouds such as Amazon EC2. In order to perform this assessment for the currently deployed configuration, we automated the process of extracting the configuration using the Amazon API. In the assessment we focused on the reachability and vulnerability of services in the virtual infrastructure, and presented a way for the visualization and automated analysis based on reachability and attack graphs. We proposed a query and policy language for the analysis which can be used to obtain insights into the configuration and to specify desired and undesired configurations. We have implemented the security assessment in a prototype and evaluated it for practical scenarios. Our approach effectively allows to remediate today’s security concerns through validation of configurations of complex cloud infrastructures.
Calling Python functions from Java using Jython
If we want to call a function written in Python from Java, which takes
as input a Java object and returns another object, we can realize it
using the following snippet build on Jython.
XenStore Dump of Rackspace Cloud
For the curious: A dump of the XenStore config for a VM running on RackSpace Cloud. Actually quite a straight forward and simple setup: using image files for both root and swap disks, and a bridged network setup for the two interfaces (one for internal, one for external). Although the network is bridged, the VM is not able obtain traffic for other VMs as they are using VLANs for separation.
Notes on Amazon EC2's Network
Shortly after publishing my notes on the EC2 architecture, I was looking into the networking setup of EC2 and in particular figuring out their address schemes. Since I am currently no longer interested in such information, I will publish my incomplete notes and the raw data gathered from about 80 instances in this post. My notes are based on information obtained from small instances in the us-east-1d zone.
IP Address Scheme
I assumed the first hop in the traceroute from a VM is the actual dom0 IP address.
Consider the private IP addresses in the form 10.X.Y.Z. I have noticed that Y is partitioned into blocks containing a /24 for dom0 IP addresses, a /24 for VMs, and a /23 for another set of VMs. For example: 10.208.176/24 is the dom0 range; 10.208.177/24 the first VM range; 10.208.178/23 the second VM range.
Based on my data, the dom0 IP addresses always end in .2 or .3, but there seems to be no pattern between a VM’s IP address and the ending of the corresponding dom0.
MAC Address Scheme
I do not have many information on this one. MAC addresses are typically in the form of 12:31:39:X:Y:Z, where X can be derived from the second octet of the private IP address. The following list gives the value of X for the second IP address octet. As an example: IP 10.210.X.Y leads to 12:31:39:09:X’:Y’, because 09 is listed for octet 210.
00 254
01 255
02 248
03 249
04 240
05 241
06 208
07 209
09 210
0A 211
0B 214
0C 215
Raw Data
The raw data can be found here. It contains network configuration information (ifconfig, traceroute, and routes) of about 80 instances from the us-east-1d zone. Let me know if you make any interesting discoveries based on that data.
Another Short Wish List for Amazon Web Services
In continuation of my first wish list, I will present another set of potential improvements and ideas for Amazon Web Services in this post. The focus lies on security groups and virtual private cloud.
Security Group: Outbound Filtering
Many people have written about potential improvements for Amazon security groups, see for example Dmitriy’s write-up, and I just want to emphasis one point from their wish lists: the possibility to filter outbound traffic. I understand that Amazon is quite pragmatic with their feature set and only provides what the majority of their customers require. For the majority the current situation with filtering inbound traffic might be good enough, but with an increase in cloud usage for “serious” purposes, filtering outbound traffic will be of importance. And honestly, I doubt the technical difficulty in implementing that feature when they are already able to filter inbound traffic.
AMI Security Group Membership Policies
AMIs can contain services which might not have a strong authentication and authorization mechanism, for example consider memcached. The problem is to ensure that instances of such an AMI are properly secured with rules of an appropriate security group, e.g., only allow web/application server to access the memcached. An AMI could be accompanied by a policy specifying the properties of a security group the instances of that AMI can only be a member of.
Amazon Simple Logging Service
Logging is a crucial part of IT infrastructure and I am surprised that Amazon has not released such a service, although the idea was already proposed in a post in the Amazon forum.
For one thing, they could publish logs of the underlying architecture relevant for specific VMs, e.g., logs from the security group filtering. On another thing, they could provide a syslogd interface to VMs in order to gather logs from customer instances. The aggregated logs can then either be processed by Amazon (for example running a simple logcheck) or further processing can be done by third-party applications (for example an IDS).
Amazon Virtual Private Cloud
I know that Amazon VPC is still considered a service in beta phase, and I won’t pick on the limit of one VPC per customer, but still I want to share two points.
Subnet Filtering
A VPC subnet can further be divided into multiple subnets (up to 20), which are aggregated in a star topology with a router in the middle. I am looking forward in having access to filtering capabilities of that a router, in order to restrict traffic between the various subnets. This would greatly enhance the possibilities of VPC subnets.
Security Groups
In case subnet filtering on the router would not be possible, having security groups in combination with VPC would be great. A VPC subnet could just be considered as a security group, which allows regular filtering between the various subnets using security groups as sources. It would also allow filtering for the individual instances in the management layer, rather than doing filtering within the VMs.
How Do You Use Amazon's Security Groups?
This time I would like to reverse the flow of information and would kindly ask you to fill out the following questionnaire about how you use the Security Groups of Amazon Web Services. It will only take a few minutes of your time.
Thanks for your participation and I welcome any form of feedback.
My Short Wish List for Amazon Web Services
In the short time I have used AWS, I came across a few issues which I would like to see changed by Amazon in the foreseeable future. It mainly concerns the management side of AWS rather than the technical one.
Fine-Grained Access Control for the API
Cloud computing is destined to be used by applications and automation infrastructure, therefore the API is an essential component in the cloud provider’s offering. So far I am pleased with the functionality of the existing API, but the access control is very rigid and inflexible. Although Amazon supports multiple types of credentials and allows multiple instances of each credential, the access associated with these can not be controlled.
If a credential is getting compromised, the attacker would get full control over ones IT infrastructure in the cloud and could use additional resources, e.g. for further malicious purposes, which are billed by the victim’s account. A typical scenario is an application running on a dedicated server using additional resources from the cloud, therefore requires the access key, or another kind of credential, to be stored on the server. If the server is getting compromised, the access key will be compromised as well and the attacker can use it for the previously mentioned cases.
Specifying in a fine-grained way for which purposes a credential can be used would limit the impact of a compromised credential. This can be on a high-level at the beginning, e.g. specifying a credential can be used for write or read access. I define write access as operations modifying the cloud, e.g. creating new instances or shutting down instances, and read access as ones only obtaining information. The latter type of access is useful for monitoring applications, where access for modifying the cloud is generally not required. In the long-term, this could be extended to limit credentials for certain kind of operations, only allowing credentials to operate on specific services, or limit the issuer of operations using a credentials (e.g. only the operations issued from the corporate network are allowed).
Notifications of Resources Usage
AWS provides usage information for the resources in the cloud in a XML format, but does not have a functionality of sending notifications for what kind of resources were used and how much they cost in a specific interval. For example I would like to receive a daily email summarizing the resources I have used and how much they cost for that specific day. I am sure one could implement such a feature with the provided usage information, but it would be much more convenient if it is provided by Amazon out of the box. Especially since the session lifetime for viewing the account activity is very short.
Resource Usage Limitations
Consider the scenario that you want your developer team to use the cloud for testing or development purposes, but you also want to limit the amount of money they are allowed to spend. If one of the developers forgets to terminate his 20 extra large instances for a longer time-frame, it will be charged at the end of the month on your credit card. Of course the previously mentioned notifications would probably prevent such a scenario, but it also makes sense to have a soft and hard limit on the resource usage, in case one is not interested in daily usage notifications.
For example you allow your developers to spend $500 on EC2, and after that limit is reached they are not able to launch new instances. Furthermore you specify $550 as the hard limit, i.e. after that amount is reached all running instances are terminated.
Of course these kind of limitations can be tricky in certain situations, e.g. when dealing with storage. Are you deleting storage if the limit is reached or not? The combination of hard and soft limits, and notifications could maybe make it feasible.
Outlook
I am also very interested in what kind of problems you are having with AWS and what you would like to see changed. I am in particular interested in security concerns and issues.
Amazon VPC with Linux
This post will cover how to connect a Amazon Virtual Private Cloud (VPC) with a Linux-based VPN gateway. In Amazon’s Network Guide for setting up a gateway connecting a VPC with existing infrastructure, they only provide specific configurations for Juniper and Cisco devices. Other devices, like a Linux-based VPN gateway, are covered by their documentation for Generic Devices, but lack specific configuration files. In this post I will explain how to setup a VPN gateway using racoon, ipsec-tools, and quagga on Ubuntu Linux.
Setup a VPC
The first thing to do is to follow the VPC tutorial and setting up all the VPC objects, like vpn-gateway, vpc, subnet etc. When specifying the customer VPN gateway, you can assume a private BGP ASN like 65530, which we will use later in the example configuration files. Just follow the steps in the tutorial and skip the part of configuring the customer VPN gateway, which we will do afterwards. Having completely configured the Amazon side of the VPN, we can immediately test the configuration of our customer VPN gateway.
Setup of IPSec
First, you have to install the necessary packages for running a VPN gateway using IPSec: apt-get install ipsec-tools racoon. In case you are running a packet filter on the VPN machine, please allow UDP port 500 for racoon/IKE.
When creating a VPN connection in the Setup a VPC section, Amazon will provide you with all the required configuration parameter for setting up the IPSec tunnel. Of particular interest are the Pre-Shared Keys and the IP addresses, which can look like the following:
Outside IP Addresses:
- Customer Gateway: : 1.2.3.4
- VPN Gateway : 72.21.209.225
Inside IP Addresses
- Customer Gateway : 169.254.255.2/30
- VPN Gateway : 169.254.255.1/30
Outside IP Addresses:
- Customer Gateway: : 1.2.3.4
- VPN Gateway : 72.21.209.193
Inside IP Addresses
- Customer Gateway : 169.254.255.6/30
- VPN Gateway : 169.254.255.5/30
We pretend that the external IP address of our customer VPN gateway is 1.2.3.4, but you will have to replace that value in the following configuration files with your actual address.
Update: The IP addresses of the VPN gateways can be in the opposite order, i.e. 72.21.209.193 for the address range 169.254.255.1/30. If this is the case, the ipsec-tools.conf have to be adjusted accordingly.
/etc/racoon/psk.txt
This file holds the pre-shared keys for the two IPSec tunnels:
72.21.209.225 password1
72.21.209.193 password2
/etc/ipsec-tools.conf
In this file we specify the IPSec policies for the two tunnels:
flush;
spdflush;
spdadd 169.254.255.2/30 169.254.255.1/30 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.225/require;
spdadd 169.254.255.1/30 169.254.255.2/30 any -P in ipsec
esp/tunnel/72.21.209.225-1.2.3.4/require;
spdadd 169.254.255.6/30 169.254.255.5/30 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.193/require;
spdadd 169.254.255.5/30 169.254.255.6/30 any -P in ipsec
esp/tunnel/72.21.209.193-1.2.3.4/require;
spdadd 169.254.255.2/30 192.168.0.0/24 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.225/require;
spdadd 192.168.0.0/24 169.254.255.2/30 any -P in ipsec
esp/tunnel/72.21.209.225-1.2.3.4/require;
spdadd 169.254.255.6/30 192.168.0.0/24 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.193/require;
spdadd 192.168.0.0/24 169.254.255.6/30 any -P in ipsec
esp/tunnel/72.21.209.193-1.2.3.4/require;
The first part specifies to use tunnels between the customer VPN gateway and the two Amazon VPN endpoints for the internal IP addresses of the VPN. Please remember to change the 1.2.3.4 address with your actual address. The second part is necessary for specifying the policy of using the same tunnels when accessing the VPC subnet 192.168.0.0/24.
/etc/racoon/racoon.conf
The racoon.conf contains the cryptographic parameters for the specific IPSec tunnels, which are obtained from the configuration provided by Amazon.
remote 72.21.209.193 {
exchange_mode main;
lifetime time 28800 seconds;
proposal {
encryption_algorithm aes128;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
generate_policy off;
}
remote 72.21.209.225 {
exchange_mode main;
lifetime time 28800 seconds;
proposal {
encryption_algorithm aes128;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
generate_policy off;
}
sainfo address 169.254.255.2/30 any address 169.254.255.1/30 any {
pfs_group 2;
lifetime time 3600 seconds;
encryption_algorithm aes128;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}
sainfo address 169.254.255.6/30 any address 169.254.255.5/30 any {
pfs_group 2;
lifetime time 3600 seconds;
encryption_algorithm aes128;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}
Setup of Inside IP Addresses and Getting Started
The next thing is to set the inside IP addresses on the network interface:
ip a a 169.254.255.2/30 dev eth0
ip a a 169.254.255.6/30 dev eth0
Followed by starting the IPSec services and establish the tunnels:
/etc/init.d/racoon start
/etc/init.d/setkey start
ping 169.254.255.1
ping 169.254.255.5
The ping commands at the end establish the tunnels and it may take a while until ICMP responses are received. The command racoonctl show-sa ipsec is also helpful to watch the state of the IPSec tunnels.
Setup of BGP Routing
After setting up the IPSec tunnels, which are hopefully working now and the Amazon VPN gateways are reachable, we have to set up the BGP routing. I have used the Quagga software router: apt-get install quagga. In /etc/quagga/daemons you have to enable the zebra and bgpd daemons. In case you are running a packet filter, please allow the Amazon VPN gateways to connect to TCP port 179 for BGP.
/etc/quagga/bgpd.conf
hostname ec2-vpn
password testPassword
enable password testPassword
!
log file /var/log/quagga/bgpd
!debug bgp events
!debug bgp zebra
debug bgp updates
!
router bgp 65530
bgp router-id 1.2.3.4
network 169.254.255.2/30
network 169.254.255.6/30
network 0.0.0.0/0
!
! aws tunnel #1 neighbour
neighbor 169.254.255.1 remote-as 7224
!
! aws tunnel #2 neighbour
neighbor 169.254.255.5 remote-as 7224
!
line vty
The router-id should be changed to the external IP address of your customer VPN gateway. The passwords should be changed as well to your preferred value. The bgpd configuration file just specifies the Amazon VPN gateways as BGP neighbours and that our machine will provide routing to the default network 0.0.0.0/0.
/etc/quagga/zebra.conf
hostname ec2-vpn
password testPassword
enable password testPassword
!
! list interfaces
interface eth0
interface lo
!
line vty
Zebra is used for managing the routing table. You should change the passwords in this configuration file as well.
Getting Started
Just activate quagga by /etc/init.d/quagga start and verify the reception of the VPC subnet routing by the Amazon VPN gateways, e.g. by checking the log file /var/log/quagga/bgpd for rcvd and UPDATE entries of the appropriate VPC subnet. You can also connect to the bgpd management console using nc localhost 2605 and type show ip bgp summary for displaying BGP information.
Afterwards you can launch an instance in your VPC following this guide and try to ping the instance from your customer VPN gateway.
Accessing the Internet from VPC Instances
In case you are wondering why your launched VPC instance can not access the Internet, I have not implemented this yet. I suppose you only have to do NAT on your VPN gateway for the VPC subnet, but I haven’t tested that.
Troubleshooting
Please follow the Amazon Troubleshooting Guide in case something does not work. If this also does not resolve your issues, you may leave a comment with appropriate log files and I will try to help out.
AWS EC2 Java Library with Mock Objects
(Re-published as a separate post)
In this post I will write a bit about programming EC2 using the Java library provided by AWS. In particular I was interested in the Mock capabilities already included in the library and how to use them, but unfortunately the documentation was a bit sparse on this topic.
Mock Capabilities
Assume you want to do unit testing of an application using the AWS EC2 API or you are just in the middle of developing an application processing data from the API. In both cases you may want to use the mock capabilities included in the previously mentioned Java library, because it will reduce the time for running the tests or your application since no network communication is required. Instead, the mock mechanism will load AWS EC2 API responses from locally stored XML files and pretends to the library user that the response came from AWS. This example illustrates how to use the Mock functionality when using AmazonEC2 service = new AmazonEC2Mock();.
Obtaining Responses
The Mock responses are stored in src/com/amazonaws/ec2/mock with the file names represent the response object for the various API calls. For example, if we perform a DescribeInstances API call, the Mock response will be loaded from DescribeInstancesResponse.xml. The default content of these XML files will cause exceptions in the XML unmarshalling code, so do not be surprised. Unfortunately AWS do not provide reasonable responses, so we have to collect them ourselves.
Certain response objects can be converted to XML, but this output can not be used for the response XML files, because the response from the AWS API is transformed into the one we will have in the application. Therefore we have to obtain the raw response, which can be extracted by enabling DEBUG log output for the library (e.g. set rootLogger to DEBUG in src/log4j.properties and ant build dist the library) and execute regular API calls. For each API request you will see the AWS EC2 API response and the transformed response used by the application. Just copy the AWS API response XML into the appropriate Mock response file (someone should automate this approach).
After obtaining all relevant responses and storing them in the Mock files, your unit tests or application should run fine without exceptions when enabling the Mock service.
Older Posts