XenStore Dump of Rackspace Cloud
For the curious: A dump of the XenStore config for a VM running on RackSpace Cloud. Actually quite a straight forward and simple setup: using image files for both root and swap disks, and a bridged network setup for the two interfaces (one for internal, one for external). Although the network is bridged, the VM is not able obtain traffic for other VMs as they are using VLANs for separation.
Notes on Amazon EC2's Network
Shortly after publishing my notes on the EC2 architecture, I was looking into the networking setup of EC2 and in particular figuring out their address schemes. Since I am currently no longer interested in such information, I will publish my incomplete notes and the raw data gathered from about 80 instances in this post. My notes are based on information obtained from small instances in the us-east-1d zone.
IP Address Scheme
I assumed the first hop in the traceroute from a VM is the actual dom0 IP address.
Consider the private IP addresses in the form 10.X.Y.Z. I have noticed that Y is partitioned into blocks containing a /24 for dom0 IP addresses, a /24 for VMs, and a /23 for another set of VMs. For example: 10.208.176/24 is the dom0 range; 10.208.177/24 the first VM range; 10.208.178/23 the second VM range.
Based on my data, the dom0 IP addresses always end in .2 or .3, but there seems to be no pattern between a VM’s IP address and the ending of the corresponding dom0.
MAC Address Scheme
I do not have many information on this one. MAC addresses are typically in the form of 12:31:39:X:Y:Z, where X can be derived from the second octet of the private IP address. The following list gives the value of X for the second IP address octet. As an example: IP 10.210.X.Y leads to 12:31:39:09:X’:Y’, because 09 is listed for octet 210.
00 254
01 255
02 248
03 249
04 240
05 241
06 208
07 209
09 210
0A 211
0B 214
0C 215
Raw Data
The raw data can be found here. It contains network configuration information (ifconfig, traceroute, and routes) of about 80 instances from the us-east-1d zone. Let me know if you make any interesting discoveries based on that data.
Another Short Wish List for Amazon Web Services
In continuation of my first wish list, I will present another set of potential improvements and ideas for Amazon Web Services in this post. The focus lies on security groups and virtual private cloud.
Security Group: Outbound Filtering
Many people have written about potential improvements for Amazon security groups, see for example Dmitriy’s write-up, and I just want to emphasis one point from their wish lists: the possibility to filter outbound traffic. I understand that Amazon is quite pragmatic with their feature set and only provides what the majority of their customers require. For the majority the current situation with filtering inbound traffic might be good enough, but with an increase in cloud usage for “serious” purposes, filtering outbound traffic will be of importance. And honestly, I doubt the technical difficulty in implementing that feature when they are already able to filter inbound traffic.
AMI Security Group Membership Policies
AMIs can contain services which might not have a strong authentication and authorization mechanism, for example consider memcached. The problem is to ensure that instances of such an AMI are properly secured with rules of an appropriate security group, e.g., only allow web/application server to access the memcached. An AMI could be accompanied by a policy specifying the properties of a security group the instances of that AMI can only be a member of.
Amazon Simple Logging Service
Logging is a crucial part of IT infrastructure and I am surprised that Amazon has not released such a service, although the idea was already proposed in a post in the Amazon forum.
For one thing, they could publish logs of the underlying architecture relevant for specific VMs, e.g., logs from the security group filtering. On another thing, they could provide a syslogd interface to VMs in order to gather logs from customer instances. The aggregated logs can then either be processed by Amazon (for example running a simple logcheck) or further processing can be done by third-party applications (for example an IDS).
Amazon Virtual Private Cloud
I know that Amazon VPC is still considered a service in beta phase, and I won’t pick on the limit of one VPC per customer, but still I want to share two points.
Subnet Filtering
A VPC subnet can further be divided into multiple subnets (up to 20), which are aggregated in a star topology with a router in the middle. I am looking forward in having access to filtering capabilities of that a router, in order to restrict traffic between the various subnets. This would greatly enhance the possibilities of VPC subnets.
Security Groups
In case subnet filtering on the router would not be possible, having security groups in combination with VPC would be great. A VPC subnet could just be considered as a security group, which allows regular filtering between the various subnets using security groups as sources. It would also allow filtering for the individual instances in the management layer, rather than doing filtering within the VMs.
How Do You Use Amazon's Security Groups?
This time I would like to reverse the flow of information and would kindly ask you to fill out the following questionnaire about how you use the Security Groups of Amazon Web Services. It will only take a few minutes of your time.
Thanks for your participation and I welcome any form of feedback.
My Short Wish List for Amazon Web Services
In the short time I have used AWS, I came across a few issues which I would like to see changed by Amazon in the foreseeable future. It mainly concerns the management side of AWS rather than the technical one.
Fine-Grained Access Control for the API
Cloud computing is destined to be used by applications and automation infrastructure, therefore the API is an essential component in the cloud provider’s offering. So far I am pleased with the functionality of the existing API, but the access control is very rigid and inflexible. Although Amazon supports multiple types of credentials and allows multiple instances of each credential, the access associated with these can not be controlled.
If a credential is getting compromised, the attacker would get full control over ones IT infrastructure in the cloud and could use additional resources, e.g. for further malicious purposes, which are billed by the victim’s account. A typical scenario is an application running on a dedicated server using additional resources from the cloud, therefore requires the access key, or another kind of credential, to be stored on the server. If the server is getting compromised, the access key will be compromised as well and the attacker can use it for the previously mentioned cases.
Specifying in a fine-grained way for which purposes a credential can be used would limit the impact of a compromised credential. This can be on a high-level at the beginning, e.g. specifying a credential can be used for write or read access. I define write access as operations modifying the cloud, e.g. creating new instances or shutting down instances, and read access as ones only obtaining information. The latter type of access is useful for monitoring applications, where access for modifying the cloud is generally not required. In the long-term, this could be extended to limit credentials for certain kind of operations, only allowing credentials to operate on specific services, or limit the issuer of operations using a credentials (e.g. only the operations issued from the corporate network are allowed).
Notifications of Resources Usage
AWS provides usage information for the resources in the cloud in a XML format, but does not have a functionality of sending notifications for what kind of resources were used and how much they cost in a specific interval. For example I would like to receive a daily email summarizing the resources I have used and how much they cost for that specific day. I am sure one could implement such a feature with the provided usage information, but it would be much more convenient if it is provided by Amazon out of the box. Especially since the session lifetime for viewing the account activity is very short.
Resource Usage Limitations
Consider the scenario that you want your developer team to use the cloud for testing or development purposes, but you also want to limit the amount of money they are allowed to spend. If one of the developers forgets to terminate his 20 extra large instances for a longer time-frame, it will be charged at the end of the month on your credit card. Of course the previously mentioned notifications would probably prevent such a scenario, but it also makes sense to have a soft and hard limit on the resource usage, in case one is not interested in daily usage notifications.
For example you allow your developers to spend $500 on EC2, and after that limit is reached they are not able to launch new instances. Furthermore you specify $550 as the hard limit, i.e. after that amount is reached all running instances are terminated.
Of course these kind of limitations can be tricky in certain situations, e.g. when dealing with storage. Are you deleting storage if the limit is reached or not? The combination of hard and soft limits, and notifications could maybe make it feasible.
Outlook
I am also very interested in what kind of problems you are having with AWS and what you would like to see changed. I am in particular interested in security concerns and issues.
Amazon VPC with Linux
This post will cover how to connect a Amazon Virtual Private Cloud (VPC) with a Linux-based VPN gateway. In Amazon’s Network Guide for setting up a gateway connecting a VPC with existing infrastructure, they only provide specific configurations for Juniper and Cisco devices. Other devices, like a Linux-based VPN gateway, are covered by their documentation for Generic Devices, but lack specific configuration files. In this post I will explain how to setup a VPN gateway using racoon, ipsec-tools, and quagga on Ubuntu Linux.
Setup a VPC
The first thing to do is to follow the VPC tutorial and setting up all the VPC objects, like vpn-gateway, vpc, subnet etc. When specifying the customer VPN gateway, you can assume a private BGP ASN like 65530, which we will use later in the example configuration files. Just follow the steps in the tutorial and skip the part of configuring the customer VPN gateway, which we will do afterwards. Having completely configured the Amazon side of the VPN, we can immediately test the configuration of our customer VPN gateway.
Setup of IPSec
First, you have to install the necessary packages for running a VPN gateway using IPSec: apt-get install ipsec-tools racoon. In case you are running a packet filter on the VPN machine, please allow UDP port 500 for racoon/IKE.
When creating a VPN connection in the Setup a VPC section, Amazon will provide you with all the required configuration parameter for setting up the IPSec tunnel. Of particular interest are the Pre-Shared Keys and the IP addresses, which can look like the following:
Outside IP Addresses:
- Customer Gateway: : 1.2.3.4
- VPN Gateway : 72.21.209.225
Inside IP Addresses
- Customer Gateway : 169.254.255.2/30
- VPN Gateway : 169.254.255.1/30
Outside IP Addresses:
- Customer Gateway: : 1.2.3.4
- VPN Gateway : 72.21.209.193
Inside IP Addresses
- Customer Gateway : 169.254.255.6/30
- VPN Gateway : 169.254.255.5/30
We pretend that the external IP address of our customer VPN gateway is 1.2.3.4, but you will have to replace that value in the following configuration files with your actual address.
Update: The IP addresses of the VPN gateways can be in the opposite order, i.e. 72.21.209.193 for the address range 169.254.255.1/30. If this is the case, the ipsec-tools.conf have to be adjusted accordingly.
/etc/racoon/psk.txt
This file holds the pre-shared keys for the two IPSec tunnels:
72.21.209.225 password1
72.21.209.193 password2
/etc/ipsec-tools.conf
In this file we specify the IPSec policies for the two tunnels:
flush;
spdflush;
spdadd 169.254.255.2/30 169.254.255.1/30 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.225/require;
spdadd 169.254.255.1/30 169.254.255.2/30 any -P in ipsec
esp/tunnel/72.21.209.225-1.2.3.4/require;
spdadd 169.254.255.6/30 169.254.255.5/30 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.193/require;
spdadd 169.254.255.5/30 169.254.255.6/30 any -P in ipsec
esp/tunnel/72.21.209.193-1.2.3.4/require;
spdadd 169.254.255.2/30 192.168.0.0/24 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.225/require;
spdadd 192.168.0.0/24 169.254.255.2/30 any -P in ipsec
esp/tunnel/72.21.209.225-1.2.3.4/require;
spdadd 169.254.255.6/30 192.168.0.0/24 any -P out ipsec
esp/tunnel/1.2.3.4-72.21.209.193/require;
spdadd 192.168.0.0/24 169.254.255.6/30 any -P in ipsec
esp/tunnel/72.21.209.193-1.2.3.4/require;
The first part specifies to use tunnels between the customer VPN gateway and the two Amazon VPN endpoints for the internal IP addresses of the VPN. Please remember to change the 1.2.3.4 address with your actual address. The second part is necessary for specifying the policy of using the same tunnels when accessing the VPC subnet 192.168.0.0/24.
/etc/racoon/racoon.conf
The racoon.conf contains the cryptographic parameters for the specific IPSec tunnels, which are obtained from the configuration provided by Amazon.
remote 72.21.209.193 {
exchange_mode main;
lifetime time 28800 seconds;
proposal {
encryption_algorithm aes128;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
generate_policy off;
}
remote 72.21.209.225 {
exchange_mode main;
lifetime time 28800 seconds;
proposal {
encryption_algorithm aes128;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
generate_policy off;
}
sainfo address 169.254.255.2/30 any address 169.254.255.1/30 any {
pfs_group 2;
lifetime time 3600 seconds;
encryption_algorithm aes128;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}
sainfo address 169.254.255.6/30 any address 169.254.255.5/30 any {
pfs_group 2;
lifetime time 3600 seconds;
encryption_algorithm aes128;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}
Setup of Inside IP Addresses and Getting Started
The next thing is to set the inside IP addresses on the network interface:
ip a a 169.254.255.2/30 dev eth0
ip a a 169.254.255.6/30 dev eth0
Followed by starting the IPSec services and establish the tunnels:
/etc/init.d/racoon start
/etc/init.d/setkey start
ping 169.254.255.1
ping 169.254.255.5
The ping commands at the end establish the tunnels and it may take a while until ICMP responses are received. The command racoonctl show-sa ipsec is also helpful to watch the state of the IPSec tunnels.
Setup of BGP Routing
After setting up the IPSec tunnels, which are hopefully working now and the Amazon VPN gateways are reachable, we have to set up the BGP routing. I have used the Quagga software router: apt-get install quagga. In /etc/quagga/daemons you have to enable the zebra and bgpd daemons. In case you are running a packet filter, please allow the Amazon VPN gateways to connect to TCP port 179 for BGP.
/etc/quagga/bgpd.conf
hostname ec2-vpn
password testPassword
enable password testPassword
!
log file /var/log/quagga/bgpd
!debug bgp events
!debug bgp zebra
debug bgp updates
!
router bgp 65530
bgp router-id 1.2.3.4
network 169.254.255.2/30
network 169.254.255.6/30
network 0.0.0.0/0
!
! aws tunnel #1 neighbour
neighbor 169.254.255.1 remote-as 7224
!
! aws tunnel #2 neighbour
neighbor 169.254.255.5 remote-as 7224
!
line vty
The router-id should be changed to the external IP address of your customer VPN gateway. The passwords should be changed as well to your preferred value. The bgpd configuration file just specifies the Amazon VPN gateways as BGP neighbours and that our machine will provide routing to the default network 0.0.0.0/0.
/etc/quagga/zebra.conf
hostname ec2-vpn
password testPassword
enable password testPassword
!
! list interfaces
interface eth0
interface lo
!
line vty
Zebra is used for managing the routing table. You should change the passwords in this configuration file as well.
Getting Started
Just activate quagga by /etc/init.d/quagga start and verify the reception of the VPC subnet routing by the Amazon VPN gateways, e.g. by checking the log file /var/log/quagga/bgpd for rcvd and UPDATE entries of the appropriate VPC subnet. You can also connect to the bgpd management console using nc localhost 2605 and type show ip bgp summary for displaying BGP information.
Afterwards you can launch an instance in your VPC following this guide and try to ping the instance from your customer VPN gateway.
Accessing the Internet from VPC Instances
In case you are wondering why your launched VPC instance can not access the Internet, I have not implemented this yet. I suppose you only have to do NAT on your VPN gateway for the VPC subnet, but I haven’t tested that.
Troubleshooting
Please follow the Amazon Troubleshooting Guide in case something does not work. If this also does not resolve your issues, you may leave a comment with appropriate log files and I will try to help out.
AWS EC2 Java Library with Mock Objects
(Re-published as a separate post)
In this post I will write a bit about programming EC2 using the Java library provided by AWS. In particular I was interested in the Mock capabilities already included in the library and how to use them, but unfortunately the documentation was a bit sparse on this topic.
Mock Capabilities
Assume you want to do unit testing of an application using the AWS EC2 API or you are just in the middle of developing an application processing data from the API. In both cases you may want to use the mock capabilities included in the previously mentioned Java library, because it will reduce the time for running the tests or your application since no network communication is required. Instead, the mock mechanism will load AWS EC2 API responses from locally stored XML files and pretends to the library user that the response came from AWS. This example illustrates how to use the Mock functionality when using AmazonEC2 service = new AmazonEC2Mock();.
Obtaining Responses
The Mock responses are stored in src/com/amazonaws/ec2/mock with the file names represent the response object for the various API calls. For example, if we perform a DescribeInstances API call, the Mock response will be loaded from DescribeInstancesResponse.xml. The default content of these XML files will cause exceptions in the XML unmarshalling code, so do not be surprised. Unfortunately AWS do not provide reasonable responses, so we have to collect them ourselves.
Certain response objects can be converted to XML, but this output can not be used for the response XML files, because the response from the AWS API is transformed into the one we will have in the application. Therefore we have to obtain the raw response, which can be extracted by enabling DEBUG log output for the library (e.g. set rootLogger to DEBUG in src/log4j.properties and ant build dist the library) and execute regular API calls. For each API request you will see the AWS EC2 API response and the transformed response used by the application. Just copy the AWS API response XML into the appropriate Mock response file (someone should automate this approach).
After obtaining all relevant responses and storing them in the Mock files, your unit tests or application should run fine without exceptions when enabling the Mock service.
AWS EC2 SSH Config
In case you are connecting to various EC2 instances frequently for testing purposes, the following SSH config might be useful for you:
host *.amazonaws.com
User root
StrictHostKeyChecking no
IdentityFile ~/.ssh/id_rsa-ec2
UserKnownHostsFile ~/.ssh/ec2_hosts
Always connect as user root to amazonaws.com boxes, do not ask for accepting the host’s key, and use the ec2 private key for authentication.
Update: It’s also useful to define a separate known hosts file, so your regular one does not get cluttered with host keys from AWS test systems.
Jitter 2010-02-04
12:53: If you have enjoyed my post about EC2’s architecture, have a look on Guy Rosen’s latest post, where he digged deeper into the resource IDs and confirmed some interesting theories.
On Amazon EC2's Underlying Architecture
..or “Amazon EC2 Demystified”, but that sounded way too cheesy.
Many people know that Amazon Web Services are one of the big players in the cloud computing business, and especially their Infrastructure as a Service offering EC2 is becoming increasingly popular. Few people know that EC2 is probably one of the biggest Xen installations deployed. But how many know how EC2 actually works and how the underlying architecture is constructed? I was curious and needed that kind of insight for my Master’s thesis, which deals with EC2 from a security perspective. The following notes were gathered out of plain curiosity and for academic purposes only. The notes are not complete and a lot of guessing is involved, so I might be wrong (please leave a comment if you think so).
Hypervisor & Dom0
As I said Amazon EC2 is probably one of the biggest Xen installations deployed. It is said that Amazon uses a heavily modified and adapted version of Xen, but unfortunately I was not able to gather information about exact version numbers. Dom0, the Xen management domain, can be either be based on Linux, NetBSD, or OpenSolaris. Based on the information I have gathered for the storage setup, I am very certain that it is Linux based. I do not know the version number of the used kernel. Amazon seems to be fond of RedHat Linux, so the Dom0 might be a RedHat Linux.
Storage
Amazon EC2 uses two different kinds of storage. One is local storage, known as Instance Storage, which is non-persistent and data will be lost after an instance terminates. The other kind is persistent, network-based storage called Elastic Block Store(EBS), which can be attached to running instances or also used as a persistent boot medium. I have excluded Amazon S3 from here. The information in this section were gathered from XenStore, which holds configuration information about all domains. A domain can read its own configuration information from XenStore using xenstore-ls, which is part of the xen utils.
Instance Storage
Instance Storage appears as 3 partitions to the VM: sda1 for root, sda2 for extra storage space (/mnt), and sda3 for swap. Typically the backend of these virtual block devices are based on loopback devices and/or LVM logical volumes. Logical volumes are considered to have a better performance and reliability compared to loopback devices. Therefore I was surprised that sda1 is using a loopback backend as noted as node = ”/dev/loop13” in the XenStore VBD entry. The actual file used by the loopback device is params = ”/mnt/instance_image_store_3/262768”. The suffix “_3” of the directory is probably based on the local domain id. The numerical filename of the image is not the same as the AMI, which is surprising. I also do not know if /mnt/instance_image_store_3 is actually a locally stored directory or if the images are mounted via e.g. NFS. The latter would make sense, because they only need to create a copy of the AMI image on the image storage server, maybe even using copy-on-write, and do not have to transfer the image over the network to the node.
The swap device is a LVM logical volume denoted as params = ”/dev/VolGroupDomU/instance_swap_store_3” in XenStore. The extra storage space /mnt is also using a logical volume backend: params = ”/dev/mapper/cow-VolGroupDomU-instance_ephemeral_store_3”. An interesting aspect is that they seem to use copy-on-write (cow) for this volume, but I don’t know why.
Elastic Block Store
The characteristics of the Elastic Block Store (EBS) lead to the conclusion that it is probably a SAN-based setup. I have guessed that they probably use iSCSI volumes exported to the nodes, and I was surprised that they use something different: Global Network Block Device (GNBD). The backend device of a EBS is listed as something like params = ”/dev/gnbd89”. Of course I don’t know how the block device exporter is designed and what kind of storage system they use on that side. Some more information about GNBD can be found here, which could also lead to the conclusion that Amazon uses RedHat’s cluster suite.
Networking
The networking setup of EC2 is quite unorthodox and I haven’t figured it out completely yet. Amazon uses a routed Xen network setup with DHCP providing private IP addresses to the VMs. A traceroute will show the private IP address of the router in Dom0, as well as the external IP address of Dom0. Furthermore, the network setup script is named script = ”/etc/xen/scripts/ec2-vif-route-dhcpd” in XenStore. A VM only has one interface with a private IP address and we have to assume that EC2 uses NAT to translate the external IP address to the internal one.
On L2, they also seem to use NAT, because the MAC address of all incoming and outgoing packets is EF:FF:FF:FF:FF:FF. They are also preventing IP spoofing and ARP poisoning, which suggests that they do filtering on each virtual interface in Dom0 based on the L2/L3 address of that particular VM. Furthermore, the Security Groups of EC2 are probably also implemented in a similar way. I would not be surprised if Amazon uses ebtables and iptables in their Dom0.
Domain Naming
Via XenStore one can also determine the name of a VM, which is something like domain = “dom_32504936” in Amazon EC2. Based on a limited sample of domain names, it seems that the suffix number is incremental. Assuming that the domain name is unique throughout the entire lifetime of EC2, it would mean that Amazon has started 32.5 million VMs in EC2. The difference between the domain name suffixes of two instances started 24 hours apart was around 82000, which could lead to the conclusion that in that time period around 82000 VMs were started (assuming the suffix is actually incremental). It would be interesting to monitor the domain name suffixes and thereby tracking the utilization of EC2 over time, e.g. how many VMs are started in specific time frames.
Outlook
So far I am satisfied with the information about the storage setup, but I will definitely need to get a better understanding of the networking. I will keep this post updated when I will gain new insights. If anyone has more information about any components, please leave a comment.
Older Posts