Thursday, July 29, 2010

Amazon S3 IP Blocks

Today I need to help a customer that uses Amazon S3 for offsite backups (using duplicity). Up until now they had been accessing S3 through a squid proxy to get the data to and from S3. This enables them to limit the outgoing HTTP connections from their hosts.

Recently, they discovered that the performance of backups has been highly variable and restores are extremely slow. In testing today we found that performance through the proxy (with just duplicity) was much slower than it was going direct to S3. Of course, this now caused some problems with firewall configuration. In order to try to limit access, we need to find the IP addresses that are used by S3. Unfortunately, this does appear to be an easy task. Google was not immediately helpful.

I then started looking at DNS. It appears that *.s3.amazonaws.com goes through a number of CNAME records before arriving at an IP address. The last step alone the way appears to be one of three host names:

  1. s3-1-w.amazonaws.com.
  2. s3-2-w.amazonaws.com.
  3. s3-3-w.amazonaws.com.
It appears that each of these host names resolve to IP addresses in different blocks.  By running whois queries against an IP address in each block I was able to discover the following blocks:
  1. 72.21.192.0/19
  2. 87.238.80.0/21
  3. 207.171.160.0/19
For now, this is good enough...

-Bob