Creating an Amazon S3 Post Policy and uploading via Curl Http Post and ruby.

This post shows you how to do the following:
1. create an amazon file upload policy document to allow uploads by users of audio files visible to the world.
2. using this policy document create a policy signature that can be used by web clients to http post/upload an mp3 file.
3. a demo command line client that will upload a sample file to your amazon s3 bucket.

K first off we're gonna assume you set up an amazon bucket, have an access_key and secret_access_key, pretty easy to do just head on over to the AWS management web console and set it up.
We'll also assume you have some file named test.mp3 you want to upload to s3.


In this demo I'll be using:
bucket=dreamlogs
sub_folder=folder # this is the subfolder where files will be uploaded.
AMAZON_ACCESS_KEY_ID=your_key
AMAZON_SECRET_ACCESS_KEY=you_secret_key
filename=test.mp3



Now we need to create an amazon s3 post policy that determines who has access to upload files to your bucket. There are many different kinds of policies with amazon web services so in this case you need something like the following:

This site got me started but seems to be pretty out of date, the doc links are dead:
http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

Here's an example:

{
  "expiration": "repl_expiration",
  "conditions": [
    {"bucket": "repl_bucket" },
    {"acl": "repl_acl" },
    ["eq", "$key", "repl_foldername/repl_username/repl_filename"],
    ["content-length-range", 0, "repl_max_content_bytes"],
    ["starts-with", "$Content-Type", "repl_content_type"],
    {"redirect": "repl_s3url/repl_bucket/repl_foldername/repl_username/repl_filename"},
  ]
}

every variable starting with the word repl shoul be replaced with whatever permission you are trying to allow, here's what I put in my java .properties file.

BUCKET=bucketname
UPLOADDIR=somedir
CONTENT_TYPE=application/octet-stream
S3URL=http://s3.amazonaws.com
ACL=public-read
# example 20 megabytes would be 20 * 1048576 = 20971520 bytes
MAX_CONTENT_BYTES=20971520

Next up we need to remove carriage returns / line feeds then B64 encode this policy.
Finally sign it with our amazon secret key. The following is the main brain of my java web service its response is Json object with the policy and signature necessary for the client to then use to formulate a post file upload request to amazon s3.

    1 package my.aws.console.proj.web;
    2 
    3 import java.text.SimpleDateFormat;
    4 import java.util.Date;
    5 import java.util.TimeZone;
    6 
    7 import javax.management.timer.Timer;
    8 import javax.servlet.http.HttpServletRequest;
    9 
   10 import my.aws.console.proj.domain.PolicySignature;
   11 import my.aws.console.proj.tooling.PolicyGenerator;
   12 
   13 import org.apache.log4j.Logger;
   14 import org.springframework.roo.addon.web.mvc.controller.RooWebScaffold;
   15 import org.springframework.stereotype.Controller;
   16 import org.springframework.web.bind.annotation.RequestMapping;
   17 import org.springframework.web.bind.annotation.RequestMethod;
   18 import org.springframework.web.bind.annotation.RequestParam;
   19 import org.springframework.web.bind.annotation.ResponseBody;
   20 
   21 
   22 @RooWebScaffold(path = "policysignatures", formBackingObject = PolicySignature.class, update = false, delete = false, create = false)
   23 @RequestMapping("/policysignatures")
   24 @Controller
   25 public class PolicySignatureService {
   26     
   27     private static Logger logger = Logger.getLogger(PolicySignatureService.class);
   28     
   29     
   30     @RequestMapping(value = "/service", method = RequestMethod.GET, headers = "Accept=application/json")
   31     @ResponseBody
   32     public Object showJson(@RequestParam(value = "facebook_id", required = true) String facebook_id, 
   33                             @RequestParam(value = "filename", required = true) String filename, 
   34                             HttpServletRequest request) {
   35         PolicyGenerator generator = new PolicyGenerator();
   36         String formattedExpirationDate = getFormattedExpirationDate();      
   37         String generatedPolicy = makeAWSPostPolicy(facebook_id, filename, generator, formattedExpirationDate);
   38         String b64EncodedPolicy = generator.b64EncodePolicy(generatedPolicy);
   39         String signature = generator.signPolicy(generator.AWSSECRETKEY, b64EncodedPolicy);
   40         PolicySignature policysignature = makePolicySignature(facebook_id,
   41                 filename, generator, b64EncodedPolicy, signature);
   42         logger.info(request.getRemoteAddr()+","+ generatedPolicy +","+ policysignature.toJson());
   43         return policysignature.toJson();
   44     }
   45 
   46 
   47     public PolicySignature makePolicySignature(String facebook_id,
   48             String filename, PolicyGenerator generator, String b64EncodedPolicy,
   49             String signature) {
   50         PolicySignature policysignature = new PolicySignature();
   51         policysignature.setPolicy(b64EncodedPolicy);
   52         policysignature.setSignature(signature);
   53         policysignature.setAWSAccessKeyId(generator.AWSKEYID);
   54         policysignature.setBucket(generator.BUCKET);
   55         policysignature.setFkey(generator.UPLOADDIR+"/"+facebook_id+"/"+filename);
   56         policysignature.setContent_type(generator.CONTENT_TYPE);
   57         policysignature.setFile("@" + filename);
   58         policysignature.setS3url(generator.S3URL + "/" + generator.BUCKET);
   59         policysignature.setRedirect(generator.S3URL + "/" + generator.BUCKET + "/" + generator.UPLOADDIR + "/" + facebook_id + "/" + filename);
   60         policysignature.setAcl(generator.ACL);
   61         return policysignature;
   62     }
   63 
   64 
   65     public String makeAWSPostPolicy(String facebook_id, String filename,
   66             PolicyGenerator generator, String formattedExpirationDate) {
   67         String generatedPolicy = generator.generatePolicy(generator.AWSKEYID, generator.AWSSECRETKEY);      
   68         generatedPolicy = generatedPolicy.replaceAll("repl_expiration", formattedExpirationDate);
   69         generatedPolicy = generatedPolicy.replaceAll("repl_bucket", generator.BUCKET);
   70         generatedPolicy = generatedPolicy.replaceAll("repl_acl", generator.ACL);
   71         generatedPolicy = generatedPolicy.replaceAll("repl_filename", filename);
   72         generatedPolicy = generatedPolicy.replaceAll("repl_content_type", generator.CONTENT_TYPE);
   73         generatedPolicy = generatedPolicy.replaceAll("repl_max_content_bytes", generator.MAX_CONTENT_BYTES);
   74         generatedPolicy = generatedPolicy.replaceAll("repl_foldername", generator.UPLOADDIR);
   75         generatedPolicy = generatedPolicy.replaceAll("repl_username", facebook_id);
   76         generatedPolicy = generatedPolicy.replaceAll("repl_s3url", generator.S3URL);
   77         generatedPolicy = generatedPolicy.replaceAll("\r", "");
   78         generatedPolicy = generatedPolicy.replaceAll("\n", "");
   79         return generatedPolicy;
   80     }
   81 
   82 
   83     public String getFormattedExpirationDate() {
   84         Date now = new Date();
   85         Date oneHourFromNow = new Date(now.getTime() + Timer.ONE_HOUR);
   86         TimeZone tz = TimeZone.getTimeZone( "UTC" );
   87         SimpleDateFormat dfm = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
   88         dfm.setTimeZone( tz );
   89         String formattedExpirationDate = dfm.format(oneHourFromNow);
   90         return formattedExpirationDate;
   91     }    
   92 }
   93 

The signing code and base 64 code was modeled after this security doc: http://docs.amazonwebservices.com/AWSSimpleQueueService/2007-05-01/SQSDeveloperGuide/SummaryOfAuthentication.html

package my.aws.console.proj.tooling;

import java.io.IOException;
import java.io.InputStream;
import java.security.SignatureException;
import java.util.Properties;



public class PolicyGenerator {

    public String BUCKET;
    public String DEFAULT_AMAZON_POLICY;
    public String AWSKEYID;
    public String UPLOADDIR;
    public String CONTENT_TYPE;
    public String S3URL;
    public String AWSSECRETKEY;
    public String ACL;
    public String MAX_CONTENT_BYTES;
    
    public PolicyGenerator() {
        InputStream policyInputStream = PolicyGenerator.class.getClassLoader().getResourceAsStream("permissive.json");
        InputStream bucketingInputStream = PolicyGenerator.class.getClassLoader().getResourceAsStream("amazon_bucketing.properties");
        InputStream credentialsInputStream = PolicyGenerator.class.getClassLoader().getResourceAsStream("AwsCredentials.properties");

        byte[] bytes = null;
        try {
            bytes = new byte[policyInputStream.available()];
            policyInputStream.read(bytes);
            DEFAULT_AMAZON_POLICY = new String(bytes);
            
            Properties awsprops = new Properties();
            awsprops.load(bucketingInputStream);
            BUCKET = awsprops.getProperty("BUCKET");
            UPLOADDIR = awsprops.getProperty("UPLOADDIR");
            CONTENT_TYPE = awsprops.getProperty("CONTENT_TYPE");
            S3URL = awsprops.getProperty("S3URL");
            ACL = awsprops.getProperty("ACL");
            MAX_CONTENT_BYTES = awsprops.getProperty("MAX_CONTENT_BYTES");
            
            Properties awscreds = new Properties();
            awscreds.load(credentialsInputStream);
            AWSKEYID = awscreds.getProperty("accessKey");
            AWSSECRETKEY = awscreds.getProperty("secretKey");

            
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public String generatePolicy(String awsid, String awskey) {
        return DEFAULT_AMAZON_POLICY;
    }
    
    public String signPolicy(String awskey, String policy){
        try {
            return  Signature.calculateRFC2104HMAC(policy, awskey);
        } catch (SignatureException e) {
            e.printStackTrace();
        }
        return "error_signing_policy";
    }

    
    public String b64EncodePolicy(String policy){
        //policy = policy.replaceAll("\n", ""); #this breaks it for some unknown reason
        policy = policy.replaceAll("\r", "");
        return Encoding.EncodeBase64(policy.getBytes());
    }
    
    
    public String readJsonFile(String policyFile) {
        InputStream inputStream = PolicyGenerator.class.getClassLoader()
                .getResourceAsStream(policyFile);
        byte[] bytes = null;
        try {
            bytes = new byte[inputStream.available()];
            inputStream.read(bytes);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return new String(bytes);
    }
    
}

As a test HTTP client we'll be using curl driven by ruby, its a pretty simple script you just have to set all the right curl flags and also worry about the order as amazon services are picky about that.

# this test will hit a server and then using the response upload a file to s3.
#
require 'json'
server = "http://192.168.254.6"
port = "8080"
facebook_user = "malcruzito"
filename= "test.mp3" #assumes this file exists in the directory you run the script from

#hit the server to get the permissions to upload
resp = `curl -H Accept:application/json "#{server}:#{port}/proj/policysignatures/service?facebook_id=#{facebook_user}&filename=#{filename}"`

parsed = JSON.parse(resp)

#puts parsed

#upload the file
curl_command = "curl -L -v -b cookiejar.txt -c cookiejar.txt "
curl_command += " -F \"policy=#{parsed['policy']}\""
curl_command += " -F \"acl=#{parsed['acl']}\""
curl_command += " -F \"redirect=#{parsed['redirect']}\""
curl_command += " -F \"signature=#{parsed['signature']}\""
curl_command += " -F \"AWSAccessKeyId=#{parsed['AWSAccessKeyId']}\""
curl_command += " -F \"bucket=#{parsed['bucket']}\""
curl_command += " -F \"key=#{parsed['fkey']}\""
curl_command += " -F \"content-type=#{parsed['content_type']}\""
curl_command += " -F \"file=#{parsed['file']}\""
curl_command += " #{parsed['s3url']}"

print curl_command +"\n"
result = `#{curl_command}`
p result