cynojure.aws – amazon web services from clojure

Thu, 11 Jun 2009

I've been using Amazon EC2, S3, and SQS services in various projects at work. For one project, I needed to build a little tool that integrated SQS and S3 services, and I decided to build it in Clojure. In the course of building the tool, I learned a bit about the various open source Java libraries for working with Amazon AWS, and the issues with integrating them with Clojure.

The libraries I used are:

• jets3t – S3 interface
• typica – SQS and EC2 interface
• Apache commons-httpclient and associated dependencies
• postgres jdbc driver

Having been away from Java development for some time now, I'd forgotten about the convoluted mess of Classpaths and Class loaders. I had a "crash" refresher course in the last week. And, it's all still a bloody mess. I did come across one-jar, which made it pretty easy to package up the various jar dependencies into a single deliverable jar file.

In the end, the tool worked out well and served its purpose. Plus, it was rewarding to be able to write code like this:

(defun process-loop [:key
		     aws-access-key
		     aws-secret-key
		     geo-db-path
		     [message-count 5]
		     [sleep 5000]
		     sqs-queue]
  (with-geo-service (geo-get-service :path geo-db-path)
    (with-sqs-queue (sqs-get-queue aws-access-key aws-secret-key sqs-queue)
      (with-connection *db*
	(let [process (fn [msgs]
			(if (empty? msgs)
			  (Thread/sleep sleep)
			  (try
			    (transaction
			     (doseq [msg msgs] (process-message msg)))
			    (doseq [msg msgs] (sqs-delete-message msg))
			    (catch java.sql.BatchUpdateException ex
			      (println "* process error:"
				       (.getNextException ex))))))]
	  (while true
	    (try
	      (process (sqs-receive-messages message-count))
	      (catch Exception ex
		(println "* error:" ex)))))))))

and know it's running on the JVM.

I've committed some of the code for the Amazon integration into the cynojure git repository, in case anyone else finds it useful.

Harmony.