One of the requirement we had was of sending thousands of email newsletters. Without a second thought we decided to use Amazon SES for sending the emails, because as with any of the cloud service there is no initial cost and we need to pay on a need basis with no commitment. Also, the documentation around AWS (Amazon Web Services) is really good and the AWS Eclipse Toolkit was useful to get started.
Many of the emails (in thousands) in our email database were invalid which resulted in a bounce or a complaint. AWS SES handles these invalid mails by sending individual email or/and notifications through AWS SNS. Here is flow for the same.
Initially we choose the notification by email. At the end, we had thousands of emails back for both bounced and complaints and had to extract the individual email address from those mails to clean our email database. This was a cumbersome task. So, we choose to alternate route of sending the bounced and complaint email addresses to SNS and then forward them to AWS SQS. Then a Java program using the AWS SDK for Java pulled the messages from the Queue.
Every thing was nice and good till now. But, when AmazonSQS.receiveMessage(ReceiveMessageRequest receiveMessageRequest) was called on the Queue only a single message was returned inspite of having around a thousand messages in the queue. The below is the probable reasoning for it from the ReceiveMessage documentation.
>> Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
To make life easy, one of the option was to invoke the ReceiveMessageRequest.setMaxNumberOfMessages(Integer maxNumberOfMessages). But, this was also returning a maximum of 10 messages at a time. An exception is thrown when the maximum number of messages is set to more than 10. We stopped automating further, by repeatedly calling AmazonSQS.receiveMessage() till the number of messages in the queue reaches zero. Checking the number of messages in the Queue can be done by calling the GetQueueAttributesResult.getAttributes() method, but this is again not returning a consistent value because of the distributed nature of the queue.
Amazon Web Services is a awesome service and I am really impressed by how quickly someone can get started, but there are small quirks here and there which need to be addressed. The purpose of this blog entry is to help those who are getting started with AWS for sending emails.
For those who are getting started with AWS, Amazon offers a free usage tier. The only caution is to stop the services when not needed, so that the usage doesn't cross the free limit.
Many of the emails (in thousands) in our email database were invalid which resulted in a bounce or a complaint. AWS SES handles these invalid mails by sending individual email or/and notifications through AWS SNS. Here is flow for the same.
** Courtesy Amazon |
Initially we choose the notification by email. At the end, we had thousands of emails back for both bounced and complaints and had to extract the individual email address from those mails to clean our email database. This was a cumbersome task. So, we choose to alternate route of sending the bounced and complaint email addresses to SNS and then forward them to AWS SQS. Then a Java program using the AWS SDK for Java pulled the messages from the Queue.
Every thing was nice and good till now. But, when AmazonSQS.receiveMessage(ReceiveMessageRequest receiveMessageRequest) was called on the Queue only a single message was returned inspite of having around a thousand messages in the queue. The below is the probable reasoning for it from the ReceiveMessage documentation.
>> Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
To make life easy, one of the option was to invoke the ReceiveMessageRequest.setMaxNumberOfMessages(Integer maxNumberOfMessages). But, this was also returning a maximum of 10 messages at a time. An exception is thrown when the maximum number of messages is set to more than 10. We stopped automating further, by repeatedly calling AmazonSQS.receiveMessage() till the number of messages in the queue reaches zero. Checking the number of messages in the Queue can be done by calling the GetQueueAttributesResult.getAttributes() method, but this is again not returning a consistent value because of the distributed nature of the queue.
Amazon Web Services is a awesome service and I am really impressed by how quickly someone can get started, but there are small quirks here and there which need to be addressed. The purpose of this blog entry is to help those who are getting started with AWS for sending emails.
For those who are getting started with AWS, Amazon offers a free usage tier. The only caution is to stop the services when not needed, so that the usage doesn't cross the free limit.
No comments:
Post a Comment