Monday, January 28, 2013

When to use Gmail's SMTP in your app and when not



This post is about the very legitimate automated emails generated by any application, such as for transactions and signups. Getting these delivered to the inbox (instead of spambox or nirvana) is the goal. 

The short answer: 

Sooner or later you may hit the sending limit. It's not when you deploy, and maybe not while beta testing. So better know the limits, risks and alternatives.

The longer answer:

If your sender address is hosted on Gmail then using their SMTP server is the obvious choice because:
  • You already have access to it
  • High availability
  • Unlikely that Gmail's smtp servers get added to block lists such as this one

How to connect to Gmail SMTP

All you need is the following information, and an SMTP library for your programming language:
  • host: smtp.gmail.com
  • port: 465
  • ssl: yes
  • user:
  • password:
The email address you use to log in must be a real account, not an alias. If you prefer to send the address from a different address, use the "from" field. Not modifying the "from" and "reply-to" increases your chances of getting delivered to the inbox.

You test it, email delivers, so you deploy your app, done.

Errors to expect

Soon after, your beta testers report to not receive any email. Checking your logs finds errors such as "550 5.4.5 Daily sending quota exceeded." or "535, response: 5.7.1 Please log in with your web browser and then try again.". What happened?

3 types of Gmail accounts

There are three types of Gmail addresses:
  1. The common Gmail domains: anything @ gmail
  2. Google Apps for free: yourname @ yourdomain
  3. Google Apps pro: yourname @ yourdomain
If you go and sign up for a new Gmail address from your desktop, then try sending mail from your (remote) server location, you'll get the 535 error quickly (it's the typical spammer pattern). You need to verify your account by SMS, and mailing goes on. For a short moment. It appears that such accounts can only send to a handful of different email addresses per day. I was not able to find official statements and numbers. The number is probably so low for new accounts, so if you have an established one it may work longer.

If you have your own domain set up for Gmail then the limit is higher. It makes sense since you have a public whois record. Google disabled signups for the free apps service a couple weeks ago. That's probably why I cannot find official information about the limits. The number of recipients per day is quoted as 500 on the internet. If you have such an account already then you can continue using it.

For the paid account the official page says 2000 unique, external recipients per day. 

Other risks

The official page has another fact:
"The value of these limits may change without notice in order to protect Google’s infrastructure"
Also, I've found unofficial/unverified information about Google lowering the daily send limit on high bounce rates. This makes perfect sense; spammers have high bounce rates. This is an open door to malicious users of your app: sign up with a couple invalid addresses, and your email system may be interrupted for a while.


Using your own SMTP

If you decide now that Gmail SMTP is not for you, there are some things to consider with your own.

If you access the SMTP of your provider, then you may face similar limits there. After all your provider has to make sure their customers don't spam, and not the whole server gets blacklisted. But this can and does happen nevertheless: Either because one of the other users spammed, or because one account was hijacked and abused. As a result your email messages may be accepted by the SMTP, but never make it to their destination. Maybe you get bounces, maybe not.

Be sure to create a Sender Policy Framework record.

Conclusions

A combined approach

I still believe that using Gmail's outgoing mailserver has its advantages. They are reliable, and in case of denial they return clear status codes. A solution with Gmail as primary, and your server as fallback, sounds like a good idea to me.

Further reading

Google's Bulk Senders Guidelines has more useful information to get delivered.

Not for marketing

Given the limits and risks, I'd definitely not use Gmail for sending anything that could be marked as spam by the receivers. Marketing, newsletters, even if the user at some point actively asked for it. Only send high priority mail such as transaction confirmations though Gmail SMTP.

Monday, January 21, 2013

Java subList() for offset and limit

The other day I needed to limit the items in a list based on traditional offset and limit criteria. I could not find any on the net. And because it bears potential for little bugs I wrote a library function with unit tests. Here's my take:

/**
 * Returns a range of a list based on traditional offset/limit criteria.
 *
 * <p>Example:<pre>
 *   ListUtil.subList(Arrays.asList(1, 2, 3, 4, 5), 3, 5) => [4,5]
 * </pre></p>
 *
 * <p>In case the offset is higher than the list length the returned 
 * sublist is empty (no exception).
 * In case the list has fewer items than limit (with optional offset applied) 
 * then the remaining items
 * are returned (if any).</p>
 *
 * <p>Impl notes: returns a {@link List#subList} in all cases to have 
 * a consistent return value.</p>
 *
 * @param list The input list.
 * @param offset 0 for now offset, >=1 for an offset.
 * @param limit -1 for no limit, >=0 for how many items to return at most, 
 *              0 is allowed.
 */
public static <T> List<T> subList(List<T> list, int offset, int limit) {
    if (offset<0) throw new IllegalArgumentException("Offset must be >=0 but was "+offset+"!");
    if (limit<-1) throw new IllegalArgumentException("Limit must be >=-1 but was "+limit+"!");

    if (offset>0) {
        if (offset >= list.size()) {
            return list.subList(0, 0); //return empty.
        }
        if (limit >-1) {
            //apply offset and limit
            return list.subList(offset, Math.min(offset+limit, list.size()));
        } else {
            //apply just offset
            return list.subList(offset, list.size());
        }
    } else if (limit >-1) {
        //apply just limit
        return list.subList(0, Math.min(limit, list.size()));
    } else {
        return list.subList(0, list.size());
    }
}

Wednesday, January 16, 2013

i18n of a Play Framework 2.0.4 Java App / Website

This post shows how I internationalized the social surfing website and app surfr.co that is built with Play. Feel free to take from it what serves you, or comment on what could be done better.

My setup:
  • Play Framework 2.0.4 using Java
  • A standard website where the language code is in the url (example: /en/pagename), and ajax requests where it's not.
  • I'm only using 2-letter language codes like "en", no countries like "en_US".
What you need:
  • Basic knowledge of Play.
  • Small to medium-sized website/app. I don't recommend it for large sites.

Some things are a bit different for Play using Scala, or for Play version 2.1 (which is not released yet). But parts may still be of use for you.

The documenation is at http://www.playframework.org/documentation/2.0.4/JavaI18N

Step 1: Define the languages.

In conf/application.conf add/edit the variable:
application.langs="en,de,ru"

Note:
  • I had to put the values in double-quotes. No quotes as in the documentation failed.
  • Order the languages by priority.

Step 2: Add the texts

As in the documentation create the files conf/messages, conf/messages.de and conf/messages.ru.
Create the files as UTF-8 files so that any text works, not just western. And don't add the BOM because Play 2.0.4 can't handle such files. On Windows you can create the files with Notepad: "save as..." and choose UTF-8. 

Note: 
  • If a text is not translated to a certain language then it always falls back to the default language.
  • Use 2 single quotes instead of 1. Example: Sam''s Pizzaland. Standard Java message file rules apply.
  • It doesn't seem to be supported to separate content by file. All goes into the one messages file. As I said... small/medium sized sites only.

Step 3: Figure out that the default handling isn't suitable

Now you're ready to use the texts in Java code and in templates. 
Messages.get("my.message.key")
Messages.get(new Lang("en"), "my.message.key")
The problem: Either you pass the user's language around everywhere (no way), or you settle with the built-in language selection (maybe). That is: using the first language that the user's browser settings mention for which you have translations.
For me, that was not acceptable. My website lets the user change the language. Unfortunately, Play does not offer a simple way to overwrite the lookup of the default language. I read that it will be supported in version 2.1, and that there are easier ways for overriding in the Scala version. So here's what I did.

Step 4: Implement language selection and lookup

I intercept each request to figure out the best language.

Create a new packageless (ouch) class named Global (ouch).
public class Global extends GlobalSettings {
    @Override
    public Action onRequest(final Request request, Method actionMethod) {
        I18n.defineBestLang(request);
        return super.onRequest(request, actionMethod);
    }
}

Note: it should theoretically be possible to name the class differently and put it into a package by configuring application.global= in the application.conf file, but it did not work for me.

The defineBestLang() method goes by this priority:
  1. Read from the url. My web urls (not ajax urls) contain the language in the form of /en/pagename.
  2. Read from cookie. For the app I'm using a 'lang' cookie.
  3. The browser's language settings, just like default Play. Something like Lang.preferred(request.acceptLanguages())
  4. Hardcoded default language.

It then stores the information in a ThreadLocal. Yes it works, I'm using just 1 thread per request. You can also use it as a start and pass it around where needed.

At this point the language is always available in controllers, templates, services. 
My I18n class also has static methods for getting the language code, and String and Html text for a certain key for the "best" language.
In order to not include that I18n class in all templates, I added it to the Build.scala file: 
templatesImport += "mypackage.I18n"

Step 5: Use the text

In a template I can now instead of @Messages.get("my.message.key") just use @I18n.text("my.message.key") for a pure text or @I18n.html("my.message.key") for html formatting.

Step 6: Routing

The front page has 2 entries in the routes file. For example the front page:
GET     /                             controllers.ControllerFactory.getApplication.index()
GET     /$lang<[a-z]{2}>   controllers.ControllerFactory.getApplication.indexI18n(lang:String)

public Result index() {
    return i18nRedirect();
}
public Result indexI18n(String lang) {
    return ok(views.html.Surfrco.index.render());
}
protected Result i18nRedirect() {
    return redirect("/"+ I18n.code()+request().uri());
}

The guest's language is auto-detected, and he's redirected to his language. This way I'm not serving duplicate content on multiple URLs. Currently I'm doing the same with other content pages (2 routes, with and without language) but it's not really necessary as long as no one links there.

Step 7: Client side i18n

My server-side messages file contains all texts as used in Java code, plus static webpage content. The client side only needs a couple phrases in JavaScript. That's why I've decided against streaming the whole messages file to the client. Instead I've created 3 files (UTF-8 again) messages.en.js etc. and serve only the one to the client:

The file's content is of the form:
function initMessages(m) {
m['my.text.key']="My Text";
...
}
if (typeof srfMessages == 'undefined') srfMessages = {};
initMessages(srfMessages);

And elsewhere is a very simple function to retrieve texts:

function t(key) {
    return srfMessages[key];
}

Note that in contrast to the server side, there is no fallback here to the default language.

And that's how i18n works for surfr.co.