Google App Engine – Caching and Downtimes (Rant)

Google App Engine:

“We will be taking memcache offline tomorrow morning from 9-10am PST (GMT-8) for routine maintenance. Calls to the memcache API will *not* throw exceptions but will instead return false for set() calls and None for get() calls (just like any other cache miss.) Your app should continue serving normally during this period, and
we’ll keep you updated on our progress.”

Google writes this as though it’s “no biggie – we’re just disabling caching for an hour – your app will operate as normal”.

If you’ve used App Engine, you know how ridiculous that is. You can’t operate an App Engine site without lots and lots of caching.

App Engine takes your generous daily quota and divides it up into tiny little minute or second long quotas. Their logic is something like this:

1. You can eat 2000 calories in one day.
   Hurray! That’s a lot of food!
2. It’s good to pace yourself and not eat it all at once.
   Absolutely. You wouldn’t want to pig out on breakfast and not be able to eat anything else all day.
3. Therefore, we will only let you eat 1.4 calories per minute.
   Well, f*ck. Pass me two thirds of a tic tac?
   

CareerCup gets about 10,000 page views per day – not the smallest site, but hardly the biggest. CareerCup cannot operate without heavy caching. By taking down caching, they took down my site for an hour. Not cool.

Now, Google could have mitigated this by removing the absurdly small quotas temporarily. CareerCup would have run slowly, but at least it would have run. Instead, though, users get punished for expensive-ish queries, with no chance to avoid it. Not cool.

This brings me to my next point:
Google, if you’re going to take down people’s sites, can you pick a better time than 9am – 10am? Try, perhaps, 2am – 3am? I know you don’t want to come into work at 2am. I know it’s not really Google culture to tell a team that they have to be at work and away from their families 2am. But you have to. You have real users operating real businesses, many of which are a whole lot bigger than CareerCup. We depend on you to keep our websites up.

Google: Don’t act like taking down memcache doesn’t disable our sites. And don’t disable our sites at 9am when you could’ve done this 2am.