AppEngine Datastore: Best Practices

After working with AppEngine and its Datastore for several months now (in production!) there's a few things I learned. Although I think you have to get some hands-on action first in order to fully grasp the concept and how to work with the Datastore, I think it's still useful to read a few tips first, keep them in mind and eventually implement them when you hit a wall for the first time during development.

The thing with consistency

It took me some time to fully understand "eventual consistency" vs. "strong consistency". This article describes it very well: Introduction to data models in Cloud Datastore

In a nutshell, eventual consistency does not guarantee when updates to the Datastore will be visible to your server, whereas strong consistency does. Usually eventual consistency is enough: fetching a list of products, etc. There is a very important usecase for strong consistency though: checkout / billing / everything that touches money. You don't want a transaction to disappear...

To put a long story to an end: use ancestor queries for strong consistency!

Speedy Datastore writes

Obviously, smaller entities are written faster than big entities. However, there's another thing that can slow down Datastore writes, even for the smallest entities: indexes! Imagine you have an entity with only a few properties, but index those in all variants (ascending, descending, etc): for each new and updated entity the Datastore has to update indexes accordingly resulting in - again - Datastore writes.

Here's how to calculate the cost of a write: Understanding write costs

And here's the overall process of saving an entity and updating indexes: Life of a Datastore Write (make sure to take a look at the articles linked in "Related links")

Tip: use indexes wisely (I usually don't create any until I really need them) and use unindexed properties for things you'll never have to query anyway!

Default values for all properties

Of course it seems convenient to use null-values for properties where there is nothing to save (yet). However, null-values are not queryable in the Datastore! Such entities simply won't show up in any query based on the property that's null. The only way to find properties with null-values is iterating all entities of a kind. Nothing you want to do repeatedly!

Upgrading your model and cleaning up your data

Chances are high you won't stay with your schema forever (sorry!). Whenever I need to adapt our schema I use the Remote API to iterate all entities and change them accordingly. Same goes for cleaning up a mess or setting "Default values for all properties" ;)

Easy refactoring and maintenance for Datastore-related code

There's two more things I'd like to mention, although they're not specific to AppEngine but good code in general:

1. Keep all code related to the Datastore in a class (in the best case one class per kind) and do not scatter different queries across your whole code.
2. In those classes per kind, store names of each property and the name of the kind as constants. This allows you to rename a property in a breeze!

I hope to be back with more tips as my journey with AppEngine continues...