Domino — Typesafe DynamoDB DSL for Go
VSCO Photo by [emw](https://vsco.co/em_w/media/585bffe071f8fb7f08addc34)_
DynamoDB… that wildly popular AWS NoSQL database with all the bells and whistles used by companies large and small, offers high performance, low maintenance and flexible pricing. I love it, some hate it, others can’t live without it. If you know what you’re getting yourself into, however, it can be a very useful backing store for large quantities of data (you really just need to ensure your access patterns are well distributed across your key space). But when it comes to using the AWS (probably auto-generated) APIs, specifically for Go, frustration levels start to spike and productivity undoubtedly declines. Lets take a peek at a non-trivial example:
sess := session.New(config)
dynamo := dynamodb.New(sess)
input := &dynamodb.QueryInput{}
keyCondition := "locale = :locale"
filterExpr := "email IN (:email1, :email2, :email3) AND size(visits) < :visits AND registrationDate > :regDate"
regDate := time.Now().Format(time.RFC3339)
exprAttrValues := map[string]*dynamodb.AttributeValue{
":locale": (&dynamodb.AttributeValue{}).SetS("s"),
":email1": (&dynamodb.AttributeValue{}).SetS("[elon@tesla.com](mailto:elon@tesla.com)"),
":email2": (&dynamodb.AttributeValue{}).SetS("[page@google.com](mailto:page@google.com)"),
":email3": (&dynamodb.AttributeValue{}).SetS("[bezos@](mailto:bezos@worlddomination.com) pwnworld.co"),
":visits": (&dynamodb.AttributeValue{}).SetN("5"),
":regDate": (&dynamodb.AttributeValue{}).SetS(regDate),
}
input.SetTableName("users")
input.SetKeyConditionExpression(keyCondition)
input.SetExpressionAttributeValues(exprAttrValues)
input.SetConsistentRead(true)
input.SetFilterExpression(filterExpr)
input.SetIndexName("locale_index")
out, err := db.QueryWithContext(context.Background(), input)
if err != nil {
return err
}
//Collect results
var users []*User
for _, av := range out.Items {
user := &User{}
if err = dynamodbattribute.UnmarshalMap(av, user); err != nil {
return err
}
users = append(users, user)
}In this example we want to perform a query on our users table looking for a specific set of users, filtering out those that have visited 5 times or less and registered after the specified date. We also specify an index to use using a raw string.
Besides being highly verbose and requiring a lot of boilerplate for a single query, it is string-ly typed with respect to the specified columns within the condition query, filter expression and the expression attribute value map declaration. These issues force the programmer (and peer reviewers) to visually verify the types of the index and table keys against what is defined in DynamoDB via the web console. Worse yet, the fields defined in the filter expression are not statically defined at all, since DynamoDB is a NoSQL database. This means that types are determined after a field is set for the first time, as in MongoDB. Furthermore, the query, and filter expressions are untyped strings themselves which can also be a source of bugs. So unless you have extremely well-documented code and don’t mind code repetition and mental variable tracking, this is at the very least very annoying. And we haven’t even considered issues that will arise during refactoring.
The first step in making this better of course is to use constants to centralize the names of the table columns:
const(
usersTable = "users"
localeField = "locale"
emailField = "email"
visitsField = "visits"
regDateField = "registrationDate"
localeIndex = "locale_index"
)
keyCondition := localeField + "= :locale"
filterExpr := emailField + " IN (:email1, :email2, :email3)" + "visits + " < :visits AND " + regDateField+" > :regDate"Certainly this is better but it doesn’t go far enough. How can we be sure that the visits field holds a number that can be compared or that it holds a number at all? How can we be sure that visits is even a field within the users table? How do we ensure that those expression variables (:email1, :email2, …) are accounted for in the expression attribute value map?
Go is a strongly typed language and types are clearly better for server side programming. Why not leverage its type checker? We can start by defining types for DynamoDB tables, fields and indices.
type (
dynamoStringField string
dynamoNumericField string
dynamoIndex string
)
type dynamoTable struct {
localeField dynamoStringField
emailField dynamoStringField
visitsField dynamoNumericField
regDateField dynamoStringField
localeIndex dynamoIndex
}
table := dynamoTable {
localeField := dynamoStringField("locale")
emailField := dynamoStringField("email")
visitsField := dynamoNumericField("visits")
regDateField := dynamoStringField("registrationDate")
localeIndex := dynamoIndex("locale_index")
}Excellent! Now we have a static schema for our table defined in code, and readable code really is the best documentation. We can continue declaring types to represent equivalent DynamoDB types, along with methods to further restrict what operations are possible against which types (think comparison operators against numeric values).
This idea, taken to its logical conclusion is the idea behind Domino, a library developed at VSCO to improve developer productivity when using DynamoDB. Domino allows us to write our table schema definitions once in code, and then utilize a very expressive DSL to construct query, filter, and key condition expressions. Lets take a look at our previous example and define a static schema for the users table:
import (
"github.com/vsco/domino"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/aws/session"
)
sess := session.New(config)
dynamo := dynamodb.New(sess)
//Statically defined table schema
type UserTable struct {
domino.DynamoTable // Composed with a DynamoTable
Locale domino.String
UserId domino.Numeric
Email domino.String
Visits domino.NumericSet
RegDate domino.String //ISO 8601 format
LocaleIndex domino.GlobalSecondaryIndex
}
// Declare the locale index
localeIndex := domino.GlobalSecondaryIndex{
Name: "locale-index",
PartitionKey: domino.String("locale"),
}
table := UserTable {
DynamoTable: domino.DynamoTable {
Name: "users",
PartitionKey: domino.NumericField("id")
GlobalSecondaryIndexes: []domino.GlobalSecondaryIndex{
localeIndex,
}
},
UserId: domino.NumericField("id"),
Email: domino.StringField("email"),
Visits: domino.NumericSetField("visits"),
RegDate: domino.StringField("registrationDate"),
LocaleIndex: localeIndex,
}And now lets utilize the expression DSL to construct our condition and filter expressions:
keyCondition := table.Locale.Equals("us")
filterExp := domino.And(
table.Email.In(
"[elon@tesla.com](mailto:elon@tesla.com)",
"[page@google.com](mailto:page@google.com)",
"[bezos@](mailto:bezos@worlddomination.com) pwnworld.co",
),
table.Visits.Size("<", 5),
table.RegDate.GreaterThan(time.Now().Format(time.RFC3339)),
)
query := table.
Query(keyCondition).
SetFilterExpression(filterExp).
SetConsistentRead(true).
SetGlobalIndex(table.LocaleIndex)
//ExecuteWith accepts a context and dynamo instance
results := query.ExecuteWith(context.Background(), dynamo)
//Check for error
if err := results.Error(); err != nil {
return err
}
//Fetch results from the Results object by iterating over
//returned rows, calling the passed in closure which
//collects and returns user objects
users := []*User{}
results.Results(func() interface{} {
r := &User{}
users = append(users, r)
return r
})The DSL allows for concise and legible expressions, and exposes methods that are appropriate for each field type. Numeric comparison operators for numeric fields and string comparison operators for strings, in addition to collection operations for set, list and map types. All condition and filter expressions are supported.
Lets take another example, this time updating the Visits field:
yesterday := time.Now().Add(-1 * time.Day).Unix()
results := table.
UpdateItem(domino.KeyValue{123, nil}).
SetUpdateExpression(
table.Visits.AddInteger(time.Now().Unix()),
).
SetConditionExpression(
table.RegDate.GreaterThan(yesterday),
).
ReturnAllOld().
ExecuteWith(context.Background(), dynamo)
if err := results.Error(); err != nil {
return err
}
oldItem := &User{}
results.Result(oldItem)In this example, we update a user row by inserting a visit timestamp into the Visits field only if they have registered since yesterday, returning all the old values from the row. (Note: this is a toy example, you should never insert elements into a set ad infinitum without capping the size somehow).
Domino supports all DynamoDB operations including support for streaming and paging rows of queried and scanned data using go channels. Please give Domino a try and tell us what you think! And if writing high performance server code with cutting edge tech is your thing feel free to reach out, we’re hiring!
