Naveen Gattuserver

Domino — Typesafe DynamoDB DSL for Go

dominos VSCO Photo by [emw](https://vsco.co/em_w/media/585bffe071f8fb7f08addc34)_

DynamoDB… that wildly popular AWS NoSQL database with all the bells and whistles used by companies large and small, offers high performance, low maintenance and flexible pricing. I love it, some hate it, others can’t live without it. If you know what you’re getting yourself into, however, it can be a very useful backing store for large quantities of data (you really just need to ensure your access patterns are well distributed across your key space). But when it comes to using the AWS (probably auto-generated) APIs, specifically for Go, frustration levels start to spike and productivity undoubtedly declines. Lets take a peek at a non-trivial example:

sess := session.New(config)
dynamo := dynamodb.New(sess)

input := &dynamodb.QueryInput{}

keyCondition := "locale = :locale"

filterExpr := "email IN (:email1, :email2, :email3) AND size(visits) < :visits AND registrationDate > :regDate"

regDate := time.Now().Format(time.RFC3339)

exprAttrValues := map[string]*dynamodb.AttributeValue{
  ":locale": (&dynamodb.AttributeValue{}).SetS("s"),
  ":email1": (&dynamodb.AttributeValue{}).SetS("[elon@tesla.com](mailto:elon@tesla.com)"),
  ":email2": (&dynamodb.AttributeValue{}).SetS("[page@google.com](mailto:page@google.com)"),
  ":email3": (&dynamodb.AttributeValue{}).SetS("[bezos@](mailto:bezos@worlddomination.com) pwnworld.co"),
  ":visits": (&dynamodb.AttributeValue{}).SetN("5"),
  ":regDate": (&dynamodb.AttributeValue{}).SetS(regDate),
}

input.SetTableName("users")
input.SetKeyConditionExpression(keyCondition)
input.SetExpressionAttributeValues(exprAttrValues)
input.SetConsistentRead(true)
input.SetFilterExpression(filterExpr)
input.SetIndexName("locale_index")

out, err := db.QueryWithContext(context.Background(), input)
if err != nil {
    return err
}

//Collect results
var users []*User
for _, av := range out.Items {
  user := &User{}
  if err = dynamodbattribute.UnmarshalMap(av, user); err != nil {
     return err
  }
  users = append(users, user)
}

In this example we want to perform a query on our users table looking for a specific set of users, filtering out those that have visited 5 times or less and registered after the specified date. We also specify an index to use using a raw string.

Besides being highly verbose and requiring a lot of boilerplate for a single query, it is string-ly typed with respect to the specified columns within the condition query, filter expression and the expression attribute value map declaration. These issues force the programmer (and peer reviewers) to visually verify the types of the index and table keys against what is defined in DynamoDB via the web console. Worse yet, the fields defined in the filter expression are not statically defined at all, since DynamoDB is a NoSQL database. This means that types are determined after a field is set for the first time, as in MongoDB. Furthermore, the query, and filter expressions are untyped strings themselves which can also be a source of bugs. So unless you have extremely well-documented code and don’t mind code repetition and mental variable tracking, this is at the very least very annoying. And we haven’t even considered issues that will arise during refactoring.

The first step in making this better of course is to use constants to centralize the names of the table columns:

const(
     usersTable      = "users"
     localeField     = "locale"
     emailField      = "email"
     visitsField     = "visits"
     regDateField    = "registrationDate"
     localeIndex     = "locale_index"
)

keyCondition := localeField + "= :locale"
filterExpr   := emailField + " IN (:email1, :email2, :email3)" + "visits + " < :visits AND " + regDateField+" > :regDate"

Certainly this is better but it doesn’t go far enough. How can we be sure that the visits field holds a number that can be compared or that it holds a number at all? How can we be sure that visits is even a field within the users table? How do we ensure that those expression variables (:email1, :email2, …) are accounted for in the expression attribute value map?

Go is a strongly typed language and types are clearly better for server side programming. Why not leverage its type checker? We can start by defining types for DynamoDB tables, fields and indices.

type (
    dynamoStringField  string
    dynamoNumericField string
    dynamoIndex        string
)

type dynamoTable struct {
    localeField  dynamoStringField
    emailField   dynamoStringField
    visitsField  dynamoNumericField
    regDateField dynamoStringField
    localeIndex  dynamoIndex
}

table := dynamoTable {
    localeField  := dynamoStringField("locale")
    emailField   := dynamoStringField("email")
    visitsField  := dynamoNumericField("visits")
    regDateField := dynamoStringField("registrationDate")
    localeIndex  := dynamoIndex("locale_index")
}

Excellent! Now we have a static schema for our table defined in code, and readable code really is the best documentation. We can continue declaring types to represent equivalent DynamoDB types, along with methods to further restrict what operations are possible against which types (think comparison operators against numeric values).

This idea, taken to its logical conclusion is the idea behind Domino, a library developed at VSCO to improve developer productivity when using DynamoDB. Domino allows us to write our table schema definitions once in code, and then utilize a very expressive DSL to construct query, filter, and key condition expressions. Lets take a look at our previous example and define a static schema for the users table:

import (
     "github.com/vsco/domino"
     "github.com/aws/aws-sdk-go/service/dynamodb"
     "github.com/aws/aws-sdk-go/aws/session"
)

sess := session.New(config)
dynamo := dynamodb.New(sess)

//Statically defined table schema
type UserTable struct {
   domino.DynamoTable   // Composed with a DynamoTable
   Locale        domino.String
   UserId        domino.Numeric
   Email         domino.String
   Visits        domino.NumericSet
   RegDate       domino.String //ISO 8601 format

   LocaleIndex    domino.GlobalSecondaryIndex
}

// Declare the locale index
localeIndex := domino.GlobalSecondaryIndex{
   Name:         "locale-index",
   PartitionKey: domino.String("locale"),
}

table := UserTable {
   DynamoTable: domino.DynamoTable {
       Name: "users",
       PartitionKey: domino.NumericField("id")
       GlobalSecondaryIndexes: []domino.GlobalSecondaryIndex{
           localeIndex,
       }
   },
   UserId:      domino.NumericField("id"),
   Email:       domino.StringField("email"),
   Visits:      domino.NumericSetField("visits"),
   RegDate:     domino.StringField("registrationDate"),
   LocaleIndex: localeIndex,
}

And now lets utilize the expression DSL to construct our condition and filter expressions:

keyCondition := table.Locale.Equals("us")

filterExp := domino.And(
      table.Email.In(
          "[elon@tesla.com](mailto:elon@tesla.com)",
          "[page@google.com](mailto:page@google.com)",
          "[bezos@](mailto:bezos@worlddomination.com) pwnworld.co",
      ),
      table.Visits.Size("<", 5),
      table.RegDate.GreaterThan(time.Now().Format(time.RFC3339)),
)

query := table.
         Query(keyCondition).
         SetFilterExpression(filterExp).
         SetConsistentRead(true).
         SetGlobalIndex(table.LocaleIndex)

//ExecuteWith accepts a context and dynamo instance
results := query.ExecuteWith(context.Background(), dynamo)

//Check for error
if err := results.Error(); err != nil {
    return err
}

//Fetch results from the Results object by iterating over
//returned rows, calling the passed in closure which
//collects and returns user objects
users := []*User{}
results.Results(func() interface{} {
    r := &User{}
    users = append(users, r)
    return r
})

The DSL allows for concise and legible expressions, and exposes methods that are appropriate for each field type. Numeric comparison operators for numeric fields and string comparison operators for strings, in addition to collection operations for set, list and map types. All condition and filter expressions are supported.

Lets take another example, this time updating the Visits field:

yesterday := time.Now().Add(-1 * time.Day).Unix()
results := table.
           UpdateItem(domino.KeyValue{123, nil}).
           SetUpdateExpression(
                 table.Visits.AddInteger(time.Now().Unix()),
           ).
           SetConditionExpression(
                 table.RegDate.GreaterThan(yesterday),
           ).
           ReturnAllOld().
           ExecuteWith(context.Background(), dynamo)

if err := results.Error(); err != nil {
    return err
}

oldItem := &User{}
results.Result(oldItem)

In this example, we update a user row by inserting a visit timestamp into the Visits field only if they have registered since yesterday, returning all the old values from the row. (Note: this is a toy example, you should never insert elements into a set ad infinitum without capping the size somehow).

Domino supports all DynamoDB operations including support for streaming and paging rows of queried and scanned data using go channels. Please give Domino a try and tell us what you think! And if writing high performance server code with cutting edge tech is your thing feel free to reach out, we’re hiring!

Share this post

|
|
|

r&d

VSCO Film X & The Imaging Lab

Zach Hodges
server

Querying S3 Data With Redshift Spectrum

Jenn Georgevich
machine learning, iOS, android

Using On-Device Machine Learning to Suggest Presets for Images in VSCO

Shruti Sharma
r&d

Reviving Kodachrome

Kyle Hale
© VSCO 2020. All rights reserved.