Proposal: Redis Cache support

[eandre]

Background

Caching is a common approach to improving user experiences through faster response times and lower latencies. While it's possible to implement a simple in-memory cache already today with Encore, it doesn't work very well when deploying Encore in a serverless environment with zero to many instances running, causing poor cache hit rates.

Many backend applications would benefit from easier and better cache support in Encore. This proposal offers a way to easily use Redis as a cache, in a way that keeps with Encore's philosophy around simplicity and ease of use. The proposed API is type-safe and portable across cloud providers.

Design

We propose adding a new package, encore.dev/storage/cache that defines a NewCluster function that creates a new Redis cluster, similarly to pubsub.NewTopic:

package cache

// NewCluster declares a new cache cluster.
func NewCluster(name string, cfg ClusterConfig) *Cluster

// ClusterConfig represents the configuration of cache clusters.
type ClusterConfig struct {
	// Specific configuration values to be decided.
}

type Cluster struct {
	// contains filtered or unexported fields
}

Cache Keyspaces

When using a cache, each cached item is stored at a particular key, which is typically an arbitrary string. If you use a cache cluster to cache different sets of data, it's important that distinct data set have non-overlapping keys.

Each value stored in the cache also has a specific type, and certain cache operations can only be performed on certain types. For example, a common cache operation is to increment an integer value that is stored in the cache. If you try to apply this operation on a value that is not an integer, an error is returned.

We propose solving both of these challenges by introducing the notion of a Cache Keyspace.

In order to use the cache, you must define a keyspace that specifies a pattern that cache keys take, as well as the data type of the values you store.

It looks like this:

package cache

// KeyspaceConfig specifies the configuration options for a cache keyspace.
type KeyspaceConfig struct {
	// KeyPattern is a string literal representing the cache key pattern for this keyspace.
	KeyPattern string

	// DefaultExpiry specifies the default key expiry for cache items
	// in this keyspace. If zero the cluster default expiry is used.
	DefaultExpiry time.Duration

	// ... more options to be decided.
}

// NewStringKeyspace creates a keyspace that stores string values in the given cluster.
//
// The type parameter K specifies the key type, which can either be a
// named struct type or a basic type (string, int, etc).
func NewStringKeyspace[K any](cluster *Cluster, cfg KeyspaceConfig) *StringKeyspace[K]

// StringKeyspace represents a set of cache keys that hold string values.
type StringKeyspace[K any] struct {
	// contains filtered or unexported fields
}

// Set sets the cache key to val.
func (*StringKeyspace[K]) Set(context.Context, key K, val string) error { ... }

// Get gets the value stored for key. It returns Nil as an error if the cache key does not exist.
func (*StringKeyspace[K]) Get(context.Context, key K) (string, error) { ... }

//  ... lots more cache operations (update, delete, get and delete, etc) ...

// Nil is the error value reported when a key is missing from the cache.
var Nil = errors.New("cache: nil")

Beyond StringKeyspace Encore will provide other keyspace types like IntKeyspace, FloatKeyspace, SetKeyspace, and other Redis data types. These are different types instead of a single Keyspace[Key, Value any] generic type since the supported operations differ depending on the type.

Specifying cache keys

As seen above, keyspaces accept a K type parameter that represents the key type. K can either be a basic type like string or int, or it can be a named struct type that contains multiple fields.

This type gets combined with the KeyPattern configuration setting to produce the actual cache key values are stored in. The KeyPattern can be thought of as a formatting pattern that describes how the K value should be turned into a string key.

Through static analysis Encore can then reason about the different cache keyspaces and ensure they all use non-overlapping keys.

Example

For example, here's how one might define a cache cluster and a per-user id cache of the number of incoming requests:

// CacheCluster is our shared cache cluster.
var CacheCluster = cache.NewCluster("cache", cache.ClusterConfig{})

// requestsByUser caches the number of incoming requests by user id.
var requestsByUser = cache.NewIntKeyspace[auth.UID](CacheCluster, cache.KeyspaceConfig{
	KeyPattern: "users/:key/requests",
	DefaultExpiry: 5 * time.Minute,
})

//encore:middleware global target=all
func UserRequestRateLimiter(req middleware.Request, next middleware.Next) middleware.Response {
    uid, ok := auth.UserID()
    if !ok {
        // No user information, let the request through.
        return next(req)
    }
    
    num, err := requestsByUser.Incr(req.Context(), uid, 1)
    if err == nil && num > 10 {
        // Rate limit exceeded, return an error.
        return middleware.Response{
            Error: errs.B().Code(errs. ResourceExhausted).Msg("rate limit exceeded").Err(),
        }
    }

    return next(req)
}

[DomBlack]

If zero the cluster default expiry is used.

What if I don't want my values to expire ever?

[eandre]

We can define this as a custom constant with a negative duration

[DomBlack]

In Cache Keys:

or it can be a named struct type that contains multiple fields.

How does /users/:key/requests" resolve if my K type is a complex type with loads of fields;

type Foo struct {
   Bar *Bar
}
type Bar struct {
   Foo *Foo
}

key := &Foo{}
key.Bar = &Bar{}
key.Bar.Foo = key

var someKeyspace = cache.NewIntKeyspace[Foo](...)

[eandre]

Each struct field would be its own path parameter, so one/:foo/two/:bar for type MyStruct { foo string; bar int }. I don't believe there's any real need to support nested fields here so we can require such structs to be flat and only contain basic types as their fields.

[Willyham]

There's a slight bit of nuance here which may or may not be worth worrying about. It would technically be possible to construct two keys which, whilst not detected by static analysis, might actually clash. E.g.

Space foo/:bar/baz Key bar
and
Space foo:/:bar Ker bar/baz

[eandre]

Yeah, I thought about that too, but not sure it's worth worrying about. We could enforce that strings get URL encoded or whatever but that seems overly specific for most use cases.

[DomBlack]

These are different types instead of a single Keyspace[Key, Value any] generic type since the supported operations differ depending on the type.

Would Encore provide a generic version like this? A large use-case for Caching is to cache expensive objects and being able to do the following would be a large usecase:

type ExpensiveThingToCompute struct { ... }

var expensiveObject = cache.NewKeyspace[String, ExpensiveThingToCompute](...)

func DoExpensiveCompute() (ExpensiveThingToCompute, error) { ... }

expensiveObject.GetExistingOrSet("BTC/USD Historical Rates", DoExpensiveCompute)

[eandre]

Yes, but I'm thinking NewObjectKeyspace[K, V any] or similar. My point was rather than we can't use generics to avoid StringKeyspace vs IntKeyspace vs ObjectKeyspace as they have different supported operations.

[Willyham]

Any arbitraray struct storage would need to serialisable, because redis can only store maps. That would mean encore would have to take an opinion on serialisation which definitely has some nuances for certain types. Might be safer to initially just offer strings, and let people build their own serialised wraps keyspace types until it's clear what the use cases are?

[eandre]

One idea I had around that was to map "objects" to effectively JSON marshalled data, and then support "maps" separately with operations around setting/getting/updating specific keys in the map. Thoughts?

[Willyham]

Yes that was the approach I was thinking of. It just means that you'd need to communicate that structs must be json serialisable to work (because not all types are - e.g errors). Alternatively you encourage people to do their own marshaling so there's no confusion?

[eandre]

I was thinking of requiring that the values satisfy Encore's rules for request/response fields which recursively validate that everything can be marshalled.

One downside is it makes it difficult to cache e.g. values from a library since Encore does not currently parse data structures defined outside of the Encore application itself. But that's a limitation that can be lifted later.

[DomBlack]

// Set sets the cache key to val.
func (*StringKeyspace[K]) Set(context.Context, key K, val string) error

One really useful capability to have is to be able to set TTL per record. For instance I might want to expire values at midnight every day. Which means when I initially set it I'd want to set the TTL as the number of seconds to midnight or a specific time.

I feel like we might need to change this API design to;

func (*Keyspace[K, V]) Set(context.Context, K, V, cache.Options...) error

Where options allows overrides on a per key basis;

keyspace.Set(ctx, "Blah", "foo", cache.ExpireIn(20 * time.Second))

[eandre]

Yeah, good point; I elided it in the API but we definitely should do that

[DomBlack]

Can a cache cluster be shared between services?

[eandre]

Yes, I think that makes sense; thoughts?

[Willyham]

Is there a slight worry that it would be 'abused' as a store for people to share state across services? Without passing too much judgement on that, it seems like encore already took a stance in not allowing database stores to be used cross-service. Is there a reason that it should be different for caches?

[eandre]

We have opened up the ability to reference databases from different services (with sqldb.Named()). With databases you can let each service have its own database but share a cluster. With redis the multiple-database support is really poor (each database is just an integer index from 0-15 without any names) so it seems more valuable to be able to share a cluster.

One idea could be to prevent keyspaces from being shared across services, but allow the same cluster to be used.

[DomBlack]

Do we need to declare a Cluster, I feel that might be leaking implementation details into the API and I'm unsure to the usecase it solves, vs directly calling cache.NewStringCache(...)

[eandre]

Is it implementation details? It seems to me you don't want a whole backend using the same cache cluster once you reach a certain size. How would you decide how cache keyspaces map to clusters?

[DomBlack]

What's the plan for support cache in tests and local development?

[eandre]

Fully supported; in tests using https://github.com/alicebob/miniredis, in local development either using a Redis docker container or miniredis again.

[Willyham]

I love the idea of providing this primitive out of the box! ❤️

One minor question - have you thought through any possible implications for the API if encore itself wants to build on top of this type with other primitives?

For example, I could see a future where it would be nice to offer a distributed rate limiter primitive. Those are usually implemented with something like redis as the backing store. In that case, are there any API changes you might want to make to allow the platform to have have access to this primitive internally? I'm thinking of things like reserving specific keys or prefixes so that you can't have collisions?

[DomBlack]

For example, I could see a future where it would be nice to offer a distributed rate limiter primitive.

For this we've got plans to use something like Doorman's algorithm to create a cooperative rate limiting system which handles scaleup automatically

[Willyham]

Regardless of the feature though, there may be internal things which encore wants to use redis for. Maybe it's worth rejecting any user keys prefixed with encore/ to reserve a non-clashing keyspace if you need it in the future?

[davidmytton]

Perhaps you're going to deal with this later, but what about allowing a custom cluster URI for a Redis instance not managed by Encore? For example if I wanted to use a managed Redis service like https://cloud.google.com/memorystore on GCP or a third-party service like Redis Enterprise Cloud. As with databases, I'd probably want to specify this per environment e.g. use Redis Enterprise Cloud in prod but still have Encore manage the local dev environment with Redis docker/miniredis.

[eandre]

Yeah, that's something we're working towards for all Encore primitives. It's coming, even if it's still a ways off.

Do note that Encore will provision managed Redis when deploying to your own cloud (Memorystore/ElastiCache/etc) from the start.

[marcuskohlberg]

For visibility: This has now been added (a long time ago!)