Store gobs as alternative to JSON

[philippgille]

Currently the storage implementations all marshal and unmarshal to/from JSON. For example, a wall.invoiceMetaData object is first marshalled to JSON and then the resulting string is stored as value in Redis.

JSON is nice because when problems occur (e.g. a customer says his HTTP client sent the correct X-Preimage header but the web service responded with an error "Corresponding invoice not found" or something similar), you can just check the storage and have a look at the stored data, easily seeing what's in there without having to decode anything.
This might be harder for the Go Map (it's only visible / only exists within the running web service) and the bbolt DB (file is locked while in use), but easier for Redis and future storages like etcd and Consul, where even some web dashboards for exploring the storage contents exist.

The downside of JSON though is that it's verbose. A binary format can be much more compact and thus require less storage.

In Go, there are "gobs". See:

• https://godoc.org/encoding/gob
• https://blog.golang.org/gobs-of-data

Before implementing this, we should first benchmark if it makes sense at all for the limited types of data we store (currently only wall.invoiceMetaData). A 50% reduction in storage would be nice, or maybe 30% is enough?

Also be aware that when storing gobs, other programming languages can't deal with the data! For example when a customer uses one lnd instance, but two web services with one using ln-paywall and the other using a compatible Java middleware, and he wants to prevent web service clients from cheating by reusing preimages.
Cheating example: Send first request to the Go web service, pay the invoice, send final request with preimage to Go web service, then send another request immediately with the same preimage to the Java web service.
If both web services would have separate storage mechanisms the client could cheat that way. So a common storage must be used, like Redis for example. But when the stored value in Redis is a gob, the Java middleware can't read it.
So:

1. Using gobs instead of JSON must be optional, off by default
2. This downside must be mentioned in the comments for the respective option (probably in the storage implementation's specific options object)