Keeping the generated data takes a lot of volume

[perpi]

Each record is approximately 25k in size, which is 230 terabytes in our annual estimate for 10 billion visiting records. It is very large, and the cost of infrastructure would be very expensive! What to do?

To whom are currently using it in the operational environment, what solution do you have for data storage?

[makma]

Hey @perpi!
Why does each record have 25kB? What do you store there?

[perpi]

Why that size: The library generates some data that can be stored as a JSON file somewhere, for example in a mysql database, and the size each record is 25KB.

Why storing: Seems it's valuable data for our data science department!

[makma]

I know some customers/users use Clickhouse to persist raw data.

[Finesse]

Store hashes instead of the raw data. For example:

// This code runs on a server
import * as crypto from 'crypto'

const components = request.body.fingerprintComponents
const hash = value => crypto.createHash('sha512').update(value).digest('base64')
// Deep-clone `components` if you don't want to mutate it
if (components.canvas?.value?.geometry) {
  components.canvas.value.geometry = hash(components.canvas.value.geometry)
}
if (components.canvas?.value?.text) {
  components.canvas.value.text = hash(components.canvas.value.text)
}
if (components.math?.value) {
  components.math.value = hash(JSON.stringify(components.math.value))
}
// `components` is now compressed

[mohaa7]

I asked the managers. Our product dep., collabrating w/ data scientists, needs to know more about users' devices. Hashing is useful for identification but not in our case since the gathered data itself is interesting.
We, in DevOps dep., are dealing with the size of data and how big it would be in the future!

[Valve]

@mohaa7 many databases can compress data very efficiently, like @makma mentioned. E.g. ClickHouse will compress and store the deltas of data, I think the simple math of item_size x number_of_items shouldn't be used.
For a real-world scenario a layered storage can be used, e.g. the most recent data (from now to last 3 months) can be stored in ClickHouse, then the data gets archived to S3, then to Glacier and this would be a good way to save on storage costs.