Reflect on the oplog implementation

[radekmie]

Disclaimer 1: This is probably going to be a mix of a couple of feature requests under one "umbrella" discussion. I may need to split it later - we'll see how the discussion goes.

Disclaimer 2: I do use the real-time aspect of Meteor really hard in a couple of apps; definitely less in some other, where it's not business critical. I don't want this to be another round of "just don't use it" or "Meteor doesn't scale" (it does) kind of discussion.

---

The current implementation of the real-time part of the Meteor-Mongo integration is a great piece of software. If it wouldn't, probably most of the community wouldn't be here today. But as time goes and the app scale, we see that it's definitely not ideal.

That's where projects like cultofcoders:redis-oplog or any of the "publications replicated using methods" came in. (I'll skip the latter as they effectively change the way our code behaves, not only the "backend" of it.) And it's awesome! It helped many projects, including the ones I've worked on. Let's just pause for a minute on the fact that this forum post has stunning 626 (!) messages from 85 users. As of today, a couple of years later, it just works with the latest version of Meteor.

Somehow in parallel (OK, nearly a year later), the discussion to replace oplog tailing with change streams has started. Back then I didn't care that much as none of the projects I worked on have had any problems with the current implementation. (Just to be clear - none of them have these problems today. We've had some hiccups, but it's all fine now.) Time has passed, some great comments were made 1, 2, 3, 4, and the discussion died naturally, as it turned out not to be so great (probably).

Now, in one of the projects, we actually do use both.

• redis-oplog is there to let us do things the "naive way" (at this point we're far from naive - every published field and reused cursor counts). As we do use bulk operations really heavily in the app (we have relatively many places where that's the only sensible way of not doing tens of updates at once), we would have to notify Redis ourselves. Luckily, the amazing oplogtoredis does the job for us. It's run separately, effectively getting rid of all of the work we would have to do manually. (On top of that, the servers don't have to push anything to Redis.)
• Change streams are part of our application that handles "history logging". It's a simple feature - we'd like to store the history of a collection for auditing purposes. The whole code takes ~200 lines to make it work (+some code to elect the main node). Of course, it can be run as an external service but it's easier for us to deploy it within the Meteor app.

---

As I think I've set up the picture, let's get to the actual feature request. Let's divide it into parts:

• Guarantee support for redis-oplog or a similar solution. I'm aware of the forks out there (I still have to try this one but oplogtoredis is a must for us; it may happen though), but I think that this piece of code is simply crucial for a big enough Meteor application with a decent amount of reactive publications in place.
  • I'm not sure how exactly I see this guarantee, sorry. Maybe make it an official package?
  • Similarly, official support of oplogtoredis would be welcome as well.
• Reconsider change streams once again. I'm not sure whether reading the entire change stream of a collection is that more data than tailing the oplog. Also, it'd make the tedious work of updating oplog parsing go away (this one).
  • Maybe a redis-oplog-like solution would work? Like a sidecar service, watching change streams and publishing it somewhere (Redis is fine)? I'm just thinking out loud now.
• Improve the current oplog tailing implementation. There were many ideas that died due to lack of attention or lack of author's time. I imagine that throttling oplog notifications or batching DPP messages (most of them come from oplog) would improve the situation of many projects.

Having said that, there's one more thing. I don't need that personally (like most people), but it may be a huge selling point sooner than later:

• Support shared MongoDB clusters. It may sound crazy while doing oplog tailing (see this feature request), but with change streams
  or oplogtoredis in place, it may be possible. It'd get rid of one of the all-time Meteor limitations.

[harryadel]

Really well put arguments with great references, thanks for sharing.
I do a agree with you that support for redis-oplog is now more important than ever given @theodorDiaconu has moved on to other stuff though if the other alternatives to oplog take off maybe maintaining redis-oplog won't be much of an issue at all aside from backwards compatibility.

[harryadel]

Actually this is a sign of greater problem of Meteor Software leadership, they don't put any effort into keeping community leaders around. @theodorDiaconu at some point wanted to merge redis-oplog into the core.

Honestly, I tried doing this (Kaviar) in Meteor :slight_smile:, because I love this framework so fucking much. After Tiny bought Meteor I was in discussions with them pushing the fact we can rejuviate it and what would be the steps to do so, I wanted redis-oplog to get merged officially. These guys showed zero interest. I felt discouraged, I thought being such an evangelist of Meteor my words and advice would weigh more. This kinda forced me to pursue my vision with a framework that allows me to do so, I had to reinvent many wheels and jump many hoops before Kaviar took this shape. But I am confident that this is the future.

Imagine how things would have been if Meteor Software attempted to keep him around. Reminds me of the departure of @arunoda.

[filipenevola]

Hi @harryadel, I'm not going to engage in this type of discussion.

We have been in contact with @theodorDiaconu in the past. But I was not the point of contact in all the cases, so I can't respond directly to his comment.

Again, I'm not going to proceed with this discussion because I don't see the point here.

This description from @radekmie is really great and I would like to focus on it instead of other things.

[harryadel]

I think you're missing the point here: now that's Meteor considering having redis-oplog into the core, wouldn't it be better to have the guy who created it continue the work?

Aside from that I agree with most of you points. I don't expect anything for free nor I do want Meteor to depend on community leaders although if you actually look into it you would realize most of the build tool work is now being handled by @zodern. Imagine if he as a community leader decided to ditch the ship. Community leaders are free to come and go right? Though I'm sure Meteor values his efforts.

Just to be clear, I don't use redis-oplog and I'm not blaming Meteor for the no longer free redis-oplog per se but not putting more effort into keeping key people around especially when the community is small already and we need more hands on deck. Finally, if redis-oplog is now going to be maintained by the core team regardless of being added to the core or not, that means less time allotted to other core work. :)

[filipenevola]

Hi @paulincai and @harryadel, redis-oplog is an important piece in the Meteor ecosystem, I had big projects already using it, and it is a great solution.

I also don't understand @harryadel arguments in many ways, but I don't want to engage in this type of discussion as I don't see the point.

I want to highlight @zodern, @zodern is a great community member, he knows a lot about Meteor internals, and we have a partnership with him for a long time already, and yes, we do sponsor him, as you can see in his GitHub profile.

We can't force people to work full-time for us, so we can't just choose who will be in the core team or not from all our community members, but we can partner beneficially for Meteor and the contributor.

We are also willing to sponsor other contributors willing to work according to our roadmap and priorities. We have always made clear and opened our channels (including my email and DMs everywhere) to people that want to contribute to Meteor to get in touch.

About the roadmap, yes, we have a new roadmap update in place, we are going to announce it with 2.6, between the items that we put there is this item "Explore bringing Redis-oplog to core;".

If you read the whole roadmap, you will see "MongoDB Change Streams Support;" so this discussion content was already included in our internal discussions and now we are bringing this to our community. So yes, this discussion from @radekmie is perfect in timing as well.

[StorytellerCZ]

Just a quick update on redis-oplog. MCP has now taken ownership of it.

[radekmie]

I experimented with the MongoDB Change Streams more and... I got some mixed feelings. For once, our history logging service I mentioned in the first post here scaled a lot and it just works, really. We are, however, in the process of moving it to Atlas Database Triggers, but only due to internal reasons.

Now, let's get to the mixed feelings part. I created a script that creates a lot observers (10k without a query and 20k with one). At the same time, it's constantly flooding the database with events by doing single and batch inserts.

index.mjs

import { MongoClient } from 'mongodb';
import { setTimeout } from 'timers/promises';

const client = new MongoClient('mongodb://localhost:27017/changestreams');
const db = client.db();
const xs = db.collection('xs');
const ys = db.collection('ys');

console.time('xs.deleteMany()');
await xs.deleteMany({});
console.timeEnd('xs.deleteMany()');

console.time('ys.deleteMany()');
await ys.deleteMany({});
console.timeEnd('ys.deleteMany()');

let changes = 0;
let streams = 0;
let streamsXs = 0;
let streamsYs = 0;
setInterval(() => {
  const memory = process.memoryUsage();
  const format = x => (x / 1024 / 1024).toFixed(1).padStart(5);
  console.log(
    `${new Date().toJSON().slice(11, -1)} ` +
    `rss: ${format(memory.rss)} ` +
    `total: ${format(memory.heapTotal)} ` +
    `used: ${format(memory.heapUsed)} ` +
    `external: ${format(memory.external)} ` +
    `buffers: ${format(memory.arrayBuffers)} ` +
    `streams: ${streams}/${streamsXs}/${streamsYs} ` +
    `changes: ${changes}`
  );
  changes = 0;
}, 1000);

const id = setInterval(() => {
  if (++streams === 10000) {
    clearInterval(id);
  }

  (async () => {
    const stream = db.watch();
    while (await stream.hasNext()) {
      await stream.next();
      ++changes;
    }
  })();
});

(async () => {
  const id = setInterval(() => {
    if (++streamsXs === 10000) {
      clearInterval(id);
    }

    (async () => {
      const stream = xs.watch([{$match: {'fullDocument.x': {$gt: Math.random() / 10000000}}}]);
      while (await stream.hasNext()) {
        await stream.next();
        ++changes;
      }
    })();
  });

  while (true) {
    await xs.insertOne({ x: Math.random() });
    await setTimeout(10);
  }
})();

(async () => {
  const id = setInterval(() => {
    if (++streamsYs === 10000) {
      clearInterval(id);
    }

    (async () => {
      const stream = ys.watch([{$match: {'fullDocument.y': {$gt: Math.random() / 10000000}}}]);
      while (await stream.hasNext()) {
        await stream.next();
        ++changes;
      }
    })();
  });

  while (true) {
    await ys.insertMany(Array.from({ length: 100 }, () => ({ y: Math.random() })));
    await setTimeout(1000);
  }
})();

docker-compose.yml

version: '3.7'
services:
  mongo:
    image: mongo:${MONGO}
    command: --config /etc/mongod.conf
    ports:
      - 27017:27017
    healthcheck:
      test: |
        if [ -x "$$(command -v mongosh)" ]
        then
          mongosh --eval "rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] })"
        else
          echo "rs.status().ok || rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] }).ok" | mongo
        fi
      interval: 10s
      start_period: 10s
    volumes:
      - ./mongod.conf:/etc/mongod.conf:ro

mongod.conf

cloud:
  monitoring:
    free:
      state: off
net:
  bindIpAll: true
replication:
  oplogSizeMB: 64
  replSetName: rs0
setParameter:
  quiet: 1
systemLog:
  quiet: true

How to run it:

MONGO=4.4.17 docker compose up # Then "5.0.13" and "6.0.2".
node index.mjs # In a separate terminal.

This script logs all of the memory-related stats as well as the number of events processed per second. The entire dataset and charts are in this Google Sheet. The weird part is the memory usage and the throughput (blue, red, and yellow chart are versions 4.4.17, 5.0.13, and 6.0.2 respectively):

As you can see, for some reason, using MongoDB 6 uses ~400MB of RAM more, and the throughput is ~12% lower (partially due to the higher memory usage, but it's a big difference). Of course, it's still a really high throughput - high enough to stay caught up with a database.

I'll look into exposing them for the app code first and only then into using them instead of oplog.