Monstache starts backoff when getting 404 (deleted object is already deleted in ES)

[ashprojects]

The Problem:
We delete document both from Mongo and ES for some real time use cases. Mongo delete events are being synced by Monstache to ES, ES returns 404 as document Monstache is trying to delete is already deleted. In this case, this error was ignored safely (~6.7.14). Monstache from 6.7.15 starts treating this as failures and tries the newly implemented backoff. So no more changes are synced, and it is dead.

Detail:
In recent version of monstache we started getting backoff for cases where document is not found in ES (As we have a use case when we directly delete from ES as well some times for some specific business use cases)
Logs:
Log 1:

    "_index": "collection_01-09-2023",
    "_id": "5d631d37-0d58-4bb6-99a9-c5d55c1a6377",
    "_version": 7313604709745426453,
    "result": "not_found",
    "_shards": {
        "total": 2,
        "successful": 2,
        "failed": 0
    },
    "_seq_no": 442740031,
    "_primary_term": 4,
    "status": 404
}

Next log, immediately after:

INFO 2023/12/18 06:26:46 Backing off for 3.9 minutes after bulk indexing failures.
--

Now monstache won't sync and we are stuck.
Earlier this was ignored as per our knowledge.

We are using latest version of monstache, updated yesterday.
This seems to cause it. v6.7.14...v6.7.15

[hanuhimanshu]

Same here. Looks like backoff logic has changed in 6.7.15, which by default assums backoff if ES gave non 200 response.

[kartik-pawar]

We downgraded to 6.7.14 for now