Is it possible to not update the updated_at value for highlights that have already been imported?

[BrandonNoad]

Hi.

Is it possible to not update the updated_at value for highlights that have already been imported?

I assume the updated_at value is set by the Readwise API and is not something October is able to control. But, updating the updated_at value for existing highlights every time you perform a sync conflicts with the recommended export method (GET https://readwise.io/api/v2/export/):

The recommended way to use this endpoint is to first sync all of a user's historical data by passing no parameters on the first request, then pageCursor until there are no pages left. Then later, if you want to pull newly updated highlights, just pass updatedAfter as the time you last pulled data.

Changing the updated_at value for highlights that didn't actually update causes them to be re-exported and results in duplicates at my export destination.

The Readwise API docs say:

Finally, we de-dupe highlights by title/author/text/source_url. So if you send a highlight with those 4 things the same (including nulls) then it will do nothing rather than create a "duplicate".

Do you know if the above still updates updated_after?

[marcus-crane]

Hi there,

Sorry for the late reply, I just got back from a vacation so I'm catching up on October related messages.

I just had a quick look and as far as I can tell, nothing October does sets the updated_at field.

The Highlight CREATE method states

Finally, we de-dupe highlights by title/author/text/source_url. So if you send a highlight with those 4 things the same (including nulls) then it will do nothing rather than create a "duplicate".

which is what October has been relying on but perhaps it's the case that this isn't quite true and the updated_at field is being altered each time.

I'll do some experimenting to see if this holds true and if so, I can reach out to the Readwise developers to confirm if that behaviour is expected or not.

This sort of falls into #16 territory where keeping a sync log rather than relying on the API to do deduplication would probably help a lot in cases like this but it's distinct enough that if I can replicate the behaviour, I'll create a Github issue to track it.

Thanks for raising this for discussion. I hadn't used the export API for anything myself so I wouldn't have noticed otherwise!

[marcus-crane]

Hey @BrandonNoad,

Just confirming that I've been able to replicate this behaviour and it seems like an oversight in the Readwise API.

I've raised it with the Readwise developers (I'm part of a developer channel that has some community project maintainers as well as members of the Readwise team) and I'll post any updates on the ticket I'm about to create.

Thanks for pointing it out

[marcus-crane]

#81

[BrandonNoad]

Hi Marcus.

Thanks for looking in to this. Hopefully Readwise will fix their API implementation for the no-op scenario.

The official Readwise Kobo import integration does not work at all for me, so thanks for building/maintaining October.

[marcus-crane]

Hi Brandon,

Just to confirm, were you relying on October for any store-purchased titles as well, given you mentioned the official integration?

I recently released a new version that ignores store-purchased titles which were never officially supported but just happened to sort of work accidentally.

While the official Readwise import integration is recommended for store-purchased titles (and doesn't work with sideloaded titles), if you were using it because the official integration didn't work for store titles then I can add a toggle to enable uploading store titles with October again.

The official Readwise integration was never intended (or expected) to work with sideloaded titles though as they can only be accessed locally.

[BrandonNoad]

were you relying on October for any store-purchased titles as well

Yes. The official integration doesn't work for me. It is never able to find any highlights. So if you can add that toggle, that would be great.

Thanks.