Annotations for certain documents don't export

[chlorophilo]

Debug log ID

F58HDAMN-refs-euc/6.7.178-6

What happened?

Hi Retorquere, and thank you for this invaluable extension.

Most of the time I have no problem with the export of data using Better Bibtex. Once before, though I tried to export annotations from a paper and I just could not get it to work. I decided there must have been something buggy and left it alone. But it has just happened again with a different paper, and I really need to transfer these annotations.

As I understand the annotations are stored separately from the pdf, so I'm reluctant to believe it is something about the source document format. Can you think of any other issue that might be document specific like this?

Best, Chlorophilo

[retorquere]

Zotero doesn't disclose the annotations to the exporter environment. I can see whether I can work around that, but this is really something zotero should fix.

[chlorophilo]

Thanks for getting back to me so quickly. Most of my annotations come through on export just fine. Why might one document produce problems on export?

[retorquere]

Depends on what you mean with that. It is possible that Zotero adds the annotations to the pdf during export, but the annotations are not currently available for exporter that generates the bibtex output. If the annotations appear in the pdf, that js done outside the scope of control of the bibtex-generating code.

[retorquere]

You can export items using BetterBibTeX JSON to see what the exporters themselves have access to. If the annotations aren't in there, I'd have to work around zotero to smuggle them in.

[chlorophilo]

I am not exporting the pdf, just the annotations. I am exporting them into a research note in Obsidian.

I use BetterBibtex with mgmeyers/obsidian-zotero-integration. When you look at the Zotero Data Explorer for seeing what gets exported from this one document there are no annotations, even though when exporting from other documents I get the relevant annotations. On mgmeyers' FAQ page in Github he says "There might be a bug in Bibtex or the Zotero Integration plugin. Zotero has the annotation but its not seen in Obsidian. Please reference https://github.com/mgmeyers/obsidian-zotero-integration/issues/107There might be a bug in Bibtex or the Zotero Integration plugin. Zotero has the annotation but its not seen in Obsidian. Please reference mgmeyers/obsidian-zotero-integration#107".

[retorquere]

Most of my annotations come through on export just fine.

How precisely are you exporting, what do you mean by "most", and where do you see these annotations after export?

I am exporting them into a research note in Obsidian.

I don't know how you are doing this.

When you look at the Zotero Data Explorer

I don't know what the Zotero Data Explorer is

seeing what gets exported from this one document there are no annotations, even though when exporting from other documents I get the relevant annotations.

I have no idea what you are describing here.

Zotero has the annotation but its not seen in Obsidian

I don't actually know how the obsidian plugin calls BBT. If the developer of the Obsidian plugin says what it needs I may be able to help, but until the dev does, there's nothing for me to fix. If the obsidian plugin is using BetterBibTeX JSON exports, it is a know problem that you don't get annotations, because Zotero doesn't provide them to exporters.

[chlorophilo]

The set up is this:

I collect articles in Zotero.
I annotate them.
I do all my research work in a markdown editor called Obsidian.
When I have finished annotating a pdf in Zotero I need to transfer the annotations (not the pdf - the two things are separate in Zotero) into Obsidian.
I can do this because Obsidian has a community plugin called Zotero Integration, built by MGMeyers. That plugin requires that Better Bibtex be installed otherwise it cannot work, so I have Better Bibtex installed on Zotero. It was because of that requirement and the community nature of GitHub that I assumed you might know of its existence. Sorry about that.
I run the Zotero Integration plugin, giving the citation key for the paper for which I want the annotations. Zotero Integration creates a new markdown note in Obsidian (called a "research note"). The header of the research note includes basic information about the cited research, and then all the annotations follow in the body of the research note.
Zotero Integration uses templates written within Obsidian. These templates lay out the annotations in the body of the research note in a really nice way that makes the imported annotations much easier to read than they appear in a note created in Zotero using the 'Add Note from Annotations' menu item.
I have run the Zotero Integration plugin dozens of times to create research notes in Obsidian based on annotations pulled from Zotero. In every instance except for one (a year ago) and the one I just reported to you the annotations have been successfully imported into a research note in Obsidian.
Zotero Data Explorer is a feature of the Zotero Integration plugin. It pulls up a list of all the fields available from Zotero for the Zotero item in question. It provides a useful check when you are doing templating in Obsidian because you can make sure the fields you want are actually available, and it helps you debug some templating errors.
Normally, when you run the Zotero Data Explorer and select the Zotero item in question, the Data Explorer output provides, among all the other fields, all the annotations in full. For the two Zotero items I mentioned, it is stated in the Data Explorer's report that there are no annotations, which is false.
As MGMeyers provides both the Zotero Integration plugin and the Zotero Data Explorer, he was pointing the finger at his own creation as the possible culprit as well as yours. I don't think he did so with bad intent. I think he is just saying that he does not quite know where the problem is.

Perhaps you can help work this out.

[retorquere]

Given your description, I think he calls into one of the JSON-RPC methods that BBT offers. We can't realistically diagnose this with you as a go-between. If MGMeyers can replicate your problem, have him get in touch, and we can get to a diagnosis.

[github-actions]

🤖 this is your friendly neighborhood build bot announcing test build 6.7.182.2856.6180 ("logging for #2856")

This update may name other issues, but the build just dropped here is for you; it just means problems already fixed in other issues have been folded into the work we are doing here. Install in Zotero by downloading test build 6.7.182.2856.6180, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[retorquere]

Please install build 6180, reproduce the problem, and send a new log.

[retorquere]

closing for inactivity.

[FeralFlora]

Given your description, I think he calls into one of the JSON-RPC methods that BBT offers

I mean, mgmeyers has done about 10 merged pull-requests on Better Bibtex's JSON-RPC method to enable / enhance this workflow, so it should be apparent which method he / Zotero Integration uses: https://github.com/retorquere/zotero-better-bibtex/pulls?q=is%3Apr+author%3Amgmeyers

[retorquere]

I'm not sure what you want me to do with this information. I didn't remember these pull requests, but I don't see how they're relevant; I have no idea what @mgmeyers does with the information picked up from these endpoints, so until we have the problem reproducible and @mgmeyers involved, there's not much I can do. I could have done some prelim diagnosis with @chlorophilo, but that conversation has stalled, so I've closed the issue.

If you have something I can act on, I'll be glad to, but I don't currently know what.

[retorquere]

@chlorophilo we were talking past each other -- you were talking about things showing up in the PDF attachments, which translators (like BBTs) leave to Zotero; I thought you were talking about the content of the exported items themselves, not the attachments, and in the exported items themselves, annotation data is not accessible.

So I can try and help with diagnosis, but it would mainly be to establish how it can be that annotations don't show up in PDFs when exported using BBT, where BBT for all attachments, like all translators, just tells Zotero "save it to this path" and then gets out the way.

[FeralFlora]

I'm not sure what you want me to do with this information.

Nothing as such, I was just surprised by the lack of co-awareness, given the past interactions and contributions.
And, since Zotero Integration has tens of thousands of users (almost 140.000 downloads of version 3.1.7), and every one of those has Better Bibtex installed, you're going to keep bumping into us "Obsidianites" 😄

[retorquere]

I don't mind bumping into users, I'm happy my software is useful and grateful for pull requests, but the last interaction on this was January, and it simply slipped out of my focus after that. My main focus is still bibtex export, the integrations sort of emerged as a side effect.

[retorquere]

On Thu, 16 May 2024 at 07:02, appwcn ***@***.***> wrote: Depends on what you mean with that. It is possible that Zotero adds the annotations to the pdf during export, but the annotations are not currently available for exporter that generates the bibtex output. If the annotations appear in the pdf, that js done outside the scope of control of the bibtex-generating code. I see why is the issue. So what you mean is that better-bibtex does not export annotations, so maybe the author of Obsidian Zotero Integration use its unique way to export annotations.

No, I thought the discussion is about what output BBT generates, not the attachments — BBT does not do anything with attachments other than handing a list of names of attachments to Zotero to save them on its behalf, so I thougth we were talking about the output BBT generates itself.

…

Message ID: ***@***.*** .com>