Multilingual bibliography #126

nrydanov · 2024-01-11T17:58:00Z

Currently, there's no any support for language entry in CSL or BibLatex. This PR can add possibility to parse and use multiple languages in single bibliography, depending on the value of language parameter for each entry.

src/csl/mod.rs

src/csl/rendering/mod.rs

src/csl/rendering/names.rs

reknih · 2024-01-24T10:32:01Z

src/lang/codes.rs

+lazy_static! {
+ static ref LANGUAGE_CODE_MAPPING: HashMap<&'static str, &'static str> =
+ HashMap::from([
+ ("english", "en"),
+ ("german", "ge"),
+ ("french", "fr"),
+ ("russian", "ru"),
+ ("italian", "it"),
+ ("chinese", "cn"),
+ ("japanese", "jp"),
+ ("ukranian", "ua")
+ ]);
+}
+
+/// This function returns mapping for required language
+pub fn get_mapping(s: &str) -> Option<&str> {
+ return LANGUAGE_CODE_MAPPING.get(s).copied();
+}


This mapping seems incomplete and strangely used. As I see it, it should only be required for BibLaTeX as CSL defines its language field in terms of ISO-639-1 codes with regions. We interpret this as RFC 5646 language tags.

According to the BibLaTeX manual, the langid entry (p. 28) controls the language of a citation. Its values shall be either a Babel/Polyglossia language tag which includes RFC 5646 language tags. The normalization to RFC 5646 belongs in the biblatex crate from where we then can create 1:1 LocaleCode structs in the interop module. You may want to use an external dependency in BibLaTeX for the language names.

Ok, will change this soon.

Could you please help me understand how and where you convert LanguageIdentifier into LocaleCode that is used everywhere?

UPD. I found out that CLI doesn't make this conversion at all. Currently, I've added following lines in reference command parsing

for entry in &bibliography { let mut item = CitationItem::with_entry(entry); item.locale = Some(LocaleCode(String::from( entry.language().unwrap().language.as_str(), ))); driver.citation(CitationRequest::new( vec![item], &style, locale.clone(), &locales, None, )) }

It works, but I can't truly understand where should this conversion actually be done. It seems that it should happen somewhere earlier than this place.

src/lang/mod.rs

tests/citeproc.rs

nrydanov · 2024-04-15T18:58:58Z

Hello anyone?...

reknih

While reviewing this PR, I have investigated how this fits into the CSL ecosystem.

citeproc.js and its use in Zotero is the reference implementation for CSL. When using Zotero to generate a bibliography, the app will ask the user to explicitly choose a language for that bibliography and will ignore the language field of each entry.

This corresponds with our current behavior of using the locale in the BibliographyRequest. Merging this PR would lead to a divergence between Hayagriva and Zotero/citeproc.js and violate the CSL spec.

There is an extension to CSL called CSL-M which aims, amongst other things, to accommodate multilingual bibliographies. In it, a CSL-M style file may contain multiple locale-specific layout attributes for bibliographies and citations as well as a fallback locale. Consider the following example:

<bibliography>
  <layout suffix="." locale="da de">
    <text macro="bibliography" />
  </layout>
  <layout suffix=".">
    <text macro="bibliography" />
  </layout>
</bibliography>

This file changes how citeproc.js retrieves term localizations:

An item with the language da-DK would be rendered with the locale="da de" layout and receive Danish terms.
An item with the language de-DE would be rendered with the locale="da de" layout but use Danish terms as well. This is because the locale attribute defines a fallback order for terms locales for all entries rendered with it.
An item with the language jp-JP does not match any locale-specific layouts and would be rendered with the locale-less last layout. Assuming the user specified their default locale as en-US in the dropdown, the citation would not use Japanese but American English terms.

You can try this using the Juris-M fork of Zotero, previously known as Multilingual Zotero (MLZ) that focusses on CSL-M support. This behavior requires the citation processor to supply multiple styles, one of which has multiple layouts (known as polyglot) and one with just one layout. You can see this in the CSL-M style repo, for example with jm-chicago-fullnote-bibliography-polyglot.csl and jm-chicago-fullnote-bibliography.csl.

I do not personally find the CSL-M implementation the best since it requires a proliferation of styles as well as manually listing each supported language in a style. However, I also do not think that we should unconditionally choose to violate the spec. A configuration option to always prefer the terms from the language field of an entry in BibliographyRequest is a possible solution, another solution is to implement CSL-M which would be a larger effort and require changes in citationberg.

What would you think to be the best course of action? Let's discuss it here before you push more code.

On a different note I noticed that this PR still contains formatting changes of unchanged code which I cannot reproduce. Please revert them to maintain a clean diff!

reknih · 2024-04-16T08:26:31Z

src/csl/mod.rs

@@ -105,12 +105,17 @@ impl<'a, T: EntryLike + Hash + PartialEq + Eq + Debug> BibliographyDriver<'a, T>
 let mut entry_set = IndexSet::new();


Suggested change

let mut entry_set = IndexSet::new();

let mut item_set = IndexSet::new();

reknih · 2024-04-16T08:34:49Z

src/main.rs

@@ -320,8 +320,11 @@ fn main() {

 let mut driver = BibliographyDriver::new();
 for entry in &bibliography {
+ let mut item = CitationItem::with_entry(entry);
+ let id = entry.language().unwrap_or_default();
+ item.locale = Some(LocaleCode(String::from(id.language.as_str())));


Instead of doing this, CitationItem::with_entry should use EntryLike's resolve_standard_variable feature to set the CitationItem's locale.

nrydanov · 2024-04-17T09:28:14Z

Thanks for your answer!

I will study your investigation and reply a little bit later. Currently, I definitely have poor domain knowledge.

nrydanov · 2024-04-17T16:55:56Z

Ok, I got it.

Well, I think that the best idea for now is to just add configuration option. Let me explain why:

First of all, the primary goal of this PR is to just add support for language entries.
I would consider thinking about big changes in project only if
- it's clear that those changes would make things much better
- here's enough time and resources to do it
- when a person, who realize this feature, completely understand what he does

I wouldn't say that this situation matches at least one of this points, because

you said that you personally "do not find CSL-M implementation the best" and there's no person in this project, who can judge it better
I spend my free time doing this PR, so I'm not sure that I will be ready for such hard work, at least alone
I need to do much more research in case we want to achieve smth bigger than small feature

Also, I find the idea of making configuration option as good one, because personally I didn't care of any CSL specs before this day and as end-user I don't care about it at all.

nrydanov · 2024-04-17T16:56:52Z

I'll also review changes in formatting and revert them. Still can't understand why do we have different formatting rules ;)

I'll also squash commits when PR is done.

reknih · 2024-05-08T14:02:28Z

Sounds good. I'll review the change once it hits this PR.

nrydanov added 2 commits January 11, 2024 18:40

[WIP] Added localization

6dbb983

Added support for language parameter

4f7bf0a

nrydanov force-pushed the feature/language-specific-terms branch from 485f99e to 4f7bf0a Compare January 11, 2024 22:17

nrydanov changed the title ~~Multimodal bibliography entries~~ Multilangual bibliography Jan 11, 2024

nrydanov mentioned this pull request Jan 11, 2024

Multilingual bibliography typst/typst#3177

Closed

nrydanov changed the title ~~Multilangual bibliography~~ Multilingual bibliography Jan 12, 2024

nrydanov added 2 commits January 12, 2024 11:34

Added docs

60d4394

Changes in naming

41b32bc

nrydanov marked this pull request as ready for review January 12, 2024 08:28

reknih reviewed Jan 24, 2024

View reviewed changes

reknih force-pushed the main branch from b278652 to b3c1970 Compare March 7, 2024 14:48

laurmaedje marked this pull request as draft March 23, 2024 11:48

nrydanov added 4 commits March 24, 2024 15:53

Merged actual main

95bdd44

Returned back lazy_static dependency

d5ba5a5

Propagated item language settings to all lookup_locale calls

7c8bf24

Added specific language support for each entry in CLI

dba2644

nrydanov marked this pull request as ready for review March 24, 2024 13:43

nrydanov requested a review from reknih March 24, 2024 13:44

nrydanov added 8 commits March 24, 2024 17:44

Reverted .gitignore changes

596de0e

Returned back hash deriving

3b3018b

Added missing term locale

1893545

Removed redundant term argument

a64beca

Removed lazy_static dependency

06f0dae

Removed language codes

c193bbf

Fixed tests

ac426e7

Made identifier retrieval more safe

9c496b5

reknih reviewed Apr 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multilingual bibliography #126

Multilingual bibliography #126

nrydanov commented Jan 11, 2024 •

edited

reknih Jan 24, 2024

nrydanov Mar 3, 2024

nrydanov Mar 24, 2024 •

edited

nrydanov commented Apr 15, 2024

reknih left a comment

reknih Apr 16, 2024

reknih Apr 16, 2024

nrydanov commented Apr 17, 2024

nrydanov commented Apr 17, 2024 •

edited

nrydanov commented Apr 17, 2024 •

edited

reknih commented May 8, 2024

		@@ -105,12 +105,17 @@ impl<'a, T: EntryLike + Hash + PartialEq + Eq + Debug> BibliographyDriver<'a, T>
		let mut entry_set = IndexSet::new();

	let mut entry_set = IndexSet::new();
	let mut item_set = IndexSet::new();

Multilingual bibliography #126

Are you sure you want to change the base?

Multilingual bibliography #126

Conversation

nrydanov commented Jan 11, 2024 • edited

reknih Jan 24, 2024

Choose a reason for hiding this comment

nrydanov Mar 3, 2024

Choose a reason for hiding this comment

nrydanov Mar 24, 2024 • edited

Choose a reason for hiding this comment

nrydanov commented Apr 15, 2024

reknih left a comment

Choose a reason for hiding this comment

reknih Apr 16, 2024

Choose a reason for hiding this comment

reknih Apr 16, 2024

Choose a reason for hiding this comment

nrydanov commented Apr 17, 2024

nrydanov commented Apr 17, 2024 • edited

nrydanov commented Apr 17, 2024 • edited

reknih commented May 8, 2024

nrydanov commented Jan 11, 2024 •

edited

nrydanov Mar 24, 2024 •

edited

nrydanov commented Apr 17, 2024 •

edited

nrydanov commented Apr 17, 2024 •

edited