Skip to content

Command-line interface (CLI) and Python library to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk). The calculation of the confidence intervals is done in the same manner as described in (Ribeiro et al., 2011).

License

Notifications You must be signed in to change notification settings

stefantaubert/tts-mos-test-mturk

Repository files navigation

tts-mos-test-mturk

PyPI PyPI MIT PyPI PyPI PyPI DOI

Command-line interface (CLI) to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk).

Features

  • init: initialize project from .json-file
  • masks
    • create: create empty mask
    • mask-workers-by-id: mask workers by their WorkerId
    • mask-workers-by-age-group: mask workers by their age group
    • mask-workers-by-gender: mask workers by their gender
    • mask-workers-by-assignments-count: mask workers by their count of assignments
    • mask-workers-by-masked-ratings-count: mask workers by their count of masked ratings
    • mask-workers-by-correlation: mask workers by their algorithm/sentence correlation
    • mask-workers-by-correlation-percent: mask workers by their algorithm/sentence correlation (percentage-wise)
    • mask-assignments-by-id: mask assignments by their AssignmentId
    • mask-assignments-by-device: mask assignments by their listening device
    • mask-assignments-by-status: mask assignments by their status
    • mask-assignments-by-time: mask assignments by their submit time
    • mask-rating-outliers: mask outlying ratings
    • merge-masks: merge masks together
    • reverse-mask: reverse mask
  • stats
    • print-mos: print MOS and CI95
    • print-masking-stats: print masking statistics
    • print-worker-stats: print worker statistics for each algorithm
    • print-assignment-stats: print assignment statistics for each worker
    • print-sentence-stats: print sentence statistics for each algorithm
    • print-data: print all data points
  • mturk
    • prepare-approval: generate approval file
    • prepare-rejection: generate rejection file
    • prepare-bonus-payment: generate bonus payment file

Installation

pip install tts-mos-test-mturk --user

Usage

usage: mos-cli [-h] [-v] {init,masks,stats,mturk} ...

CLI to evaluate text-to-speech MOS studies done on MTurk.

positional arguments:
  {init,masks,stats,mturk}
                                        description
    init                                initialize project from .json-file
    masks                               masks commands
    stats                               stats commands
    mturk                               mturk commands

options:
  -h, --help                            show this help message and exit
  -v, --version                         show program's version number and exit

Project JSON example

{
  "algorithms": [
    "alg1",
    "alg2",
    "alg3",
    "alg4"
  ],
  "files": [
    "file1",
    "file2",
    "file3"
  ],
  "workers": {
    "worker1": {
      "gender": "male",
      "age_group": "18-29",
      "assignments": {
        "assignment1": {
          "device": "headphone",
          "state": "Approved",
          "hit": "hit1",
          "time": "13.07.23 05:08:04",
          "ratings": [
            {
              "algorithm": "alg1",
              "file": "file1",
              "votes": {
                "naturalness": 3,
                "intelligibility": 5
              }
            },
            {
              "algorithm": "alg2",
              "file": "file3",
              "votes": {
                "naturalness": 2,
                "intelligibility": 4
              }
            }
          ]
        }
      }
    }
  }
}

For a longer example see etc/example.json. It contains 4 algorithms and 120 files which were rated by 36 dummy workers in batches of 10 files per assignment. An example parsing of that file is under etc/example.sh.

Roadmap

  • add masks mask-assignments-not-of-last-month/week/day
  • make device, state, hit and time optional
  • make tax value optional

Dependencies

  • numpy
  • pandas
  • tqdm
  • ordered-set>=4.1.0
  • mean-opinion-score==0.0.2

Contributing

If you notice an error, please don't hesitate to open an issue.

Development setup

# update
sudo apt update
# install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be run
sudo apt install python3-pip \
  python3.8 python3.8-dev python3.8-distutils python3.8-venv \
  python3.9 python3.9-dev python3.9-distutils python3.9-venv \
  python3.10 python3.10-dev python3.10-distutils python3.10-venv \
  python3.11 python3.11-dev python3.11-distutils python3.11-venv
# install pipenv for creation of virtual environments
python3.8 -m pip install pipenv --user

# check out repo
git clone https://github.com/stefantaubert/tts-mos-test-mturk.git
cd tts-mos-test-mturk
# create virtual environment
python3.8 -m pipenv install --dev

Running the tests

# first install the tool like in "Development setup"
# then, navigate into the directory of the repo (if not already done)
cd tts-mos-test-mturk
# activate environment
python3.8 -m pipenv shell
# run tests
tox

Final lines of test result output:

  py38: commands succeeded
  py39: commands succeeded
  py310: commands succeeded
  py311: commands succeeded
  congratulations :)

License

MIT License

Acknowledgments

  • Ribeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419. https://doi.org/10.1109/ICASSP.2011.5946971

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410

Citation

If you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see About => Cite this repository).

Taubert, S. (2024). tts-mos-test-mturk (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.10461539

Changelog

  • v0.0.2 (2024-01-05)
    • Bugfix:
      • mask mask-workers-by-correlation-percent: sorting was not always correct
    • Added:
      • mask mask-workers-by-correlation-percent: added option to include masked workers in percentage calculation
      • mturk prepare-bonus-payment: added logging of fees for Mechanical Turk
      • mturk prepare-approval: added logging of fees for Mechanical Turk
      • added mask mask-assignments-by-status
      • added mask mask-assignments-by-time
      • added parsing of HITId
      • added option to mask assignments before preparing rejection CSV
      • added ! before mask name reverses mask on input
      • added masks mask-assignments-by-id
      • added masks mask-workers-by-id
    • Changed:
      • moved template creation and preparation to another repository
      • removed worktime parsing
  • v0.0.1 (2023-02-23)
    • Initial release

About

Command-line interface (CLI) and Python library to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk). The calculation of the confidence intervals is done in the same manner as described in (Ribeiro et al., 2011).

Topics

Resources

License

Stars

Watchers

Forks