Description

Implementation of language models under a new art.estimators.language_modeling submodule. Currently only Hugging Face language models using a PyTorch back-end have been implemented. This is implemented as the HuggingFaceLanguageModel which is a generic estimator that is able to run basic functionality on any Hugging Face model.

This new language model estimator takes in a Hugging Face model and tokenizer and acts as a basic ART wrapper for now until attacks and defenses for language models are implemented. Currently the estimator only supports the following tasks:

• Tokenization to be fed into the model
• Encoding strings to tokens
• Decoding tokens to strings
• Running inference on the model using a string input (with auto tokenization)
• Running text generation on the model using a string input (with auto tokenization)

Inference on the estimator will simply return the output dictionary from running inference on the HuggingFace model. The estimator currently does not support training or loss gradients as these are more complex features that will be added later. Once this PR is merged in, additional issues will be created to implement training and loss gradients which will be done as separate PRs.

A demo notebook in notebooks/hugging_face_language_model.ipynb was created to illustrate the usage.

Fixes #2336

Type of change

Please check all relevant options.

• Improvement (non-breaking)
• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

• Unit tests for the HuggingFaceLanguageModel estimator.

Test Configuration:

• OS
• Python version
• ART version or commit number
• TensorFlow / Keras / PyTorch / MXNet version

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• My changes have been tested using both CPU and GPU devices