Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

High performance unsupervised text tokenization for Ruby

R package for Byte Pair Encoding based on YouTokenToMe

Kotlin multiplatform BPE tokenizer library for OpenAI models

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

Feature extraction from sequential data

Генерация новостных заголовков

Byte Pair Encoding (BPE)

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

Auto summarization from BPE tokenization

Code for the publication of WWW'22

an efficient ranked retrieval system for English corpora, optimised with VBE and BPE.

Modern Eager TensorFlow implementation of Attention Is All You Need

An Introduction to Natural Language Processing (NLP)

Byte-pair encoding implementation in Python.

Named entity recognition in Malayalam using BiLSTM and TENER (Transformer Encoder)

This project aims to implement word-based, character-based and subword-based tokenization techniques.

Byte pair encoding tokenizer as used in some large language models.