Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch
-
Updated
Apr 10, 2023 - Python
Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch
My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)
RAN: Recurrent Attention Networks for Long-text Modeling | Findings of ACL23
LongQLoRA: Extent Context Length of LLMs Efficiently
[DEPRECIATED] Very fast, large music transformer with 8k sequence length, efficient heptabit MIDI notes encoding, true full MIDI instruments range, chords counters and outro tokens
Streamlined variant of Long-Range Arena with pinned dependencies, automated data downloads, and deterministic shuffling.
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
LooGLE: Long Context Evaluation for Long-Context Language Models
Finetuning and evaluating LLMs to extract GHG emissions from PDF reports using RAG and grammar-based decoding.
Implementation of paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"
needle in a haystack for LLMs
The official repo for "LLoCo: Learning Long Contexts Offline"
The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The official implementation of "Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks"
LongAlign: A Recipe for Long Context Alignment Encompassing Data, Training, and Evaluation
Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
Add a description, image, and links to the long-context topic page so that developers can more easily learn about it.
To associate your repository with the long-context topic, visit your repo's landing page and select "manage topics."