transformers/docs/source/en/model_doc/fnet.md

This model was released on 2021-05-09 and added to Hugging Face Transformers on 2021-09-20 and contributed by gchhablani.

FNet

FNet: Mixing Tokens with Fourier Transforms demonstrates that Transformer encoders can be significantly accelerated by replacing self-attention layers with simple linear mixers or even an unparameterized Fourier Transform. This FNet approach achieves 92–97% of BERT’s accuracy on the GLUE benchmark while training 80% faster on GPUs and 70% faster on TPUs at standard input lengths. On longer sequences, FNet maintains competitive accuracy compared to efficient Transformers while being faster across most sequence lengths. Additionally, FNet has a smaller memory footprint, making it especially efficient for smaller models, which can outperform Transformer counterparts under the same speed and accuracy constraints.

import torch
from transformers import pipeline

pipeline = pipeline(task="fill-mask", model="google/fnet-base", dtype="auto")
pipeline("Plants create [MASK] through a process known as photosynthesis.")

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("google/fnet-base", dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("google/fnet-base")

inputs = tokenizer("Plants create [MASK] through a process known as photosynthesis.", return_tensors="pt")
outputs = model(**inputs)
mask_token_id = tokenizer.mask_token_id
mask_position = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_word = tokenizer.decode(outputs.logits[0, mask_position].argmax(dim=-1))
print(f"Predicted word: {predicted_word}")

Usage tips

• FNet doesn't use attention masks since it's based on Fourier Transform. The model trained with maximum sequence length 512 (including pad tokens). Use the same maximum sequence length for fine-tuning and inference.

FNetConfig

autodoc FNetConfig

FNetTokenizer

autodoc FNetTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary

FNetTokenizerFast

autodoc FNetTokenizerFast

FNetModel

autodoc FNetModel - forward

FNetForPreTraining

autodoc FNetForPreTraining - forward

FNetForMaskedLM

autodoc FNetForMaskedLM - forward

FNetForNextSentencePrediction

autodoc FNetForNextSentencePrediction - forward

FNetForSequenceClassification

autodoc FNetForSequenceClassification - forward

FNetForMultipleChoice

autodoc FNetForMultipleChoice - forward

FNetForTokenClassification

autodoc FNetForTokenClassification - forward

FNetForQuestionAnswering

autodoc FNetForQuestionAnswering - forward