transformers/docs/source/en/model_doc/bert-generation.md

This model was released on 2019-07-29 and added to Hugging Face Transformers on 2020-11-16 and contributed by patrickvonplaten.

BertGeneration

BertGeneration leverages pre-trained BERT checkpoints for sequence-to-sequence tasks using an EncoderDecoderModel framework. This approach achieves state-of-the-art results in Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion, demonstrating the utility of initializing both encoder and decoder with pre-trained models.

import torch
from transformers import pipeline

pipeline = pipeline(task="text2text-generation", model="google/bert_for_seq_generation_L-24_bbc_encoder", dtype="auto")
pipeline("Plants generate energy through a process known as  ")

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("google/bert_for_seq_generation_L-24_bbc_encoder", dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("google/bert_for_seq_generation_L-24_bbc_encoder")

inputs = tokenizer("Plants generate energy through a process known as  ", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

Usage tips

• Use [BertGenerationEncoder] and [BertGenerationDecoder] with [EncoderDecoderModel] for sequence-to-sequence tasks.
• Summarization, sentence splitting, sentence fusion, and translation don't require special tokens in the input.
• Don't add EOS tokens to the end of inputs for most generation tasks.

BertGenerationConfig

autodoc BertGenerationConfig

BertGenerationTokenizer

autodoc BertGenerationTokenizer - save_vocabulary

BertGenerationEncoder

autodoc BertGenerationEncoder - forward

BertGenerationDecoder

autodoc BertGenerationDecoder - forward