transformers/docs/source/en/model_doc/prophetnet.md

This model was released on 2020-01-13 and added to Hugging Face Transformers on 2020-11-16.

ProphetNet

ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training is an encoder-decoder model that employs future n-gram prediction and an n-stream self-attention mechanism. Unlike traditional models that predict the next token, ProphetNet predicts the next n tokens simultaneously, enhancing its ability to plan for future tokens and reducing overfitting on local correlations. Pre-trained on both base (16GB) and large (160GB) datasets, ProphetNet outperforms other models on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks.

import torch
from transformers import pipeline

pipeline = pipeline(task="text-generation", model="microsoft/prophetnet-large-uncased", dtype="auto",)
pipeline("Plants create energy through a process known as photosynthesis.")

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased")
model = AutoModelForCausalLM.from_pretrained("microsoft/prophetnet-large-uncased", dtype="auto",)

inputs = tokenizer("Plants create energy through a process known as photosynthesis.", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

Usage tips

• Pad inputs on the right. ProphetNet uses absolute position embeddings.
• The model architecture is based on the original Transformer. It replaces the "standard" self-attention mechanism in the decoder with a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.

ProphetNetConfig

autodoc ProphetNetConfig

ProphetNetTokenizer

autodoc ProphetNetTokenizer

ProphetNet specific outputs

autodoc models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqLMOutput

autodoc models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqModelOutput

autodoc models.prophetnet.modeling_prophetnet.ProphetNetDecoderModelOutput

autodoc models.prophetnet.modeling_prophetnet.ProphetNetDecoderLMOutput

ProphetNetModel

autodoc ProphetNetModel - forward

ProphetNetEncoder

autodoc ProphetNetEncoder - forward

ProphetNetDecoder

autodoc ProphetNetDecoder - forward

ProphetNetForConditionalGeneration

autodoc ProphetNetForConditionalGeneration - forward

ProphetNetForCausalLM

autodoc ProphetNetForCausalLM - forward