mirror of
https://github.com/huggingface/transformers.git
synced 2025-10-20 17:13:56 +08:00
75 lines
2.7 KiB
Markdown
75 lines
2.7 KiB
Markdown
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
*This model was released on 2019-07-29 and added to Hugging Face Transformers on 2020-11-16 and contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).*
|
|
|
|
# BertGeneration
|
|
|
|
[BertGeneration](https://huggingface.co/papers/1907.12461) leverages pre-trained BERT checkpoints for sequence-to-sequence tasks using an EncoderDecoderModel framework. This approach achieves state-of-the-art results in Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion, demonstrating the utility of initializing both encoder and decoder with pre-trained models.
|
|
|
|
<hfoptions id="usage">
|
|
<hfoption id="Pipeline">
|
|
|
|
```py
|
|
import torch
|
|
from transformers import pipeline
|
|
|
|
pipeline = pipeline(task="text2text-generation", model="google/bert_for_seq_generation_L-24_bbc_encoder", dtype="auto")
|
|
pipeline("Plants generate energy through a process known as ")
|
|
```
|
|
|
|
</hfoption>
|
|
<hfoption id="AutoModel">
|
|
|
|
```py
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("google/bert_for_seq_generation_L-24_bbc_encoder", dtype="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("google/bert_for_seq_generation_L-24_bbc_encoder")
|
|
|
|
inputs = tokenizer("Plants generate energy through a process known as ", return_tensors="pt")
|
|
outputs = model.generate(**inputs, max_length=50)
|
|
print(tokenizer.decode(outputs[0]))
|
|
```
|
|
|
|
</hfoption>
|
|
</hfoptions>
|
|
|
|
## Usage tips
|
|
|
|
- Use [`BertGenerationEncoder`] and [`BertGenerationDecoder`] with [`EncoderDecoderModel`] for sequence-to-sequence tasks.
|
|
- Summarization, sentence splitting, sentence fusion, and translation don't require special tokens in the input.
|
|
- Don't add `EOS` tokens to the end of inputs for most generation tasks.
|
|
|
|
## BertGenerationConfig
|
|
|
|
[[autodoc]] BertGenerationConfig
|
|
|
|
## BertGenerationTokenizer
|
|
|
|
[[autodoc]] BertGenerationTokenizer
|
|
- save_vocabulary
|
|
|
|
## BertGenerationEncoder
|
|
|
|
[[autodoc]] BertGenerationEncoder
|
|
- forward
|
|
|
|
## BertGenerationDecoder
|
|
|
|
[[autodoc]] BertGenerationDecoder
|
|
- forward
|