<!--Copyright 2021 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->
*This model was released on 2020-02-12 and added to Hugging Face Transformers on 2023-06-20 and contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).*

# T5v1.1

[T5v1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) builds on the T5 architecture with several key modifications. It replaces the standard ReLU activation in the feed-forward layers with GEGLU, which can improve learning efficiency. Dropout was disabled during pre-training on the C4 dataset, enhancing model quality, though it should be used during fine-tuning. The model does not share parameters between the embedding and classifier layers, and pre-training was done exclusively on C4 without mixing in downstream tasks.

<hfoptions id="usage">
<hfoption id="Pipeline">

```py
import torch
from transformers import pipeline

pipeline = pipeline(task="text2text-generation", model="google/t5-v1_1-base", dtype="auto",)
pipeline("translate English to French: Plants create energy through a process known as photosynthesis.")
```

</hfoption>
<hfoption id="AutoModel">

```py
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/t5-v1_1-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-v1_1-base", dtype="auto",)

inputs = tokenizer("translate English to French: Plants create energy through a process known as photosynthesis.", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
```

</hfoption>
</hfoptions>

## Usage tips

- T5 Version 1.1 was only pre-trained on C4 without supervised training. Fine-tune the model before using it on downstream tasks (unlike the original T5 model).
- Since T5v1.1 was pre-trained unsupervised, task prefixes don't help during single-task fine-tuning.
- Use task prefixes for multi-task fine-tuning.