transformers/docs/source/en/model_doc/florence2.md

This model was released on 2024-06-16 and added to Hugging Face Transformers on 2025-08-20.

Florence-2

Overview

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages the FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.

You can find all the original Florence-2 checkpoints under the Florence-2 collection.

Tip

This model was contributed by ducviet00. Click on the Florence-2 models in the right sidebar for more examples of how to apply Florence-2 to different vision and language tasks.

The example below demonstrates how to perform object detection with [Pipeline] or the [AutoModel] class.

import torch
import requests
from PIL import Image
from transformers import pipeline

pipeline = pipeline(
    "image-text-to-text",
    model="florence-community/Florence-2-base",
    device=0,
    dtype=torch.bfloat16
)

pipeline(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true",
    text="<OD>"
)

import torch
import requests
from PIL import Image
from transformers import AutoProcessor, Florence2ForConditionalGeneration

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

model = Florence2ForConditionalGeneration.from_pretrained("microsoft/Florence-2-base", dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base")

task_prompt = "<OD>"
inputs = processor(text=task_prompt, images=image, return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

image_size = image.size
parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=image_size)
print(parsed_answer)

Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the Quantization overview for more available quantization backends.

The example below uses bitsandbytes to quantize the model to 4-bit.

# pip install bitsandbytes
import torch
import requests
from PIL import Image
from transformers import AutoProcessor, Florence2ForConditionalGeneration, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)

model = Florence2ForConditionalGeneration.from_pretrained(
    "microsoft/Florence-2-large",
    dtype=torch.bfloat16,
    device_map="auto",
    quantization_config=quantization_config
)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large")

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

task_prompt = "<OD>"
inputs = processor(text=task_prompt, images=image, return_tensors="pt").to(model.device, torch.bfloat16)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

image_size = image.size
parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=image_size)

print(parsed_answer)

Notes

• Florence-2 is a prompt-based model. You need to provide a task prompt to tell the model what to do. Supported tasks are:
  • <OCR>
  • <OCR_WITH_REGION>
  • <CAPTION>
  • <DETAILED_CAPTION>
  • <MORE_DETAILED_CAPTION>
  • <OD>
  • <DENSE_REGION_CAPTION>
  • <CAPTION_TO_PHRASE_GROUNDING>
  • <REFERRING_EXPRESSION_SEGMENTATION>
  • <REGION_TO_SEGMENTATION>
  • <OPEN_VOCABULARY_DETECTION>
  • <REGION_TO_CATEGORY>
  • <REGION_TO_DESCRIPTION>
  • <REGION_TO_OCR>
  • <REGION_PROPOSAL>
• The raw output of the model is a string that needs to be parsed. The [Florence2Processor] has a [~Florence2Processor.post_process_generation] method that can parse the string into a more usable format, like bounding boxes and labels for object detection.

Resources

• Florence-2 technical report
• Jupyter Notebook for inference and visualization of Florence-2-large model

Florence2VisionConfig

autodoc Florence2VisionConfig

Florence2Config

autodoc Florence2Config

Florence2Processor

autodoc Florence2Processor

Florence2Model

autodoc Florence2Model - forward

Florence2ForConditionalGeneration

autodoc Florence2ForConditionalGeneration - forward

Florence2VisionBackbone

autodoc Florence2VisionBackbone - forward