<!--Copyright 2023 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->
*This model was released on 2023-12-01 and added to Hugging Face Transformers on 2023-12-13 and contributed by [ybelkada](https://huggingface.co/ybelkada).*

<div style="float: right;">
    <div class="flex flex-wrap space-x-1">
        <img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
        <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
    </div>
</div>

# VipLlava

[VipLlava](https://huggingface.co/papers/2312.00784) introduces a novel multimodal model that enhances region-specific comprehension in vision-language tasks. It achieves this by allowing users to mark images with natural cues such as "red bounding boxes" or "pointed arrows" during training. This approach eliminates the need for complex region encodings and achieves state-of-the-art performance on benchmarks like Visual7W, PointQA, and Visual Commonsense Reasoning. Additionally, ViP-Bench is presented as a comprehensive benchmark to evaluate models' capabilities in understanding visual prompts across various dimensions.

<hfoptions id="usage">
<hfoption id="VipLlavaForConditionalGeneration">

```py
import torch
import requests
from PIL import Image
from transformers import AutoProcessor, VipLlavaForConditionalGeneration

model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", dtype="auto")
processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf")

prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\n{}###Assistant:"
question = "Can you please describe this image?"
prompt = prompt.format(question)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16)
generate_ids = model.generate(**inputs, max_new_tokens=20)
print(processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True))
```

</hfoption>
</hfoptions>

## VipLlavaConfig

[[autodoc]] VipLlavaConfig

## VipLlavaForConditionalGeneration

[[autodoc]] VipLlavaForConditionalGeneration
    - forward

## VipLlavaModel

[[autodoc]] VipLlavaModel
    - forward

