使用本地vlm注释图片
在 [ ]:
Copied!
%pip install -q docling[vlm] ipython
%pip install -q docling[vlm] ipython
Note: you may need to restart the kernel to use updated packages.
在 [1]:
Copied!
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
在 [2]:
Copied!
# The source document
DOC_SOURCE = "https://arxiv.org/pdf/2501.17887"
# 源文档
DOC_SOURCE = "https://arxiv.org/pdf/2501.17887"
使用Granite Vision描述图片¶
本节将在本地运行 ibm-granite/granite-vision-3.1-2b-preview 模型以描述文档的图片。
在 [3]：
Copied!
from docling.datamodel.pipeline_options import granite_picture_description
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = (
granite_picture_description # <-- the model choice
)
pipeline_options.picture_description_options.prompt = (
"Describe the image in three sentences. Be consise and accurate."
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
doc = converter.convert(DOC_SOURCE).document
from docling.datamodel.pipeline_options import granite_picture_description
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = (
granite_picture_description # <-- 模型选择
)
pipeline_options.picture_description_options.prompt = (
"用三句话描述图像。要简洁准确。"
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
doc = converter.convert(DOC_SOURCE).document
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
在 [4]：
Copied!
from docling_core.types.doc.document import PictureDescriptionData
from IPython import display
html_buffer = []
# display the first 5 pictures and their captions and annotations:
for pic in doc.pictures[:5]:
html_item = (
f"<h3>Picture <code>{pic.self_ref}</code></h3>"
f'<img src="{str(pic.image.uri)}" /><br />'
f"<h4>Caption</h4>{pic.caption_text(doc=doc)}<br />"
)
for annotation in pic.annotations:
if not isinstance(annotation, PictureDescriptionData):
continue
html_item += (
f"<h4>Annotations ({annotation.provenance})</h4>{annotation.text}<br />\n"
)
html_buffer.append(html_item)
display.HTML("<hr />".join(html_buffer))
from docling_core.types.doc.document import PictureDescriptionData
from IPython import display
html_buffer = []
# display the first 5 pictures and their captions and annotations:
for pic in doc.pictures[:5]:
html_item = (
f"图片
"
f'%7D)
' f"
" ) for annotation in pic.annotations: if not isinstance(annotation, PictureDescriptionData): continue html_item += ( f"
\n" ) html_buffer.append(html_item) display.HTML("
".join(html_buffer))
图片 {pic.self_ref}
"
f'' f"
标题
{pic.caption_text(doc=doc)}" ) for annotation in pic.annotations: if not isinstance(annotation, PictureDescriptionData): continue html_item += ( f"
注释 ({annotation.provenance})
{annotation.text}\n" ) html_buffer.append(html_item) display.HTML("
".join(html_buffer))
输出[4]：
图片 #/pictures/0
标题
Figure 1: Sketch of Docling's pipelines and usage model. Both PDF pipeline and simple pipeline build up a DoclingDocument representation, which can be further enriched. Downstream applications can utilize Docling's API to inspect, export, or chunk the document for various purposes.注释 (ibm-granite/granite-vision-3.1-2b-preview)
In this image we can see a poster with some text and images.图片 #/pictures/1
标题
Figure 2: Dataset categories and sample counts for documents and pages.注释 (ibm-granite/granite-vision-3.1-2b-preview)
In this image we can see a pie chart. In the pie chart we can see the categories and the number of documents in each category.图片 #/pictures/2
标题
Figure 3: Distribution of conversion times for all documents, ordered by number of pages in a document, on all system configurations. Every dot represents one document. Log/log scale is used to even the spacing, since both number of pages and conversion times have long-tail distributions.注释 (ibm-granite/granite-vision-3.1-2b-preview)
In this image we can see a graph. On the x-axis we can see the number of pages. On the y-axis we can see the seconds.图片 #/pictures/3
标题
Figure 4: Contributions of PDF backend and AI models to the conversion time of a page (in seconds per page). Lower is better. Left: Ranges of time contributions for each model to pages it was applied on (i.e., OCR was applied only on pages with bitmaps, table structure was applied only on pages with tables). Right: Average time contribution to a page in the benchmark dataset (factoring in zero-time contribution for OCR and table structure models on pages without bitmaps or tables) .注释 (ibm-granite/granite-vision-3.1-2b-preview)
In this image we can see a bar chart and a line chart. In the bar chart we can see the values of Pdf Parse, OCR, Layout, Table Structure, Page Total and Page. In the line chart we can see the values of Pdf Parse, OCR, Layout, Table Structure, Page Total and Page.图片 #/pictures/4
标题
Figure 5: Conversion time in seconds per page on our dataset in three scenarios, across all assets and system configurations. Lower bars are better. The configuration includes OCR and table structure recognition ( fast table option on Docling and MinerU, hi res in unstructured, as shown in table 1).注释 (ibm-granite/granite-vision-3.1-2b-preview)
In this image we can see a bar chart. In the chart we can see the CPU, Max, GPU, and sec/page.使用SmolVLM描述图片¶
本节将本地运行HuggingFaceTB/SmolVLM-256M-Instruct模型，以描述文档中的图片。
在 [7]：
Copied!
from docling.datamodel.pipeline_options import smolvlm_picture_description
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = (
smolvlm_picture_description # <-- the model choice
)
pipeline_options.picture_description_options.prompt = (
"Describe the image in three sentences. Be consise and accurate."
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
doc = converter.convert(DOC_SOURCE).document
from docling.datamodel.pipeline_options import smolvlm_picture_description
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = (
smolvlm_picture_description # <-- 模型选择
)
pipeline_options.picture_description_options.prompt = (
"用三句话描述图像。要简洁准确。"
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
doc = converter.convert(DOC_SOURCE).document
在 [6]:
Copied!
from docling_core.types.doc.document import PictureDescriptionData
from IPython import display
html_buffer = []
# display the first 5 pictures and their captions and annotations:
for pic in doc.pictures[:5]:
html_item = (
f"<h3>Picture <code>{pic.self_ref}</code></h3>"
f'<img src="{str(pic.image.uri)}" /><br />'
f"<h4>Caption</h4>{pic.caption_text(doc=doc)}<br />"
)
for annotation in pic.annotations:
if not isinstance(annotation, PictureDescriptionData):
continue
html_item += (
f"<h4>Annotations ({annotation.provenance})</h4>{annotation.text}<br />\n"
)
html_buffer.append(html_item)
display.HTML("<hr />".join(html_buffer))
from docling_core.types.doc.document import PictureDescriptionData
from IPython import display
html_buffer = []
# display the first 5 pictures and their captions and annotations:
for pic in doc.pictures[:5]:
html_item = (
f"图片
"
f'
' f"
" ) for annotation in pic.annotations: if not isinstance(annotation, PictureDescriptionData): continue html_item += ( f"
\n" ) html_buffer.append(html_item) display.HTML("
".join(html_buffer))
图片 {pic.self_ref}
"
f'' f"
标题
{pic.caption_text(doc=doc)}" ) for annotation in pic.annotations: if not isinstance(annotation, PictureDescriptionData): continue html_item += ( f"
注释 ({annotation.provenance})
{annotation.text}\n" ) html_buffer.append(html_item) display.HTML("
".join(html_buffer))
输出[6]：
图片 #/pictures/0
标题
Figure 1: Sketch of Docling's pipelines and usage model. Both PDF pipeline and simple pipeline build up a DoclingDocument representation, which can be further enriched. Downstream applications can utilize Docling's API to inspect, export, or chunk the document for various purposes.注释 (HuggingFaceTB/SmolVLM-256M-Instruct)
This is a page that has different types of documents on it.图片 #/pictures/1
标题
Figure 2: Dataset categories and sample counts for documents and pages.注释 (HuggingFaceTB/SmolVLM-256M-Instruct)
Here is a page-by-page list of documents per category: - Science - Articles - Law and Regulations - Articles - Misc.图片 #/pictures/2
标题
Figure 3: Distribution of conversion times for all documents, ordered by number of pages in a document, on all system configurations. Every dot represents one document. Log/log scale is used to even the spacing, since both number of pages and conversion times have long-tail distributions.注释（HuggingFaceTB/SmolVLM-256M-Instruct）
The image is a bar chart that shows the number of pages of a website as a function of the number of pages of the website. The x-axis represents the number of pages, ranging from 100 to 10,000. The y-axis represents the number of pages, ranging from 100 to 10,000. The chart is labeled "Number of pages" and has a legend at the top of the chart that indicates the number of pages. The chart shows a clear trend: as the number of pages increases, the number of pages decreases. This is evident from the following points: - The number of pages increases from 100 to 1000. - The number of pages decreases from 1000 to 10,000. - The number of pages increases from 10,000 to 10,000.图片 #/pictures/3
标题
Figure 4: Contributions of PDF backend and AI models to the conversion time of a page (in seconds per page). Lower is better. Left: Ranges of time contributions for each model to pages it was applied on (i.e., OCR was applied only on pages with bitmaps, table structure was applied only on pages with tables). Right: Average time contribution to a page in the benchmark dataset (factoring in zero-time contribution for OCR and table structure models on pages without bitmaps or tables) .注释 (HuggingFaceTB/SmolVLM-256M-Instruct)
bar chart with different colored bars representing different data points.图片 #/pictures/4
标题
Figure 5: Conversion time in seconds per page on our dataset in three scenarios, across all assets and system configurations. Lower bars are better. The configuration includes OCR and table structure recognition ( fast table option on Docling and MinerU, hi res in unstructured, as shown in table 1).注释 (HuggingFaceTB/SmolVLM-256M-Instruct)
A bar chart with the following information: - The x-axis represents the number of pages, ranging from 0 to 14. - The y-axis represents the page count, ranging from 0 to 14. - The chart has three categories: Marker, Unstructured, and Detailed. - The x-axis is labeled "see/page." - The y-axis is labeled "Page Count." - The chart shows that the Marker category has the highest number of pages, followed by the Unstructured category, and then the Detailed category.使用其他视觉模型¶
上述示例也可以使用其他视觉模型重现。
Docling 选项 PictureDescriptionVlmOptions 允许您从 Hugging Face Hub 指定您喜欢的视觉模型。
在 [8]:
Copied!
from docling.datamodel.pipeline_options import PictureDescriptionVlmOptions
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = PictureDescriptionVlmOptions(
repo_id="", # <-- add here the Hugging Face repo_id of your favorite VLM
prompt="Describe the image in three sentences. Be consise and accurate.",
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
# Uncomment to run:
# doc = converter.convert(DOC_SOURCE).document
from docling.datamodel.pipeline_options import PictureDescriptionVlmOptions
pipeline_options = PdfPipelineOptions()
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = PictureDescriptionVlmOptions(
repo_id="", # <-- 在这里添加您最喜欢的VLM的Hugging Face repo_id
prompt="用三句话描述图像。简洁而准确。",
)
pipeline_options.images_scale = 2.0
pipeline_options.generate_picture_images = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=pipeline_options,
)
}
)
# 取消注释以运行：
# doc = converter.convert(DOC_SOURCE).document
在 [ ]:
Copied!