A common pitfall when handling scanned PDFs is to run a full pass with traditional OCR tools first, then spend a lot of time cleaning up the erroneous results. This article draws on a large scanned-PDF extraction task executed by Hermes Agent and abstracts a more general processing logic: how to complete the recognition and structured organization of scanned PDFs efficiently and with high quality at minimal sunk cost.

1. First, Decide: Is It an “Image Book” or a “Document”?

The first step after getting a PDF is not to extract directly, but to detect its type. Many scanned PDFs look like a .pdf on the surface yet contain no extractable text layer inside.

This step usually only takes a few seconds, but it can prevent hours of wasted subsequent effort.

2. The Role of Local OCR Should Be Minimized

Once you confirm it is an image-based PDF, many people’s second reaction is to fire up a traditional OCR tool such as Tesseract and run full recognition and cleanup. In real tasks, however, the marginal benefit of this approach is often low:

• Local OCR handles printed English reasonably well, but it is error-prone on Chinese definitions in mixed Chinese-English layouts;
• Cleanup scripts can only deal with layout noise (headers, footers, table lines), not semantic-level recognition errors;
• When the end goal is structured, directly usable, high-quality text, investment in cleanup code can easily become a sunk cost.

3. The Decisive Step: Bringing in a Vision Language Model for Direct Recognition

In this task, once Hermes Agent confirmed that the local OCR output for Chinese was unacceptable, it wrote its own Python script and directly called an external Vision-Language model (Qwen/Qwen2-VL-72B-Instruct) to re-recognize the content. This was the turning point that ultimately determined output quality.

3.1 Lessons from Model Selection

The initial test used a dedicated model named OCR, but that model had weak instruction-following capabilities—it repeatedly output the review tables and marking symbols from the page and could not extract only the target content as required. This shows that:

3.2 Secure Practices for API Calls

The external API key was not hardcoded into the script; it was injected via environment variables configured in ~/.hermes/.env (such as SILICONFLOW_API_KEY), read by the script through os.environ. This practice meets security requirements:

• The key is not exposed in any code file or log;
• Credentials are stored in a controlled config location in the user’s home directory;
• It is only valid for the lifespan of the current session.

For an AI agent, retrieving environment variables through ~/.hermes/.env is the standard and secure way to call external services.

4. Page Stitching Strategy: Merge Consecutive Pages and Recognize Them Together

Scanned books, textbooks, and exam papers often contain content that flows continuously across pages. If each page is uploaded separately to the model, it can lead to:

• Fragmented context, causing the model to misjudge the relationship between content on two pages;
• Cross-page items being truncated or missed.

The strategy used in this task was:

Specific steps:

• Use pdf2image to convert consecutive pages into images;
• Use PIL to stitch multiple images vertically into one long image;
• Compress as JPEG, encode as base64, and upload via the API.

The advantage of this approach is that it preserves the continuity and spatial relationships of reading, reduces context switching during API calls, and also saves the overhead of multiple requests.

5. Prompt Design: Hard Constraints for Structured Output

To get the Vision model to output clean text that can be written directly to a file, the prompt needs to include the following hard constraints:

1. Explicit exclusions: List all non-target content on the page (tables, circle markings, page numbers, usage instructions, etc.), and require the model to actively ignore them;
2. Format template: Specify the output format for each entry, for example number. English word phonetic part-of-speech Chinese definition;
3. Fixed header: Force the output to begin with a unified heading structure to make downstream automation easier;
4. Reference text aid: Append the initial local OCR results to the end of the prompt, letting the model do “comparative correction” based on the image rather than guessing blind from scratch. This significantly improves output accuracy and completeness.

6. Core Methodology: Verify First, Then Run in Full

Whatever toolchain you use, the most important step when handling scanned PDFs is:

6.1 What to Verify with Minimal Effort

Before launching a full batch process, the agent should first verify the following elements at minimal cost:

• Page-number mapping pattern: How many days/chapters? How many pages per chapter? Where are the boundaries?
• Model recognition ability: At the current DPI, can the model accurately recognize fonts, phonetic symbols, and mixed Chinese-English content?
• Prompt stability: Is the output format consistent across 1–3 consecutive samples? Is any content missing?
• Cost estimate: Is the token consumption per page/chapter within budget? Can the DPI be downgraded?

6.2 Dynamic Resolution Scaling

Image token cost is roughly proportional to resolution. You can first test the Vision model at a low DPI (such as 100 DPI); if the recognition rate is already sufficient, there is no need to use 150 DPI or 250 DPI. For printed materials with large fonts, 100 DPI is often enough, reducing image encoding cost by about 50% or more.

6.3 Possibility of Structured Output

If the API supports JSON mode, you can ask the model directly to return a structured array (such as {"word", "phonetic", "pos", "definition"}), entirely eliminating post-processing cleanup code and improving automation and maintainability.

7. Delivery Structure: Keep Raw Data and Refined Results Separate

For tasks that may produce multiple versions of results, a dual-directory isolation design is recommended:

The benefits of this design:

• Raw data is not lost;
• The refined result of any unit can be diffed against the original output;
• Rollback is always possible because nothing is overwritten.

8. Pre-Flight Checklist for AI Agents

If you are an AI agent and receive a task to “extract structured content from a scanned PDF”, follow this order:

1. PDF type detection: sample with pdfinfo + pdftotext to confirm whether it is an image-based PDF;
2. Structure-pattern detection: use the lowest-cost method (low-resolution OCR or a small number of Vision API calls) to confirm the “page → chapter/unit” mapping pattern;
3. Solution validation: process the first 1–3 units with the Vision model to verify output format, completeness, and accuracy;
4. API key security: read it from ~/.hermes/.env environment variables, never hardcode it;
5. Full concurrent execution: run the batch only after confirming everything works, saving results to a separate directory without overwriting the raw data.

9. Conclusion

For content extraction from scanned PDFs, the order in which you choose your toolchain matters more than the capability of any single tool. Traditional OCR is well-suited as a low-cost structural-probe sentinel, but if the goal is directly usable, high-quality structured text, a Vision-Language model is the ultimate guarantee of quality.

The most expensive mistake is usually not picking the wrong model, but launching a full batch process without minimal verification first. Agents should establish a rhythm of “validate a sample → adjust parameters → run the full batch” early on, to control sunk costs and ensure the reliability of the final deliverable.