GPT-4 vision for process documentation

Documentation lies.

Not intentionally. But the moment someone writes “click the blue button in the top right corner,” that button moves, changes color, or disappears in the next release. I’ve watched teams spend weeks documenting processes that were obsolete before the document got approved.

Research from Stanford shows GPT-4 Vision can extract text and understand UI layouts with over 65% accuracy, which beats most manual documentation attempts. But that number misses the point. The real breakthrough is not accuracy - it is that vision AI understands context.

The documentation problem

Writing down what you do is expensive. Really expensive.

You take screenshots, crop them, annotate them, write explanations, format everything, get it reviewed, publish it, and then watch it become wrong. Process documentation tools like Scribe tried to solve this by auto-capturing screenshots, but they still rely on you to provide the narrative. You’re still translating visual information into words.

The assumption has always been: humans see the screen, understand what’s happening, then explain it to other humans through text. Each step introduces error. What you see is not quite what you describe. What you describe is not quite what readers understand.

GPT-4 Vision eliminates the middle translation. You give it the screenshot. It tells you what’s happening.

What GPT-4V actually sees

Traditional OCR reads text. That’s it. It sees “Submit” and “Cancel” as words, not as buttons with spatial relationships and visual hierarchy.

GPT-4 Vision sees UI elements in context. According to OpenAI’s documentation, the model can interpret images alongside text in a single API call, understanding both what elements are present and how they relate to each other.

What that means practically: you screenshot your CRM deal creation flow. GPT-4V does not just read the field labels - it understands that “Company Name” comes before “Contact Person” because that is the logical workflow. It sees that the red asterisk means required field. It notices the grayed-out “Save” button is disabled until you fill in certain fields.

This is how humans actually use software. We do not read every label. We see patterns, relationships, states.

The detail parameter in the API controls how thoroughly GPT-4V analyzes images. Low detail mode uses 85 tokens and processes a 512px version - fine for simple screenshots. High detail mode can use up to 1,100 tokens but sees every pixel of a 2400px image. For gpt4 vision documentation work, high detail is worth it.

The screenshot workflow

I stopped writing process docs at Tallyfy. Started taking screenshots instead.

The workflow is simple. Painfully simple compared to traditional documentation:

1. Do the process while taking screenshots (Command+Shift+4 on Mac, Win+Shift+S on Windows)
2. Drop screenshots into a folder with sequential naming
3. Send each image to GPT-4 Vision with a prompt: “Explain what this screen does and what action the user should take”
4. Review and compile the AI’s explanations
5. Done

What used to take hours now takes minutes. And the output is better.

Better because GPT-4V describes what it sees, not what I think I see. It catches details I’d skip. It notices UI patterns I’ve become blind to through familiarity. When Microsoft’s research team tested vision models on UI understanding, they found that visual-language models could achieve state-of-the-art results by treating screenshots as structured data rather than unstructured images.

The key is in the prompt. “Explain this screen” is too vague. “Describe the purpose of this dialog box, list the required fields, and explain what happens when you click Save” gives you actionable documentation.

For batch processing, I wrote a simple script that walks through screenshot folders and generates a markdown file for each workflow. Processing is near-instant per image. The API costs are minimal compared to paying someone to write docs manually.

What actually changes

Accuracy improves because you are not fighting the telephone game of visual-to-verbal-to-visual translation.

I tested this with our customer onboarding process. Created documentation the old way (manual writing) and the new way (screenshot + GPT-4V). Had new hires follow both. The vision-based docs had zero ambiguity issues. The manual docs? Three places where people got confused because my written description did not match what they actually saw on screen.

Time savings compound. Initial documentation is faster, but maintenance is where you really win. When we update our UI, I retake screenshots and regenerate docs in minutes. The old way would be hunting through a lengthy document, updating text, replacing images, reformatting everything.

A more subtle benefit: documentation becomes queryable. Instead of “here is how to do X,” you have visual records of every state in your system. Someone asks “what does this error look like?” You have the screenshot. GPT-4V can even compare screenshots to spot differences between versions.

Recent benchmarks show GPT-4 Vision performs particularly well on infographics and charts - better than traditional OCR. For business software with dashboards and complex UI, this matters. Your sales dashboard is not plain text. It’s visual information architecture.

Making it work

Start with high-value, high-change processes. Don’t document your entire system at once.

Pick one workflow that changes often or confuses new users. Take screenshots of every step. Run them through GPT-4V. Compare the output to your existing docs. You’ll see immediately whether this approach works for your use case.

Image quality matters. Blurry screenshots get blurry documentation. Take screenshots at actual resolution, not scaled down. For web applications, I use full-page screenshots rather than just viewport captures - tools like GoFullPage work well.

Prompt engineering is simpler than you think. I use three basic templates:

• For explanatory docs: “Describe what this screen does, what information it displays, and what actions are available.”
• For step-by-step guides: “Explain what the user should do on this screen to [specific goal]. List required fields and note any validation rules visible.”
• For troubleshooting: “Identify any error messages, warnings, or unusual states visible in this screenshot and explain what they mean.”

The gpt4 vision documentation approach has limits. It cannot see dynamic behavior - hover states, animations, conditional logic that depends on data. For those, you need video or separate annotation. And it is not great at precise object localization - if you need exact pixel coordinates, stick with traditional computer vision tools.

Cost control: use the detail parameter wisely. Not every screenshot needs high-detail analysis. Login screens and simple forms work fine in low-detail mode. Complex dashboards and data-heavy views benefit from high detail. I batch process screenshots overnight during low-usage hours.

Version control your screenshots. I keep them in the same repo as code, organized by feature and release version. When someone reports a bug, I can see exactly what the UI looked like in that release.

The biggest mindset shift: stop trying to make documentation comprehensive. Make it visual and queryable instead. You do not need to document every possible path through your software. Document the main flows visually, then let GPT-4V answer specific questions from screenshots as they come up.

One last thing that surprised me - this works for documenting other companies’ software too. When we integrate with third-party tools, I screenshot their UI and use GPT-4 Vision to generate integration guides. Faster than reading their docs, and often more accurate because I am documenting what I actually see in their current version.

Documentation still lies sometimes. But now it lies less, updates faster, and costs a fraction of what it used to.