DocuForge: Making the Pipeline Reliable

In the previous DocuForge post, I wrote about moving from a 4090-powered orchestrator, complete with a robot arm pressing the PC’s power button, to a smaller pipeline for paperless-ngx.

The idea was to separate the work: a lightweight model for OCR, a heavier model for classification, and a configurable time window so the second one would leave my MacBook alone during the day.

Since then, most of the work has been less photogenic. Keeping classification instructions inside the model’s context. Preventing tags from multiplying. Making sure a failed API call does not quietly remove a document from the queue.

The robot arm was easier to explain at dinner.

The architecture has stayed small

DocuForge still receives a webhook when paperless-ngx adds a document. It runs OCR, writes the text back to paperless, then queues the document for metadata processing:

paperless-ngx
    → webhook
    → OCR
    → text saved in paperless
    → SQLite tagging queue
    → scheduled classification
    → metadata updated in paperless

The two stages still have separate model settings. The previous post covered GLM-OCR and Qwen 3.5 9B, but neither is hardwired into the pipeline. It can also use paperless-ngx’s existing OCR text and skip the vision model entirely.

I have kept that flexibility because the useful part is the workflow. There is no reason to rerun OCR on every document if the text already extracted by paperless is good enough.

The more interesting changes are around what happens when that workflow gets an awkward input.

Valid JSON, containing absolutely nothing

One particularly annoying problem showed up with long documents. The model returned valid JSON, but all the useful fields were empty.

The request completed. The response parsed. There was just nothing worth applying to the document.

The cause was the input budget. DocuForge assembled the classification instructions first, then appended up to 64,000 characters of document text. With an 8,192-token context configured, that was far too much for the request in question.

The recorded Ollama log showed the input being reduced from roughly 18,000 tokens to about 4,100. In that case, it kept the tail of the prompt. The document text survived; the instructions at the beginning did not.

I had carefully explained the required JSON schema, the naming rules, and how to classify the document, then effectively removed that explanation before the model could use it.

Not my best prompt engineering.

The fix was straightforward: keep the instructions whole, and give the document text its own configurable budget. The default is now 8,000 characters. Whenever text is shortened, DocuForge logs it, and each tagging response logs Ollama’s evaluated prompt token count.

That character limit is a practical default, not an exact token calculation. The instructions and the injected metadata names also take space. A larger vocabulary or a different model can change what fits.

There is a trade-off, too: classification only sees the beginning of a document that exceeds the limit. If the important information is on the last page, this approach can miss it. DocuForge does not currently split a long document into chunks and combine the results.

For now, I prefer a visible, understandable limit. It gives me something concrete to adjust instead of an empty response and a vague suspicion that the model is having a bad day.

A payslip does not need twelve tags

Getting a response is only half the problem. The response also needs to fit the filing system.

The bundled prompt has become fairly specific about that. A payslip should be a Bulletin de paie, with Salaire as its tag. It does not need a small cloud of related concepts such as employment, finance, remuneration, and labour law.

Those words may all be relevant. They are not all useful labels for finding a payslip later.

DocuForge can inject the existing paperless tags, document types, and correspondents into the prompt. An important detail was preserving their original spelling. The internal lookup maps normalize names for comparison, but those normalized keys are poor examples to give the model: lowercased names with missing accents tend to come back as lowercased names with missing accents.

The prompt now gets the original names. Normalization stays in the lookup code, where it belongs.

There are also a few controls around new tags:

• explicit synonym mappings, such as Paie:Salaire;
• an option to reject unknown tags;
• optional fuzzy matching against existing tags;
• an optional review tag when a new classification tag is created.

Fuzzy matching is off by default. Two labels looking similar is useful evidence, but it is not always enough to decide that they mean the same thing.

The review marker is excluded from the tags shown to the model. Otherwise, a tag meaning “check this later” could become just another category the model helpfully applies to everything.

Automation should reduce the amount of filing I do. It should not give me a second filing system to maintain.

Which service returned the 404?

The tagging queue is backed by SQLite. Pending document IDs survive a container restart, and the document ID is the primary key, so enqueuing the same ID again does not create a second row.

But persistence only helps if the application makes the right decision about when to remove a row.

A successful metadata update removes the document from the queue. A document that has been deleted from paperless should also be removed: there is nothing left to process.

The subtle case is a 404.

If paperless returns 404 while looking up that document, it is gone. If Ollama returns 404 because the requested model is missing, the document is still there. Treating both errors as “document no longer exists” would quietly discard work that should be retried.

The code now distinguishes those cases explicitly. Only a missing-document response from the paperless lookup becomes the exception that permits removal. Other failures keep the item queued and increase its failure count. A rejected metadata update is a failure too, even if inference succeeded.

The configured failure threshold produces a warning for manual intervention. It does not delete the item or stop future attempts.

This is not a complicated retry system. It is enough to let a temporary dependency failure remain a temporary dependency failure.

Updating one field without losing the others

DocuForge uses a custom boolean field called AI Processed to mark completed documents. The webhook path checks it before starting OCR, so a completed document can be skipped.

Setting that flag exposed another ordinary API detail: the custom_fields list in an update replaces the existing list.

Sending just the AI flag would therefore remove unrelated custom fields already attached to the document. The update now carries those fields forward and adds or replaces only the DocuForge flag.

Existing tags have their own setting. DocuForge can merge its suggestions with them, but replacing tags remains the default. That choice matters if some tags come from manual filing or another paperless workflow.

The model can propose metadata. The application still has to understand what applying that metadata will do.

Imports and restarts are part of the workflow

Batch imports also needed some restraint. Webhook OCR now goes through a dedicated worker pool, with one worker by default. Several incoming documents can wait their turn rather than all starting vision inference at once.

OCR requests have configurable timeouts and bounded retries for connection errors and timeouts. During a normal shutdown, the service waits for submitted OCR and tagging tasks to finish before closing shared resources.

There is still a boundary here: the SQLite queue persists the tagging stage, while pending webhook OCR work lives in memory. Graceful shutdown helps with ordinary restarts; it does not make an abrupt process kill or a power cut safe for every accepted OCR task.

That is worth being clear about. A durable tagging queue is useful, but it is not a guarantee that the entire pipeline can recover every piece of work after any interruption.

Testing the uninteresting failures

The project now has regression tests around these behaviours: preserving custom fields, keeping failed tagging work queued, distinguishing the two kinds of 404, retaining prompt instructions, and draining workers during shutdown.

They use controlled substitutes for paperless and Ollama, with real queue logic where it matters. I can check whether a failed request leaves a document queued without starting a model or waiting until midnight.

Those tests do not tell me whether a model will classify every invoice correctly. That still needs real documents and inspection of the results. They do tell me whether the surrounding code behaves sensibly when classification or an API call goes wrong.

DocuForge is still my first proper Python project. A lot of this work feels familiar from backend development: know what an update replaces, keep track of unfinished work, and make failures visible.

There are still things to improve, particularly around long documents and recovery before the tagging queue. But the architecture has stayed recognizable, and the recent changes make its limits easier to see.

The robot arm remains the better anecdote. The queue keeping a document after a failed request is the part I would rather rely on.