Enterprise AI Has to Read the Mess

Enterprise data rarely arrives as a clean paragraph.

A merchandiser deciding whether a product listing is complete may need a supplier spreadsheet, a catalog PDF, two product photographs, packaging text, and the current record in the commerce system. The size is visible only in the image. The material appears only in the PDF. The spreadsheet uses an old category name. The system record is missing a unit.

The useful AI product is not the one that describes each file. It is the one that helps a person produce a defensible record from all of them.

Begin with the decision

Do not start by indexing every file type the company owns.

Choose a decision or workflow:

Is this product record ready to publish?
Does this brokerage document contain a reportable transaction?
Is this damaged-goods claim supported by the evidence?
Does the invoice match the contract and delivery record?

Then map the evidence a skilled person uses, where it lives, how fresh it must be, who may access it, and what contradictions matter.

The unit of value is an evidence-backed decision packet.

Separate extraction from decision

A durable architecture keeps distinct stages:

source capture
  → modality-specific extraction
  → normalized claims and entities
  → cross-source reconciliation
  → deterministic validation
  → confidence and exceptions
  → human decision and recorded outcome

This separation matters because each stage fails differently.

Optical extraction can misread a decimal. Vision can confuse the product with background packaging. A table parser can lose merged-cell meaning. Retrieval can select an outdated contract. The final model can reconcile the wrong evidence perfectly.

When the stages are explicit, the reviewer can see where uncertainty entered.

A product-attribute workbench

Consider a workflow that proposes structured product attributes from images, catalogs, and supplier data.

The system might produce:

Attribute	Proposed value	Evidence	Status
Material	Stainless steel	Catalog page 4	High confidence
Width	18 in	Supplier sheet B27	Needs unit check
Color	Matte black	Front image	Visual estimate
Indoor/outdoor	Outdoor	Catalog copy	Conflicts with current record

The reviewer is not presented with one “92% confidence” badge. They can inspect each field, open the cited source, compare the current value, accept or correct the proposal, and record why.

That interaction turns multimodal extraction into operational data quality.

The same pattern applies to brokerage documents. Extracted transactions or account fields become useful when they retain page-level source context, confidence, validation state, and a path for a reviewer to resolve ambiguity.

Preserve provenance as data moves

Every normalized claim should retain a pointer to its source:

document and page
spreadsheet and cell range
image and region
audio and time range
source-system record and retrieval time

Provenance supports review, correction, audit, and deletion. It also prevents a generated summary from becoming a mysterious new source of truth.

When sources conflict, do not collapse them silently. Represent the conflict:

supplier sheet: 18 inches
catalog PDF: 17.5 inches
current system: 18 (unit missing)
resolution: human review required

The model may recommend a resolution. The product should preserve the disagreement.

Evaluate combinations, not only modalities

A system can score well on document extraction and image understanding separately while failing whenever the decisive fact spans both.

Build test bundles with:

clean and degraded scans
cropped or misleading images
merged spreadsheet cells
stale system records
duplicate sources
conflicting values
missing evidence
correct “insufficient information” outcomes

Score field accuracy, provenance accuracy, conflict detection, exception routing, and final decision quality. Break results down by modality combination. An aggregate can hide that every image-plus-table case fails.

Permissions must survive normalization

Combining data can create a more sensitive artifact than any individual source.

A case packet may join customer identity, financial records, images, and internal policy. The normalized record, retrieval index, trace, and review screen need permissions derived from the sources. Extraction should never become a permission bypass.

Carry source identity, allowed use, retention, and deletion obligations forward. Avoid copying raw data into logs and eval sets merely because the new format is convenient.

The review surface is part of the model

The reviewer should be able to:

inspect cited evidence in context
compare conflicting sources
correct individual fields
distinguish extraction uncertainty from policy uncertainty
change the recommendation
record the reason

Those corrections should improve the system, but not all corrections belong in the same place. A misread unit may improve an extractor. An outdated source may change retrieval. A policy exception may become a business rule. A novel case may become an eval.

If every correction is treated as “train the model more,” the team loses the architecture.

Enterprise multimodality is not about accepting more file types. It is about preserving evidence while messy inputs become accountable decisions.