The funniest sentence in enterprise AI is still some version of "the model passed our evals." Passed which evals? Run where? Against what test set? With which model build, prompt, hardware, adapter, collection version, and tiny environment change that nobody wrote down because the dashboard looked green and everyone wanted lunch?

Red Hat's latest EvalHub article lands directly on that sore spot. On June 16, Red Hat Developer published a walkthrough for storing AI evaluation records as immutable OCI artifacts. The practical idea is simple and wonderfully annoying to anyone selling benchmark perfume: if a model is good enough to ship, the evidence should survive outside the experiment tracker.

A benchmark score without provenance is not evidence. It is a screenshot with confidence.

The Problem Is Not Running Evals

Teams are getting better at running evaluations. They compare model versions, test retrieval changes, gate releases in CI, and dump metrics into tools like MLflow. That is progress. But Red Hat is pointing at the next problem: queryable is not the same thing as tamper-evident.

MLflow gives teams a useful place to search experiment runs. EvalHub already records each evaluation with configuration, model version, collection version, and hardware tags. That helps developers answer normal questions like "what changed between yesterday's run and today's suspiciously confident disaster?"

The catch is that an experiment database is still a live system. Rows can be deleted, tracking servers can be rebuilt, metadata can drift, and six months later the team may have a compliance review where everyone is reenacting a detective scene over a dashboard nobody fully trusts.

OCI Turns the Result Into an Artifact

EvalHub's new OCI persistence layer takes the output directory from an evaluation run and pushes it to an OCI-compliant registry. That can be a registry such as Quay or another OCI distribution-compatible backend. Instead of treating the result as a dashboard entry alone, EvalHub treats it like a build artifact.

evaluation run
  -> MLflow experiment record for search
  -> OCI artifact for preservation
  -> sha256 digest for verification
  -> deployment gate can check the exact evidence

The important bit is the digest. EvalHub retrieves a content digest in the familiar sha256:... form and stores the artifact reference in JobResults.oci_artifact, then writes it into MLflow metadata alongside the experiment record. If the bytes change, the digest changes. If the registry cannot return the artifact by digest later, that is also a useful answer.

This is not glamorous AI infrastructure. It is better than glamorous: it is boring in the way receipts, locks, checksums, and audit logs are boring. The system has a memory.

How the Plumbing Works

In Red Hat's walkthrough, OCI export is opt-in per evaluation job through an exports block. When the run completes, EvalHub calls callbacks.create_oci_artifact(), which hands the work to an OCIArtifactPersister. That persister packages the result directory, authenticates to the registry, pushes with oras, captures the registry's content digest, and returns the full artifact reference.

The configuration includes registry coordinates such as oci_host and oci_repository. A custom oci_tag is optional. If it is omitted, EvalHub can generate a deterministic tag from the evaluation context, based on inputs like job ID, provider ID, benchmark ID, and benchmark index. The digest remains the authority, but the tag makes the artifact easier to find without keeping a sticky note under somebody's keyboard.

Metadata Is Where This Gets Useful

The article also calls out annotations. That sounds dull until you remember that half of AI governance is really a search problem wearing a suit.

  • Model family: Which model line did this run evaluate?
  • Collection version: Which benchmark or evaluation set was used?
  • Environment: Was this staging, production-like, or a local science project?
  • Compliance tags: Which policy bucket or release gate does the result support?
  • Hardware tags: Was the run on the same class of accelerator that production uses?

Put those labels on the artifact, store the digest, and suddenly the evaluation result can travel through the same operational machinery teams already use for container images and supply-chain evidence. The model team does not have to invent a private archive ceremony. It can use a registry, a pull command, and a hash check.

Kubernetes Gets a Sidecar, Because Of Course It Does

Red Hat also describes two authentication paths. In local mode, EvalHub can use the developer's Docker config. In Kubernetes mode, the EvalHub-managed pod uses an adapter container and a sidecar. The adapter asks the persister to push the artifact, the push routes through the sidecar proxy, the sidecar handles registry authentication using a Kubernetes dockerconfigjson secret, and the artifact reference still points at the original registry host.

That is a lot of plumbing for a benchmark file, but that is the point. Serious AI operations are becoming less like prompt tinkering and more like release engineering. If a model is going to approve a loan, summarize medical notes, triage security alerts, or write code inside a production repository, the eval record should be more durable than a vibes chart.

Why This Matters

The industry has spent two years arguing about benchmark scores as if they were sports stats. The more important question is not whether a model got a bigger number on Tuesday. It is whether a company can prove what it tested before it shipped, recover the exact evidence later, and make a deployment gate depend on something stronger than institutional memory.

EvalHub's OCI approach fits a larger shift: AI systems are being pulled into ordinary software supply-chain discipline. Results need provenance. Config needs versioning. Release gates need machine-checkable inputs. Auditors need something better than "the notebook used to be here."

There is also a cultural lesson hiding in the infrastructure. AI teams love demos because demos are clean. Production is not clean. Production asks rude questions: who changed the dataset, why did this model pass yesterday, which run approved the deployment, and can you prove the artifact is still the same one?

The Takeaway

Red Hat is not saying every toy chatbot needs a courtroom-grade evidence locker. It is saying that enterprise AI evaluation is leaving the screenshot era. If an evaluation matters, it should be preserved as an artifact, tied to a digest, tagged with enough metadata to find later, and linked back to the experiment record people actually use day to day.

That is a small systems move with a big implication: the benchmark is no longer just a number. It is a thing you can pull, verify, and put in front of a deployment gate. More of that, please.

Sources