Immersive Archive

The lab is now extending the archive from the devices to the ideas behind them. Mort is an agent that answers visitors' questions as Heilig, from his own papers, and points to the item in the archive each answer comes from. Heilig's family approved the project. The aim is to let people explore the history of VR by using the early machines and by asking the person who imagined them.

Mort is built around a knowledge base. An ingestion workflow I built in Dify fills it from the archive. When a visitor talks to Mort, the conversation runs on Estuary, the lab's open-source framework for real-time conversational agents, which calls the knowledge base whenever a question needs the archive. Connecting the knowledge base to Estuary was joint work across the lab.

From the archive to the knowledge base

The lab's archivist digitizes each item and catalogues it with a six-digit item number and its file type. The workflow then routes each item by its format. A vision model transcribes the handwritten notebooks and scanned pages, and describes the photographs. Typed documents have their text extracted, and audio recordings are transcribed from speech. Video is never described by a model: the archivist writes a description and a link for each video.

Every passage that comes out of a branch carries its item number and a link to the source file. This is what lets Mort cite the archive, and what lets anyone check an answer against the page it came from. The knowledge base uses multimodal embedding, so the image of each original page is stored next to its transcription and retrieval can return the page itself.

Heilig's handwriting is the hardest material. Multimodal language models do most of the reading here, and they bring problems of their own. A model sometimes refuses a page because of its content filter, which is why refused pages go to a fallback model. Their output also always reads fluently, so a misread word looks as convincing as a correct one and is hard to catch in proofreading.

From a question to an answer

A visitor's question first reaches Estuary, which holds Mort's character profile and the memory of the conversation. When the question needs the archive, it is rewritten into a search query, and the knowledge base returns the four passages that look most relevant, ranked by similarity. Mort answers from those passages in the first person and cites their item numbers. When the passages do not support an answer, he says the archive is silent instead of filling the gap.

A question that does not need the archive, such as a greeting, is answered from the profile and the memory alone. Only answers that go through the knowledge base can be traced to the archive. Mort's knowledge ends in May 1997, when Heilig died, and output guardrails check each reply before the visitor sees it.

The research is practice-based. The problems below came up while building and refining Mort, and the evaluation grew out of them.

Speaking for someone who cannot answer back

Mort speaks in the first person as a man who died in 1997 and cannot correct him. The limits on what he says are part of the design for that reason. Heilig's family approved the project and will read the character transcripts, so the people closest to him have a say in what he is made to sound like. His knowledge ends in May 1997, so he is not made to comment on what came after. When the papers have no answer he says so, because a visitor has no way to tell a filled gap from a real one. Every answer that comes from the archive carries its item number, so the visitor is not asked to take Mort's word for it. The case the evaluation cares about most, an answer that sounds like Heilig and departs from the record, is the one that misleads a visitor most easily, and it puts in Heilig's mouth a view he may never have held.

Two axes

Each answer is checked on two axes. Archive fidelity asks whether it agrees with Heilig's archive as it is on the original pages. It is judged claim by claim, since one answer can mix supported and unsupported claims. When the archive has no answer, Mort keeps archive fidelity by saying so, and when the archive gives different accounts at different times, by reporting both. Character fidelity asks whether the answer sounds and reasons like Heilig as he comes across in his own writing.

Crossing the two axes gives four cases. The one a visitor is least likely to notice sounds like Heilig and departs from the record, because it is convincing.

A faithful answer that is still false

In a 1994 lecture, Heilig said that a hundred thousand people died in the bombing of Bremen. Published records give a far lower number. In archival terms the lecture is authentic but not reliable on this fact. Mort, faithful to the record, repeats the figure in Heilig's voice, so the answer is right on both axes and still false. Answers like this are flagged as a problem of the source and kept apart from Mort's failures, following the distinction between faithfulness to a source and factual accuracy.

Tracing a failure to its source

To find where a failure comes from, each answer is traced along a chain from the original page to Mort's reply, and the failure is assigned to the earliest link where it goes wrong. Common RAG evaluations check whether the retrieved passages are relevant and whether the answer follows them. They take the knowledge base as given, so failures in coverage and transcription fall outside them.

Testing Mort

Mort is tested on a fixed set of questions, each with a gold answer tied to archive item numbers. The questions range from facts in a single document to answers that need several documents, and include questions the archive cannot answer. Each answer is scored for faithfulness to the gold answer, correct citations, correct refusal and knowledge leakage, and a second rater will score a subset.

Questions the archive cannot answer are not all alike, and each kind calls for a different response.

Refusal is tested in two versions on the same questions. A cold refusal says it does not know and closes the exchange. An in-character refusal says the record is silent, points to what the archive holds nearby, and stays in Heilig's register.

For character, Mort is interviewed in character and the sessions are recorded word for word. The review is planned with a psychologist and with Heilig's family, who read the fixed transcripts against a written protocol adapted from InCharacter. The report will state how many reviewers there were, since the sample is small.

The transcriptions in the knowledge base are being proofread first, and results will be reported after that.

A scripted playback of how Mort is designed to behave, based on Heilig's patents and his 1955 essay, all public documents. For each question the documents are checked, and the agent either answers and shows its evidence, corrects the question, or declines and says why.

Mort answers in text today. In a prototype on Slack, he can already bring up archival images and video during a conversation. The lab is building him as a MetaHuman character in VR with real-time voice, so that visitors can speak with him.

The larger goal is a Mort who guides visitors through his own archive in a headset, as an immersive documentary, with each answer traceable to the page it came from.

Outputs so far

• A working prototype of Mort in the lab.
• A paper on archive fidelity and character fidelity in Mort, in preparation.
• The same approach to building a knowledge base from an archive carries over to Memory Space: Chinatown AR, where I contribute to the knowledge base and the language model.