LLMs Hallucinations & Ontological Shadows

*The Fabric of Hallucinations (Hieronymus Bosch)*

Preamble

Hallucinations represent a major drawbacks for LLMs producing too many outcomes factually incorrect or nonsensical. Different schemes can be used to reduce models’ hallucinations, from dedicated training and external knowledge source on the one hand, fine-tuning with techniques like Retrieval-Augmented Generation (RAG) or Chain-of-Verification (CoVe) on the other hand. But those approaches remain ad-hoc solutions that mostly rely on expertise; more principled solutions can be achieved using ontological references.

The Roots of Hallucinations

LLMs hallucinations stem from two intrinsic shortcomings:

The truth of discourses remains undecidable without factual references, but LLMs are meant to be trained on texts and references, not directly on facts
The meaning of discourses remains undecidable without reference categories, but LLMs are meant to identify emerging categories directly from mining digital environment

Addressing those shortcomings could remove the root causes of hallucinations; to that effect the first step is to frame LLMs behavior in terms of knowledge interoperability.

Given LLMs’ conversational purpose, queries are by nature nominal and set by intensional modalities as they may pertain to abstract or concrete topics, or actual or virtual ones.

Along that reasoning non-factual hallucinations happen when intensional modalities of queries are not aligned with extensional ones of facts: observed, asserted, assessed, deduced, or managed. Concomitantly, nonsensical hallucinations happen when the logical content of outcomes cannot be aligned with the features of identified (or supporting) categories, eg: identification (nominal or physical), instanciation (actual or virtual).

In theory both kinds of hallucinations could be mended by pairing LLMs with ontologies, in practice that’s not an option, if only because such ontologies would need to embrace all possible realities (facts) and all possible theories (categories). Still, if the truth of outcomes cannot be fully established, it may be possible to trace hallucinations by looking for their shadows.

The Weeding of Hallucinations

Shadows & Ghosts

Whatever the interlocutor, human or otherwise, the meaning of a query is set by its context and intent, making room for different interpretations; humans rely on intuition and pragmatics to sort out misunderstandings and pinpoint topics and intents, machines appear to lack both capabilities. If consistency proofs are off, a weeding protocol is necessary to remove hallucinations, that can be done with shadows and ghosts.

Shadows are categories that could represent facts or realise concepts; while they may or may not be present (hence the name), if found they can be used to bear out the relevancy of the element considered.

Ghosts are disembodied or orphan categories, the former for categories not representing facts, the latter for ones not realising concepts; they can serve as shadows’ counter-evidence hinting at hallucinations.

Weeding Protocols

Checking factual hallucinations (left):

The reality of a factual reference is in doubt (a)
A factual reference can be directly established (b)
No factual reference can be directly established (c)
Looking for alternative type with a factual reference (d)
An alternative (≈) type is found (e) with a confirmed realisation (f)
No alternative can be found with a confirmed realisation (g)

*Checking hallucinations for facts (left) and reasonings (right)*

Checking reasonings hallucinations (right):

The consistency of a reasoning is in doubt (a)
A category supporting the reasoning can be found (b)
No category can be directly found in support of the reasoning (c)
Looking for alternative categories supporting the reasoning (d)
An alternative (≈) category can be found that support the reasoning (e) and is consistent with facts (f)
No alternative category (or model) can be found that both support the reasoning and is consistent with facts (g)

At this point, while those protocols introduce some guide-rails, they are not as specific about the actual weeding of hallucinations as techniques like Retrieval-Augmented Generation (RAG) or Chain-of-Verification (CoVe). What is missing are operational probing methods.

How to Prob Shadows

The rationale behind the weeding approach is to find shadow categories that could corroborate questionable factual references or reasonings or, a contrario, using ghosts as proof of hallucinations.

Regarding factual references (diagram, left) the primary objective is to associate doubtful references (a) with undoubted ones (b) sharing the same provenance or vicinity (c):

Provenance: verified references sharing the source and/or space/time coordinates
Vicinity of the reference considered: verified factual references bound by the same structure, activity, or time

Modalities are then compared for intents (actual, virtual, …) and facts (observed, assessed, …) in order to identify shadows (d).

*Probing shadows for factual (left) and reasoning (right) hallucination*s

Compared to searches for factual references (diagram, left) based on shared identification mechanisms (#), searches for close undoubted reasonings (diagram, right) rely on scope and analogies (≈):

Scope: categories pertaining to the same domain that could support the reasonings under consideration
Analogies: categories sharing representation patterns (identification, agency, life cycle, …) that could support the reasonings under consideration

While the presence of shadow categories is not enough to dismiss hallucinations, they can suggest fine-tuned prompts or their absence and ghosts can finally substantiate hallucinations.

Representation

Queries about Mozart would entail different realities: actual and physical (Mozart as a person), symbolic (fictions about Mozart), or virtual (Mozart as a character).

*Excerpt of ontological references for “Mozart” queries (CaKe & OWL/Protégé)*

Taking advantage of a CaKe modalities, probing factual references will use prompts to compare intensions (actual or virtual) and extensions (observed fact or document) modalities. Concomitantly, the probing of reasonings will use prompts to find supporting patterns of identification (eg biological or regulatory) or agency.

LLMs Hallucinations & Ontological Shadows

Preamble

The Roots of Hallucinations