Overview
What began as an investigation into mojibake and malformed characters across my homelab documentation developed into a broader documentation-integrity engineering effort.
The visible encoding errors were only one part of the problem.
My documentation now feeds multiple downstream systems: version control, automated inventories, AI/RAG retrieval, generated source packs, and public portfolio material. Repairing individual strings without understanding which copy was authoritative could have created more drift than it removed.
The remediation therefore became an exercise in source authority, deterministic transformation, provenance, validation, and change control.
The Problem
The initial audit identified character sequences consistent with encoding corruption.
But not every unusual character was defective.
The corpus also contained:
-
valid Unicode;
-
historical evidence;
-
deliberately malformed validation fixtures;
-
generated representations;
-
terminal/display anomalies;
-
current canonical documentation.
That meant a global search-and-replace was unacceptable.
Before changing anything, each finding had to answer two questions:
-
Is the source actually corrupted?
-
Is this the authoritative copy that should be repaired?
Canonical First
The resulting architecture establishes a clear source hierarchy:
Canonical Documentation
↓
Audit & Validation
↓
Controlled Change
↓
Derived Generation
↓
Integrity Validation
↓
AI / Retrieval Consumers
Human-authored canonical documentation remains authoritative.
Generated AI/RAG material is treated as derived and rebuildable. If a generated representation contains an error inherited from its source, I correct the canonical source and regenerate the derivative rather than editing both independently.
This prevents two copies of the same logical document from developing separate histories.
Controlled Remediation
The encoding-remediation process used a bounded workflow:
Recovery Point
↓
Read-Only Inventory
↓
Finding Classification
↓
Prove Transformation
↓
Back Up Canonical Targets
↓
Transactional Repair
↓
Strict UTF-8 Validation
↓
Hash / Integrity Validation
↓
Derived Regeneration
The completed remediation audited 49 files and required repairs to three canonical files.
Nine derived retrieval bundles were subsequently regenerated from the corrected canonical state.
Post-remediation validation produced zero actionable mojibake findings while deliberately preserving historical evidence and malformed test fixtures where those artifacts still served a valid purpose.
Fail Closed, Don’t Guess
One of the most important design decisions was requiring a repair transformation to be deterministic.
A suspicious character sequence was not replaced simply because another character looked more appropriate.
Where the encoding transformation could be proven, the repair could proceed.
Where it could not, automation stopped.
That distinction matters in any knowledge corpus:
A stopped automation creates a review task. A guessed transformation can silently rewrite institutional knowledge.
Encoding Integrity Wasn’t Enough
The remediation uncovered another category of problem.
Some documentation was perfectly valid UTF-8 but contained stale lifecycle statements.
A current document might still describe a program step as waiting for regeneration after that regeneration had already occurred elsewhere in the accepted program state.
Encoding validation cannot detect that.
The integrity model therefore expanded:
Documentation Integrity
=
Byte / Encoding Integrity
+
Lifecycle / State Integrity
Current-state documents were reconciled against accepted program state while historical documents remained untouched as point-in-time evidence.
Provenance by Design
Generated knowledge artifacts now retain or allow reconstruction of provenance such as:
-
stable source identity;
-
source-relative location;
-
canonical-plane designation;
-
file size and modification time;
-
SHA-256;
-
generation/inventory time;
-
lifecycle classification;
-
ingestion state;
-
sensitivity and exclusion decisions.
Machine-generated manifests containing filenames use explicit UTF-8.
A lossy report is never allowed to become path authority.
AI/RAG Is a Consumer, Not an Authority
The same model governs my local AI environment.
Approved documentation is exported deterministically into a separate derived knowledge plane.
Retrieval is bounded by lifecycle and provenance controls, while the AI-facing knowledge interface remains read-only.
The retrieval system can help identify conflicts or generate proposals, but it does not silently modify canonical documentation or infrastructure.
Additional knowledge domains are also default-deny rather than automatically inheriting access.
Results
| Measure | Result |
|---|---|
| Files audited | 49 |
| Canonical files repaired | 3 |
| Derived retrieval bundles regenerated | 9 |
| Strict UTF-8 validation | PASS |
| Unexpected BOM | None |
| Actionable mojibake after remediation | 0 |
| Integrity/hash validation | PASS |
| Rollback validation | PASS |
| Historical evidence preservation | PASS |
The result was not simply cleaner Markdown.
It established a controlled chain from human-authored documentation to machine-consumed knowledge.
Lessons Learned
Encoding errors are symptoms
Mojibake demonstrates that an encoding boundary failed somewhere. Without sufficient evidence, it does not prove which application or command caused the original corruption.
Unicode is not corruption
Automated remediation must distinguish valid international characters and symbols from actual encoding damage.
Canonical authority must be explicit
Once the same knowledge exists in documentation, Git, AI indexes, generated reports, and retrieval bundles, source authority becomes an architectural control.
Derived knowledge should be rebuildable
If an AI knowledge mirror cannot safely be regenerated, it has effectively become another undocumented source of truth.
Integrity includes meaning
A document can have perfect encoding and hashes while describing an obsolete operational state.
Preserve evidence
A zero-finding scanner report is less important than having an understood disposition for every remaining finding.
Outcome
The resulting lifecycle is:
Authoritative Human Source
↓
Controlled Change
↓
Integrity + Lifecycle Validation
↓
Reviewed Promotion
↓
Deterministic Derived Generation
↓
Provenance / Checksum Validation
↓
Bounded Retrieval
The encoding incident ultimately became an opportunity to engineer a more reliable knowledge-management system.
For AI-assisted infrastructure work in particular, that foundation matters: better retrieval cannot compensate for corrupted, ambiguous, or stale source material.
Security Boundary
This public case study intentionally excludes internal addressing, hostnames, credentials, administrative paths, detailed topology, private knowledge content, recovery artifacts, firewall rules, and other operationally sensitive information.
The focus is the engineering methodology rather than the protected environment.