Tools or techniques for conveying mental models?

Write a book. Hire an editor.

Ontology logs are a great formalism for communicating ontologies. For long-lived processes with lots of inner state, state diagrams and state charts are a great way to document a system.

Users of a system usually cannot actually perceive the system's structure. There are multiple reasons for this. One reason is that Nakatomi space is only perceptible to users who misuse the system; there's a region of state space reserved for weird machines and other unintended behavior. Another reason is that the purpose of a system is what it does; a user might only perceive what the system does for them, generating a personal reason for the system's existence, without recognizing what the system's designers and maintainers intend for the system to do in full.

This is the central problem of education, right? You’ve listed two of the main categories - learning by doing and expert instruction; the third is teaching.

I think the relevant question is how you can make models that are easier to acquire as you go along (reuse of similar principles and designs) or render it unnecessary to fully acquire them (abstraction, with well defined leakiness).

In software engineering, I find it helpful to frame mental models around “user stories” and “technical implementation”. Generally user stories are primary requirements (what you want to archive) and technical implantation are secondary (what needs to happen to achieve your goals).

User stories describe the value you deliver to customers.

Technical implementation describe the constraints of the system you built that limits your user stories.

Sometimes these overlap and you have user stories constrained by technical implementations, or you choose technical implementations to achieve user stories. But many times you can frame system behaviors as one or the other.

From there, you choose whatever tool helps you the best. UML is good for entity maps. Flow charts are good for... flows. State diagrams are good for describing allowed and disallowed states and their transitions. Tables of variables and states to enumerate all values. Draw diagrams. Honestly the best way to learn how to draw systems is to ask 3 different people to draw what they think. You’ll be surprised by the variety of ways people express the same ideas. Most are equally valid but highly different aspects of the subject, just like art.

Teaching.

I'd draw a distinction between presentations meant to communicate (your conveying), and presentations meant to perform the task at hand (not mentioned, but "executing").

Executing models convey poorly, often because they have poor abstraction boundaries (in computing, almost always leaky). Figuring out what something is doing can be obscured by mountains of efforts put into making it do that efficiently, hence the need to scribble on napkins ("here is, at a high level, at the level of abstraction relevant to your current mental model, how the system functions").

Documentation is a separate artifact which tends to lag behind executing models, barring painstaking maintenance. Even more painstaking is coupling documentation directly to code, or even inverting the preeminence of documentation over code, a la literate programming.

So what often makes sense for communicating a mental model is exactly the minimal amount of effort and maintenance, i.e. scribbling on napkins and actually working with the system over time.

There's a reason new hires get started with fixing easy bugs, not writing design documents.

Writing.

Diagramming, mostly more formally, although I still break out jspaint now and again to doodle in realtime while sharing my screen (I also do this with Figjam et al if it's something someone else might find useful to look at later). Diagramming and pointing work because they're some of the oldest and still most useful tools for directing attention that we have. Even before we become verbal we point and cry; pointing remains much more immediate than language for location and navigation, hence the continued market for laser pointers as linguistic prostheses. Peter Gårdenfors' The Geometry of Meaning: Semantics of Conceptual Spaces goes into this in some detail. Barbara Tversky's got a ton of work in the general diagramming-and-structuring-cognitive-spaces vein also, and Peter Cheng's paper "What Constitutes an Effective Representation?" lays out some fairly quantitative criteria.