[neomutt-devel] (Re-)Building the Documentation

[Richard Russon]

Everything in NeoMutt is documented, but...
If you don't know what you're looking for, you're unlikely to find it.

# The Problem

The docs are:
- Huge
- Hard to maintain
- Hard to search
- Hard to build
- Heavily dependent on the code
- Entirely text

The guide needs to be split up and re-grouped, thematically:
a chapter on the pager; a chapter on IMAP, ...

Some projects' documentation have handy links: "Click here to improve
the docs".  It would be wonderful if we could do this.

# Document Format

The docs are written in DocBook XML.  It's a rich language allowing you
can make <note>Notes</note>, <example>Examples</example>,
<chapter>Chapters</chapter> and <section>Sections</section>.
As well as normal HTML things like titles and paragraphs.

If the docs were stored in a big XML file, things would be easy.
Open a DocBook-compatible editor and off you go...
Unfortunately, `manual.xml` isn't the start of the journey, it's the
middle!

# Generating manual.xml

This diagram shows the NeoMutt build process.  It includes the steps
needed to build manual.xml

    https://www.neomutt.org/images/make.png

To reduce the duplication of text, source of the source code is parsed
to generate the manual.

## OPS files

These are processed by a perl script `gen-map-doc`

    https://github.com/neomutt/neomutt/blob/master/OPS

    function ID            description

    OP_ATTACH_VIEW_MAILCAP "force viewing of attachment using mailcap"
    OP_ATTACH_VIEW_TEXT    "view attachment as text"
    OP_ATTACH_COLLAPSE     "Toggle display of subparts"

## functions.h

This is processed by a perl script `gen-map-doc`

    https://github.com/neomutt/neomutt/blob/master/functions.h

    function name       function ID    default key

    { "top-page",       OP_TOP_PAGE,   "H" },
    { "next-entry",     OP_NEXT_ENTRY, "j" },
    { "previous-entry", OP_PREV_ENTRY, "k" },

## init.h

This is processed by a program `makedoc`.  It reads the variable, its
type and default value.  It also processed the description which is
written in NROFF (man) format.

    https://github.com/neomutt/neomutt/blob/master/init.h

    { "ascii_chars", DT_BOOL, R_BOTH, OPTASCIICHARS, 0 },
    /*
    ** .pp
    ** If \fIset\fP, Mutt will use plain ASCII characters when
    ** displaying thread ** and attachment trees, instead of the default
    ** \fIACS\fP characters.
    */

## Some Assembly Required

Finally we can build `manual.xml`.  Take `doc/manual.xml.head` add the
functions, commands and variables, then add `doc/manual.xml.tail`

`manual.xml` is now complete and valid DocBook.

# Rendering manual.xml

DocBook comes with some standard XSLT files.  They define
transformations from DocBook to HTML.  Chapters are turned into separate
pages.  Sections are given \<h1\>, \<h2\> headings, etc.

Also, \<note\> and \<example\> are turned into \<span\> or \<div\> with classes,
so that they can be themed.  This is also where \<quote\>s are turned into
“smart quoted” strings.

We now have some HTML, that looks machine-generated.

# Generating the Website

NeoMutt's website is hosted by GitHub's Pages.  It's a static site which
uses Jekyll (https://jekyllrb.com) to allow some templating and "magic".

To turn manual.xml into a website requires quite a lot of post-processing.
The main script is `build` in the docbook repo:

    https://github.com/neomutt/docbook

`build` takes the HTML files and tidies them, cuts out the bits we want
and gives them a YAML header (for Jekyll).  It also fixes some of the
links between documents.  Along the way we also fix any formatting
problems that we introduced (smart quotes).

# The Future

Mutt's re-use strategy means that if you fix one piece of documentation,
then it will be replicated into all the forms of documentation.
But, it means you need to know where the source is.  This is preventing
users from helping us.

## Step 1 - Separation

The docs should be completely separate from the code.

Good:
- Docs can be maintained by non-technical users
- Simplifies build process
- Simplifies website generation
    
Bad:
- Requires synchronisation between docs and code

## Step 2 - Conversion

I like DocBook.  It's expressive and capable, but it's not user-friendly.
A good alternative would be Markdown.

Good:
- Easy to read and edit
- Can be converted to HTML
- Supported by our current website

Bad:
- Less expressive
- Initial conversion will take a lot of work

## Step 3 - Reorganisation

Once the docs are easily editable, then we can encourage users to help.

To do:
- Split guide into more manageable sections
- Add more HowTos - simple answers to specific problems
- More pictures of NeoMutt in action

# Conclusion

NeoMutt is a complex program to use.  It's a big problem for new users.

By re-working the docs, we can encourage new users to NeoMutt and make
sure that all users get the most out of NeoMutt.

These changes won't reduce the amount of work by much, but crucially,
they will reduce the technical difficulty of the work.

These are jobs what will transform NeoMutt!
(These are not jobs for the faint-hearted!)

---

What do you think?

Are *YOU* up for a challenge?

Please help.

Cheers,
    Rich / FlatCap
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 854 bytes
Desc: not available
URL: <http://mailman.neomutt.org/pipermail/neomutt-devel-neomutt.org/attachments/20170611/ddf31128/attachment.sig>