Skip to content

Have you ever wondered how org-mode toggles the visibility of headings?

See comments on r/emacs.

Have you ever wondered how org-mode toggles the visibility of headings?

YES!!! Me too!

Let's get into it ;)

org-mode visibility of headings

org-mode is built on top of outline-mode which is responsible for the visibility changes of the headings.

How does it work?

outline-mode uses overlays, specifically the overlay property invisible to toggle the visibility of the headings:

  1. To hide the body of a heading, outline-mode makes an overlay from the end of the line of the heading to the end of the body of the heading, and sets the property invisible of the overlay to be the symbol outline. Hence, the part of the buffer with this overlay is "replaced" (visually, not the content of the buffer) by ellipsis. Why is this? Because, when outline-mode is turned on, it adds the cons (outline . t) to the variable buffer-invisibility-spec which becomes buffer local and is responsible for the invisibility of each buffer.
  2. To make the body of a heading visible, outline-mode removes any overlays in the body of the heading that have its property invisible set to the symbol outline.

To see exactly how this is achieved you can refer to the functions outline-flag-region, outline-hide-entry and outline-show-entry defined in the file lisp/outline.el and also the definition of the mode outline-mode in the same file.

You can get emacs's source code by running the following command:

git clone git://git.sv.gnu.org/emacs.git

OK!

The mechanism of outline-mode uses overlays, buffer-invisibility-spec and ellipsis.

But how do those "emacs/elisp features" play together?

In the next parts of this post, we build examples using them to try to get a good feel for their use.

Text properties

In a buffer, each character point can have text properties attached to it that can be used to do many things (like controlling the appearance of the character).

For instance, in an emacs-lisp-mode buffer, with the following s-exp, and the cursor (the point) after the first parenthesis:

(setq my-var nil)

if we run:

M-x eval-expression RET (text-properties-at (point))

we get:

(face font-lock-keyword-face fontified t)

The character point "s" (point 2, i.e. the "s" at the second position in the buffer) has:

  1. the text property face equal to the face font-lock-keyword-face which is why it is displayed with a different foreground color (depending on your theme) than the text my-var for instance,
  2. the text property fontified equal to t which we don't describe here.

We can read more about the special text properties in the manual ("Elisp Special Properties").

If we want more information (not only the text properties) about the character point "s" (point 2), we can run (still with with the cursor after the first parenthesis):

M-x describe-char

which pops up the following help buffer:

             position: 2 of 18 (6%), column: 1
            character: s (displayed as s) (codepoint 115, #o163, #x73)
              charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x73
               script: latin
               syntax: w    which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET 73" or "C-x 8 RET LATIN SMALL LETTER S"
          buffer code: #x73
            file code: #x73 (encoded by coding system prefer-utf-8-unix)
              display: by this font (glyph code)
    ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x56)

Character code properties: customize what to show
  name: LATIN SMALL LETTER S
  general-category: Ll (Letter, Lowercase)
  decomposition: (115) ('s')

There are text properties here:
  face                 font-lock-keyword-face
  fontified            t

Note that your result may differ in your running emacs (different fonts, maybe information about overlays if you are using hl-line-mode, …).

Why are we talking about text properties if the mechanism of outline-mode uses overlays?

Because:

  1. Both text properties and overlays can "alter/control" the appearance of the buffer's text on the screen and so we have to know something important that is (from the manual "Elisp Overlay Properties"):

    all overlays take priority over text properties.

  2. buffer invisibility can also be achieve with text properties (for instance, this is what org-mode does to hide the brackets and the link part of links like this [[link][description]]), and it is important to notice it.

Overlays, invisible overlay property, buffer-invisibility-spec

We can make a part of a buffer invisible using:

  1. the invisible text property (of that part),
  2. the invisible overlay property ("on top of that part").

The "admitted" values of the invisible overlay property (or text property) and the invisibility effect expected depend on the value of the variable buffer-invisibility-spec.

In this section:

  1. we define overlays,
  2. we set the variable buffer-invisibility-spec,
  3. we give different values to the invisible property of the overlays,
  4. we observe the appearance of the buffer,
  5. we repeat step 2) to 4) several times.
  6. we hope we get a good feeling of invisibility in Emacs.

Also note that all the evaluations of the s-expressions are done in the minibuffer with M-x eval-expression and the point in the buffer we operate on, that we call *invisible*.

Let's switch to the new "fresh" buffer *invisible* in fundamental-mode by evaluating the following s-exp:

(switch-to-buffer (get-buffer-create "*invisible*"))

Let's insert the characters XXXXXX at the beginning of the buffer *invisible*:

XXXXXX

buffer-invisibility-spec equal to t

Now if we evaluate the variable buffer-invisibility-spec, we should get t (the default) in the echo area.

If not, we set this variable to t like this:

(setq buffer-invisibility-spec t)

Now, we make an overlay "on top" of XXXXXX (from point 1 to point 7 in the buffer) that we assign to the variable ov-x using make-overlay:

(setq ov-x (make-overlay 1 7))

and we see the following in the echo area:

#<overlay from 1 to 7 in *invisible*>

Now, by setting the property invisible of the overlay ov-x to t using the function overlay-put like this

(overlay-put ov-x 'invisible t)

we make the characters XXXXXX disappear.

This is due to the value of buffer-invisibility-spec equal to t (the default) which means that text is invisible if it has a non-nil invisible (text or overlay) property.

Now, evaluating the following s-exp sets invisible property of the overlay ov-x to nil

(overlay-put ov-x 'invisible nil)

makes the characters XXXXXX to reappear in the buffer *invisible*.

We also could have removed the overlay ov-x to make the characters XXXXXX to reappear. Let's see how.

First, as previously, we set the invisible property of the overlay ov-x to t to make the characters XXXXXX to disappear:

(overlay-put ov-x 'invisible t)

Then, instead of setting back the invisible overlay property to nil of ov-x we remove it. To do so, we use the function remove-overlays that let you remove all the overlays in a range of the buffer that have a specific property set to some value (in our case the property invisible set to t in the range 1 to 7 of the buffer).

So evaluating the following s-exp:

(remove-overlays 1 7 'invisible t)

removes the overlay ov-x in the buffer *invisible* and make the characters XXXXXX to reappear.

buffer-invisibility-spec equal to nil

As we removed the overlay ov-x, we redefined it as previously by evaluating the following s-exp:

(setq ov-x (make-overlay 1 7))

Let's set buffer-invisibility-spec to nil:

(setq buffer-invisibility-spec nil)

Then, by evaluating the following s-exp, we expect the characters XXXXXX to disappear:

(overlay-put ov-x 'invisible t)

BUT they don't.

This is normal, as we've just set buffer-invisibility-spec to nil, we've "disabled" the invisibility feature in the buffer *invisible*.

Now, we restore the invisible property of the overlay ov-x so as not to interfere with the next example by evaluating:

(overlay-put ov-x 'invisible nil)

buffer-invisibility-spec equal to ((foo) t)

Let's add the characters YYYYYY after the characters XXXXXX with 3 dashes --- in between such that the buffer *invisible* is now:

XXXXXX---YYYYYY

Now, we make an overlay "on top" of YYYYYY (from point 10 to point 16 in the buffer) that we assign to the variable ov-y using make-overlay:

(setq ov-y (make-overlay 10 16))

We set back buffer-invisibility-spec to t (the default):

(setq buffer-invisibility-spec t)

Then we add the list (foo) to the variable buffer-invisibility-spec using the function add-to-invisibility-spec as follow:

(add-to-invisibility-spec '(foo))

Now, the value of buffer-invisibility-spec is ((foo) t).

This implies that, now to make a part of the buffer invisible, the invisible property must be foo or t. Before, it could have been any value that is non-nil.

This way we can toggle the visibility of some parts of the buffer while other parts remain invisible (see org-toggle-link-display for instance).

Let's make XXXXXX disappear "permanently" by setting the invisible property of ov-x to t:

(overlay-put ov-x 'invisible t)

The characters XXXXXX disappear and the buffer *invisible* is now:

---YYYYYY

Now, we set the invisible property of ov-y to be equal to foo:

(overlay-put ov-y 'invisible 'foo)

The characters YYYYYY disappear and the buffer *invisible* is now:

---

Now, what we can do is to make YYYYYY appears again by removing (foo) from the invisibility spec buffer-invisibility-spec while the characters XXXXXX stay invisible:

(remove-from-invisibility-spec '(foo))

Now, the buffer *invisible* is:

---YYYYYY

Note that:

  1. the overlay ov-x still has its property invisible equal to t and,
  2. the overlay ov-y still has its property invisible equal to foo.

You can verify it by evaluating the following s-exp:

(overlay-get ov-x 'invisible) ; t
(overlay-get ov-y 'invisible) ; foo

Ellipsis and buffer-invisibility-spec equal to ((foo . t) t)

Default ellipsis

If the variable buffer-invisibility-spec as a list contains a cons (foo . t), every continuous part of the buffer with the invisible property set to foo is replaced by ellipsis which are by default ....

The buffer *invisible* still contains the characters XXXXXX---YYYYYY, but maybe not all the characters are visible. So let's put our buffer in an appropriate state for this section.

We removes all the overlays in the buffer (which makes all the content of the buffer visible again). We redifined the ov-x and ov-y as previously (same part of the buffer (1 to 7) and (10 to 16)). And we set buffer-invisibility-spec to be ((foo . t) t). We can do this by evaluating the following expression (in the minibuffer with point in the buffer *invisible*):

(progn
  (remove-overlays)
  (setq ov-x (make-overlay 1 7))
  (setq ov-y (make-overlay 10 16))
  (setq buffer-invisibility-spec t)
  (add-to-invisibility-spec '(foo . t)))

The buffer *invisible* is now:

XXXXXX---YYYYYY

By evaluating the following s-exp, we set the invisible property of the overlay ov-y to foo

(overlay-put ov-y 'invisible 'foo)

and this replaces (visually not the content of the buffer) the characters YYYYYY by the default ellipsis ... and the buffer *invisible* looks like this:

XXXXXX---...

Custom ellipsis modifying the display table

We assume with the buffer *invisible* is in the same state as in the previous section.

Our goal in this section is to modify the default ellipsis ....

To do so we:

  1. create a display table with the function make-display-table,
  2. we set its special slot 4 (responsible of the display of the ellipsis) which must be a vector of glyph using the function set-display-table-slot,
  3. we set the variable buffer-display-table of the buffer *invisible* to be this new display table,
  4. we observe the appearance of the buffer *invisible*.

So by evaluating the following s-exp:

(let ((tbl (make-display-table))
      (glyph-vector
       (vector (make-glyph-code ?\ 'font-lock-warning-face)
               (make-glyph-code ?\; 'font-lock-warning-face)
               (make-glyph-code ?- 'font-lock-warning-face)
               (make-glyph-code ?\) 'font-lock-warning-face))))
  (set-display-table-slot tbl 4 glyph-vector)
  (setq buffer-display-table tbl))

the buffer *invisible* should looks like this (if the invisible property of the overlay ov-y is still equal to foo):

XXXXXX--- ;-)

You can read more about character display and display table in the manual ("Elisp Character Display" docs).

WE ARE DONE :-)