Libsoldout ========== Overview: 1. Introduction 2. Usage 3. Internals Introduction ------------ For some projects of mine, I wanted a lightweight C library that can parse John Gruber's [markdown](http://daringfireball.net/projects/markdown/) format into whatever I want, and that is easily extensible. The only C implementations of markdown that I know of are [Discount] (http://www.pell.portland.or.us/~orc/Code/markdown/) and [PEG-markdown] (http://github.com/jgm/peg-markdown/tree/master). Discount seemed a little bit too integrated and focused on HTML output for my taste, and PEG-markdown seemed to have a lot of dependencies and stuff. So I wrote my own. I like to keep things simple, so I wrote a function that performs *only* markdown parsing: no file reading or writing, no (X)HTML considerations, etc. The actual output is performed by a set of dedicated callback functions, called here a renderer. Some example renderers are provided, but you are free to use your own to output in any format you like. This callback mechanism make libsoldout so flexible that it does not need any flag or external information besides input text and renderer to operate. Usage ----- ### Library function call The only exported function in libsoldout is `markdown()`: void markdown(struct buf *ob, struct buf *ib, const struct mkd_renderer *rndr); - `ob` is the output buffer, where the renderer will append data, - `ib` is the input buffer, where the markdown text should be stored prior to the `markdown()` call, - `rndr` is a pointer to the renderer structure. How to use these structures is explained in the following sections. ### Buffers: struct buf I use `struct buf` extensively in input and output buffers. The initial idea was constructing a Pascal-string like structure, to be able to store both text and binary data. Hence the members `data`, a char pointer to the buffer data, and `size` containing the data length. When using a `struct buf` as an output buffer, it is useful to pre-allocate the memory area before filling it, so I added an `asize` member containing the allocated size of the memory pointed by `data`. When accumulating data in a growing memory area, there is a trade-off between memory usage and speed: the more bytes are added each time, the less `realloc()` is called, which means potentially less `memcpy()` to a new zone, so a faster code, but more memory being allocated for nothing. To set the trade-off on a case-by-case basis, there is a `unit` member in the structure: when more memory is needed, `asize` is augmented by a multiple of `unit`. So the larger `unit`, the more memory is allocated at once, the less `realloc()` is called. To further improve code efficiency by removing unneeded memcpy, I added a reference count to the structure: the `ref` member. Buffers are created using `bufnew()` whose only argument is the value for `unit`. `bufrelease()` decreases the reference count of a buffer, and frees it when this count is zero. `bufset()` is used to set a `struct buf` pointer to point to the given buffer, increasing reference count and dealing with special cases like volatile buffers. Usually data from `struct buf` are read through direct access of its members `data` and `size`. One interesting trick which might not be widely known is how to printf a buffer (or any kind of non-zero-terminated string) that doesn't contains any zero, using the `%.*s`. For example: printf("Buffer string: \"%.*s\"\n", (int)buf->size, buf->data); In case you really need a zero-terminated string, you can call `bufnullterm()` which appends a zero character without changing `size`, hence the buffer being virtually the same (and will no longer be zero-terminated after the following data append) but `data` can be used as a regular C string. The most common functions to append data into buffers are: - `bufprintf()` which behaves like any \*printf function, - `bufput()` which is similar to `memcpy()`, - `bufputs()` which appends a zero-terminated string to a buffer, - `BUFPUTSL()` which is a macro to replace `bufputs()` when using string literals, because then the data size is known at compile-time, this saves a call to `strlen()`, - `bufputc()` for single-character appends. Modification of existing data in a buffer is also performed through direct access of structure members. This covers the basics to handle my `struct buf`, but there might still be some interesting stuff to be learned from the header. ### Renderer: struct mkd_renderer Libsoldout only performs the parsing of markdown input, the construction of the output is left to a *renderer*, which is a set of callback functions called when markdown elements are encountered. Pointers to these functions are gathered into a `struct mkd_renderer` along with some renderer-related data. I think the struct declaration is pretty obvious: struct mkd_renderer { /* document level callbacks */ void (*prolog)(struct buf *ob, void *opaque); void (*epilog)(struct buf *ob, void *opaque); /* block level callbacks - NULL skips the block */ void (*blockcode)(struct buf *ob, struct buf *text, void *opaque); void (*blockquote)(struct buf *ob, struct buf *text, void *opaque); void (*blockhtml)(struct buf *ob, struct buf *text, void *opaque); void (*header)(struct buf *ob, struct buf *text, int level, void *opaque); void (*hrule)(struct buf *ob, void *opaque); void (*list)(struct buf *ob, struct buf *text, int flags, void *opaque); void (*listitem)(struct buf *ob, struct buf *text, int flags, void *opaque); void (*paragraph)(struct buf *ob, struct buf *text, void *opaque); void (*table)(struct buf *ob, struct buf *head_row, struct buf *rows, void *opaque); void (*table_cell)(struct buf *ob, struct buf *text, int flags, void *opaque); void (*table_row)(struct buf *ob, struct buf *cells, int flags, void *opaque); /* span level callbacks - NULL or return 0 prints the span verbatim */ int (*autolink)(struct buf *ob, struct buf *link, enum mkd_autolink type, void *opaque); int (*codespan)(struct buf *ob, struct buf *text, void *opaque); int (*double_emphasis)(struct buf *ob, struct buf *text, char c, void *opaque); int (*emphasis)(struct buf *ob, struct buf *text, char c,void*opaque); int (*image)(struct buf *ob, struct buf *link, struct buf *title, struct buf *alt, void *opaque); int (*linebreak)(struct buf *ob, void *opaque); int (*link)(struct buf *ob, struct buf *link, struct buf *title, struct buf *content, void *opaque); int (*raw_html_tag)(struct buf *ob, struct buf *tag, void *opaque); int (*triple_emphasis)(struct buf *ob, struct buf *text, char c, void *opaque); /* low level callbacks - NULL copies input directly into the output */ void (*entity)(struct buf *ob, struct buf *entity, void *opaque); void (*normal_text)(struct buf *ob, struct buf *text, void *opaque); /* renderer data */ int max_work_stack; /* prevent arbitrary deep recursion */ const char *emph_chars; /* chars that trigger emphasis rendering */ void *opaque; /* opaque data send to every rendering callback */ }; The first argument of a renderer function is always the output buffer, where the function is supposed to write its output. It's not necessarily related to the output buffer given to `markdown()` because in some cases render into a temporary buffer is needed. The last argument of a renderer function is always an opaque pointer, which is equal to the `opaque` member of `struct mkd_renderer`. The name "opaque" might not be well-chosen, but it means a pointer *opaque for the parser, **not** for the renderer*. It means that my parser passes around blindly the pointer which contains data you know about, in case you need to store an internal state or whatever. I have not found anything to put in this pointer in my example renderers, so it is set to NULL in the structure and the callbacks don't use it. `emph_chars` is a zero-terminated string which contains the set of characters that trigger emphasis. In regular markdown, emphasis is only triggered by '\_' and '\*', but in some extensions it might be useful to add other characters to this list. For example in my extension to handle `` and `` spans, delimited respectively by "++" and "--", I have added '+' and '-' to `emph_chars`. The character that triggered the emphasis is then passed to `emphasis`, `double_emphasis` and `triple_emphasis` through the parameter `c`. Function pointers in `struct mkd_renderer` can be NULL, but it has a different meaning whether the callback is block-level or span-level. A null block-level callback will make the corresponding block disappear from the output, as if the callback was an empty function. A null span-level callback will cause the corresponding element to be treated as normal characters, copied verbatim to the output. So for example, to disable link and images (e.g. because you consider them as dangerous), just put a null pointer in `rndr.link` and `rndr.image` and the bracketed stuff will be present as-is in the output. While a null pointer in `header` will remove all header-looking blocks. If you want an otherwise standard markdown-to-XHTML conversion, you can take the example `mkd_xhtml` struct, copy it into your own `struct mkd_renderer` and then assign NULL to `link` and `image` members. Moreover, span-level callbacks return an integer, which tells whether the renderer accepts to render the item (non-zero return value) or whether it should be copied verbatim (zero return value). This allows you to only accept some specific inputs. For example, my extension for `` and `` spans asks *exactly* two '-' or '+' as delimiters, when `emphasis` and `triple_emphasis` are called with '-' or '+', they return 0. Special care should be taken when writing `autolink`, `link` and `image` callbacks, because the arguments `link`, `title` and `alt` are unsanitized data taken directly from the input file. It is up to the renderer to escape whatever needs escaping to prevent bad things from happening. To help you writing renderers, the function `lus_attr_escape()` escapes all problematic characters in (X)HTML: `'<'`, `'>'`, `'&'` and `'"'`. The `normal_text` callback should also perform whatever escape is needed to have the output looking like the input data. ### libsoldout extension: PHP-Markdown-like tables Tables are one of the few extensions that are quite difficult and/or hacky to implement using vanilla Markdown parser and a renderer. Thus a support has been introduced into the parser, using dedicated callbacks: - `table_cell`, which is called with the span-level contents of the cell; - `table_row`, which is called with data returned by `table_cell`; - `table`, which called with data returned by `table_row`. The input format to describe tables is taken from PHP-Markdown, and looks like this: header 1 | header 2 | header 3 | header 4 ------------|:-------------:|--------------:|:-------------- first line | centered | right-aligned | left-aligned second line | centered |: centered :| left-aligned third line |: left-aligned | right-aligned | right-aligned : column-separator | don't need | to be | aligned in the source | extra spectators | are allowed | at both ends | of the line | | correct number of cell per row is not enforced | | pipe characters can be embedded in cell text by escaping it: \| | Each row of the input text is a single row in the output, except the header rule, which is purely syntactic. Each cell in a row is delimited by a pipe (`|`) character. Optionally, a pipe character can also be present at the beginning and/or at the end of the line. Column separator don't have to be aligned in the input, but it makes the input more readable. There is no check of "squareness" of the table: `table_cell` is called once for each cell provided in the input, which can be a number of times different from one row to the other. If the output *has* to respect a given number of cell per row, it's up to the renderer to enforce it, using state transmitted through the `opaque` pointer. The header rule is a line containing only horizontal blanks (space and tab), dashes (`-`), colons (`:`) and separator. Moreover, it *must* be the second line of the table. In case such a header rule is detected, the first line of the table is considered as a header, and passed as the `head_row` argument to `table` callback. Moreover `table_row` and `table_cell` are called for that specific row with `MKD_CELL_HEAD` flag. Alignment is defined on a per-cell basis, and specified by a colon (`:`) at the very beginning of the input span (i.e. directly after the `|` separator, or as the first character on the line) and/or at the very end of it (i.e. directly before the separator, or as the last character on the line). A cell with such a leading colon only is left-aligned (`MKD_CELL_ALIGN_LEFT`), one with a trailing colon only is right-aligned (`MKD_CELL_ALIGN_RIGHT`), and one with both is centered (`MKD_CELL_ALIGN_CENTER`). A column-wise default alignment can be specified with the same syntax on the header rule. ### Renderer examples While libsoldout is designed to perform only the parsing of markdown files, and to let you provide the renderer callbacks, a few renderers have been included, both to illustrate how to write a set of renderer functions and to allow anybody who do not need special extensions to use libsoldout without hassle. All the examples provided here come with two flavors, `_html` producing HTML code (self-closing tags are rendered like this: `
`), and `_xhtml` producing XHTML code (self-closing tags like `
`). #### Standard markdown renderer `mkd_html` and `mkd_xhtml` implement standard Markdown to (X)HTML translation without any extension. #### Discount-ish renderer `discount_html` and `discount_xhtml` implement on top of the standard markdown *some* of the extensions found in Discount. Actually, all Discount extensions that are not provided here cannot be easily implemented in libsoldout without touching to the parsing code, hence they do not belong strictly to the renderer realm. However some (maybe all, not sure about tables) extensions can be implemented fairly easily with libsoldout by using both a dedicated renderer and some preprocessing to make the extension look like something closer to the original markdown syntax. Here is a list of all extensions included in these renderers: - image size specification, by appending " =(width)x(height)" to the link, - pseudo-protocols in links: * abbr:_description_ for `...` * class:_name_ for `...` * id:_name_ for `...` * raw:_text_ for verbatim unprocessed _text_ inclusion - class blocks: blockquotes beginning with %_class_% will be rendered as a `div` of the given class(es). #### Natasha's own extensions `nat_html` and `nat_xhtml` implement on top of Discount extensions some things that I need to convert losslessly my existing HTML into extended markdown. Here is a list of these extensions : - id attribute for headers, using the syntax _id_#_Header text_ - class attribute for paragraphs, by putting class name(s) between parenthesis at the very beginning of the paragraph - `` and `` spans, using respectively `++` and `--` as delimiters (with emphasis-like restrictions, i.e. an opening delimiter cannot be followed by a whitespace, and a closing delimiter cannot be preceded by a whitespace). - plain `` without attribute, using emphasis-like delimiter `|` Follows an example use of all of them: ###atx_id#ID was chosen to look nice in atx-style headers ### setext_id#Though it will also work in setext-style headers ---------------------------------------------------------- Here is a paragraph with --deleted-- and ++inserted++ text. I use CSS rules to render poetry and other verses, using a plain `` for each verse, and enclosing each group of verses in a `

`. Here is how it would look like: (verse)|And on the pedestal these words appear:| |"My name is Ozymandias, king of kings:| |Look on my works, ye Mighty, and despair!"| Internals --------- Here I explain the structure of `markdown.c`, and how this parser works. I use a logical order, which is roughly chronological, which means going roughly from the bottom of the file to the top. ### markdown() The markdown function is divided into four parts: setup of the `struct render`, first pass on the input, actual parsing, and clean-up. #### render structure A `struct render` is passed around most of the functions, and it contains every information specific about the render. `make` is a copy of the `struct mkd_renderer` given to `markdown()`. The rendering callbacks are actually called from there. `refs` is a dynamic sorted array of link references (`struct link_ref`). It is filled from the input file during the first pass. A link reference is a structure of three buffers, `id`, `link` and `title`, whose functions are straightforward. `work` is a dynamic array of working buffers. Short-lived working buffers are needed throughout the parser, and doing a lot of `malloc()` and `free()` is quite inefficient. Instead, when a working buffer is allocated, it is kept in this array to be reused next time a working buffer is needed. `active_char` is a C array of function pointers, used for span-level parsing: a null pointer is affecter to all inactive characters, and a specialized callback is stored for active characters. This initialization is the bulk of the first part, because characters should only be marked active when the rendering callback pointer is non-null. #### First pass on the input During the first pass on the input, newlines are normalized and reference lines taken out of the input, and stored into `rndr.refs`. It makes use of the helper function `is_ref()`, which parses the given line, checking whether it matches the reference syntax. Offsets of the reference components are kept while progressing in the line, and on the first syntax error 0 is returned and the line is considered as an input line. When all the tests are passed, a new `struct link_ref` is created and sorted into `rndr.refs`. #### Second pass `markdown()` does not do much here, the result of the first pass is fed to `parse_block()` which fills the output buffer `ob`. #### Clean-up References allocated during the first pass, and working buffers allocated during the second pass are freed there, before returning. ### Block-level parsing The core of block-level parsing is the function `parse_block()`, which runs over the whole input (on the first call, the input is the output on the first pass, but `parse_block()` can be called recursively for blocks inside blocks, e.g. for blockquotes). The kind of block at the beginning of the input is determined using the `prefix_*` functions, then the correct `parse_` function is called for the current block. All specialized `parse_` functions returns a `size_t` which is the size of the current block. This lets `parse_block()` know where to start looking for the following block. Some blocks are easy to handle, for example blocks of code: the `parse_blockcode()` functions only scans the input, accumulating lines in a working buffer after stripping the blockcode prefix, and stopping at the first non-empty non-blockcode-prefixed line. It then calls the rendering function for block codes and returns. Other blocks are more complicated, like paragraphs who can actually be setext-style headers, or list items, which require a special subparse to follow Markdown rules where sublist creation is more laxist than list creation. Most block functions call `parse_inline()` for span-level parsing, before handing the result to the block renderer callback. #### HTML block parsing Of interest is the `parse_htmlblock()` function: according to Markdown webpage, HTML blocks must be delimited by unindented block-level tags, with the opening tag being preceded by a blank line, and the closing tag being followed by a blank line. When looking at the reference implementation, `Markdown.pl`, it appeared that when this doesn't find a match, a more laxist syntax is tried, where the closing tag can be indented, it only has to be at the end of line and followed by a blank line. But when looking at the test suite, it appeared that a single line `

foo
` surrounded by blank lines should be recognized as a block, regardless of the "matching" unindented closing tag at the end of the document. This meant that only the laxist approach should be used. This why the first pass is commented with a `#if 0`. If you want a strict HTML block parsing, as described on the webpage, you should instead comment the second pass. Keeping both first and second passes yields the same behaviour as `Markdown.pl` v1.0.1. I have to admit I do not really care that much about these differences, as I do not intend to use personally any inline HTML, because I will either parse unsafe input, then inline HTML is too dangerous, or my own input, but I use Markdown when I'm not confident in my HTML correctness, so it would be useless to include HTML in my input. However I am aware this feature can matter for some people, and any patch or suggestion to "fix" this behaviour will be welcome. ### Span-level parsing The core of span-level parsing is the function `parse_inline()`, which is pretty different from `parse_block()`. It is based around the `active_char[]` vector table in the render structure. The main loop is composed of two parts : first the next active character is looked for. The string of inactive characters is directly handed over to `normal_text` rendering callback. When a character is active, its corresponding entry in the `active_char[]` is a pointer to one of the `char_*`functions. Most of these functions do a pretty straightforward work in handling their role. The most complicated of these functions is `char_link`, which responds to `'['`. This is because of the many possibilities offered by markdown to use this character : it can either be a part of a link or an image, and then it can be inline or reference style or a shortcut reference style. Emphasis is another interesting piece of code, in that when encountering an emphasis character, it first looks whether it is single or double or triple emphasis, an then goes forward looking for a match. ### Proof that recursion depth is bounded by `max_work_stack` The core of the code here is that when entering the functions `parse_inline()` and `parse_block()`, if the current size of the working buffer stack (`rndr->work`) is above `max_work_stack`, the parsing is short-circuited and the input is appended as-is. Let's prove now that this actually works, i.e. that it does put an upper bound on the nested function call depth. **Step 1**: there is no function calling itself directly in `markdown.c`. This is quite easy to check, though a bit tedious. This proves that a stack overflow involves a recursion cycle of a least two functions. **Step 2**: most of the functions in `markdown.c` are declared by their definition, which means that these functions can only call functions appearing before them in the source file. This provides a strict hierarchy, which prevents any multiple-function recursion cycle. So only exceptions to the hierarchy are left to check. **Step 3**: there are only 3 functions that break the above-mentioned hierarchy: * `markdown()`, which is declared through the inclusion of `markdown.h` at the very beginning of `markdown.c`. However an easy text search shows that it's actually never called here, which obviously prevents it from being part of a recursion cycle. * `parse_block()`, which is declared at the beginning of the block-level section, but defined at the end. * `parse_inline()`, which uses functions pointer to dispatch active character handling towards `char_*` functions below. So at this point I have proved that any recursion cycle *always* involves `parse_block()` or `parse_inline()`. So checking a depth-indicator only in these functions is enough to prevent recursion cycles. **Step 4**: `rndr->work.size` is a good depth-indicator, because all calls to `parse_block()` or `parse_inline()` happen after at least one working buffer allocation. This is again a bit tedious to check: * `parse_block()` is called in `markdown()`, which is irrelevant, and in `parse_blockquote()` and `parse_listitem()`, which allocate respectively one and two working buffers at the very beginning of the function; * `parse_inline()` is called in `parse_emph1()`, `parse_emph2()`, `parse_emph3()`, `char_link()`, `parse_paragraph()` (twice), and each time it's called right after allocating a new working buffer; and in `parse_listitem()` which allocates two working buffers at the very beginning of the functions. Therefore, `rndr->work.size` will always increase between calls of `parse_block()` or `parse_inline()`, which in turns proves that putting an upper bound on `rndr->work.size` prevents arbitrarily deep recursions, and therefore stack overflows when the upper bound is well chosen. ### Utility functions Throughout the parsing the need of a working buffer frequently arise. A naive approach is to allocate a working buffer each time one is needed, and release it afterwards. However it leads to a lot of allocations, deallocations and reallocations (when the buffer grows), which costs a lot of time. So I added a `work` dynamic array pointer, which a special meaning to the `size` and `asize` members: in this array, The `size` first members are active working buffers that are still in use, and the remaining members up to `asize` are allocated but no longer used working buffers. When a function needs a working buffer, it first compare `size` to `asize`. When they are equal, it means there is no available working buffer, and a new one is created and appended (`push`ed) to the array. Otherwise it increases `size` and takes the already-allocated buffer as its working buffer, resetting its size. When the working buffer is no longer needed, the `size` of the array is just decreased, meaning the buffer is still allocated but ready to be taken by the next function in need. When the parsing is over, every working buffer should be marked as ready to be reused, hence the assertion of `size` being zero in `markdown()`. The buffers in the array are finally freed.