This page is part of a static HTML representation of the TiddlyWiki at https://tiddlywiki.com/dev/

TiddlyWiki Core Application

17th July 2014 at 8:38pm

The microkernel builds up the base functionality to manage tiddlers by providing a basic wiki store and a barebone tiddler model. The microkernel can also load a set of (decrypted) tiddlers and provides a module system, enabling a developer to extend the kernel and add functionality with module tiddlers and plug-ins.

For instance, the TiddlyWiki Core Application is provided as a single plug-in. In this part we want to focus on the TiddlyWiki core plug-in. After describing how new functionality is added directly to the wiki store, we show how the core plug-in realises persistence of tiddlers. The last part describes how TiddlyWiki builds an UI out of tiddlers and WikiText.

The TiddlyWiki Application consists of a microkernel and several modules building up the full application.

Startup Process

Modules with module-type: startup have to export a function named startup and may export a name property to identify this module. The startup function will be executed by the boot kernel at the end of the boot process.

// From boot.js:
// Gather up any startup modules
$tw.boot.remainingStartupModules = []; // Array of startup modules
$tw.modules.forEachModuleOfType("startup",function(title,module) {
	if(module.startup) {
		$tw.boot.remainingStartupModules.push(module);
	}
});
// Keep track of the startup tasks that have been executed
$tw.boot.executedStartupModules = Object.create(null);
$tw.boot.disabledStartupModules = $tw.boot.disabledStartupModules || [];
// Repeatedly execute the next eligible task
$tw.boot.executeNextStartupTask();

executeNextStartupTask() will execute the remaining startup modules in remainingStartupModules. A startup module can export the variables before and/or after, each containing an array of names of other startup modules. executeNextStartupTask() will use this information to execute the modules in the correct order.

Startup modules can be marked as synchronous by exporting synchronous = true. If synchronous is set to false, the startup function is executed with an callback function as the first argument. The startup function has to call this function to allow subsequent startup modules to get executed. This is necessary when the startup function itself uses asynchronous calls.

Extending the Store

Event Mechanism

Most of the following mechanisms need a way to get notified, when anything in the wiki store changes. The core -plug-in adds an event system to the bare wiki store. The event system provides the ability to listen to events. The most important is the "change" event which notifies the listeners when tiddlers have changed, with a list of the changed tiddlers. The event mechanism is one of the few mechanisms which needs a hook at the microkernel: The microkernel contains an empty function "enqueueTiddlerEvent(event)" and calls this function when a tiddler is added or deleted. The event mechanism from the core plug-in overwrites this function with it's own implementation. The functions providing the event system are added via the wikimethod module type.

Caching

The core plug-in adds two caches to the wiki store. A global cache and a cache for each tiddler. While the global cache is cleared whenever any tiddler changes, the tiddler cache is only cleared when the associated tiddler changes. The idea of the tiddler cache is, that sometimes we want to calculate information from a tiddler but don't want to recalculate every time we need this information. Imagine a tiddler which contains links to other tiddlers and at one point we want to have a list of the linked tiddlers (LinksFilter). With the tiddler cache we can parse the WikiText, extract the linked tiddlers and put this list in the cache. Until the tiddler is changed, the cache can be used to access this information in the next requests.

The same idea holds for the global cache. For example when a filter string is parsed and a list of matching tiddlers is constructed. This list can be put in the global cache for later reuse until something changes in the store.

Like the Event Mechanism, the cache needs hook functions in the microkernel. The microkernel calls clearGlobalCache() and clearCache(tiddlertitle) when a tiddler changes. The core's cache mechanism overwrites this functions. The functions providing the cache system are added via the wikimethod module type.

System Tiddlers

The core plug-in introduces a segregation of tiddlers. The tiddler model is central in the whole TiddlyWiki application and can be used in various roles. Because of the fact that a TiddlyWiki user works with tiddlers (taking the role of a wiki page) and tiddlers are the building blocks of the whole application (including modules, UI elements, etc.) we need to identify the internal tiddlers.

The core plug-in introduces the concept of system tiddlers. It builds on the convention that application internal tiddler names start with $:/. Then the core plug-in introduces a set of new functions to the wiki store which are used to retrieve tiddlers like getTiddlers(options) and forEachTiddler(callback, options). These functions work with all tiddlers in the store but the options parameter provides the ability to sort tiddlers by a field-name and exclude tiddlers with a specific tag. By default it doesn't return system tiddlers. To get a list of all tiddlers including system tiddlers, this must be requested explicitly via the options. If a function wants to present a list of tiddlers to the user it can use this new functions so that internal application tiddlers wouldn't clutter the resulting list. These functions are added via the wikimethod module type.

Tags and Filter Mechanism

Tags

The core plug-in extends the store by a simple mechanism to tag a tiddler. This provides the functionality to

  • retrieve tiddlers with a specific tag
  • retreive a hashmap of { tag: [tiddler1, tiddler3, tiddler3] }
  • list or iterate tiddlers not tagged with a specific tag

The tags are stored directly in the tiddler. Each tiddler can have a field named "tag", which contains an array of its tags. The above functions use this field to calculate their results and cache them in a global cache.

Organising information with tags is easy, intuitive and is common throughout the web. In most cases the functions mentioned above are not enough and a more sophisticated way of querying is needed. But instead of focusing only on tags TiddlyWiki introduces a querying system that isn't bound to tags or single tiddlers.

Filter mechanism

This filter mechanism is build on the idea of a pipeline of single filter operators. Filters are noted as strings and can look like

[tag[task]!tag[done]interesting[very]]

This example would (implicitly) put all available tiddlers into the pipe. The first operator tag[task] would only pass tiddlers which are tagged with "task". The !tag[done] operator is negated and only passes tiddlers which are not tagged with "done". The last filter operator passes only tiddlers with a field "interesting" set to "very". So as a result this filter would be used to obtain all tiddlers which are marked as task, aren't already done and are very interesting.

There are many filter operators already build into the core plug-in including the mentioned tag- and field operators, filter operators to sort the tiddlerlist by a specified field, etc. But more sophisticated operators are possible, too. An example would be the search-operator. This filter operator looks for the searched string in the text and in the tags of the tiddlers. If the provided filter operators are not enough, a developer can add new filters by adding a module with the filteroperator type.

System Tags

Tags and the filter mechanism are used throughout the core plug-in for internal puproses. Tags which start with $:/ are normally hidden from the casual user, similar to System Tiddlers

The filter mechanism is added to the wiki store with the wikimethod module type.

Extended Persistence

The microkernel only contains a bare store and some deserializers to load tiddlers from JSON files or from the DOM of the current HTML file. The core plug-in adds some more deserializers and a new mechanism for persisting and synchronising tiddlers.

This mechanism is provided as a global module in $:/core/modules/syncer.js. The saver module has three responsibilities:

  1. Save the whole wiki.
  2. Provide the ability to download single tiddlers as files.
  3. Synchronise the local wiki store with a remote wiki store, i.e. running in Node.js

The syncer module is connected mainly to two other modules. For one it registers to changes at the wiki store (Event Mechanism) and if any changes occur they are synced to the remote store. Then it provides a function saveWiki(options). This function can be used by other modules. For example the RootWidget uses this function to save the whole wiki or start downloading single tiddlers.

The syncer itself does not provide a concrete implementation of saving, downloading or syncing the tiddlers. Instead it loads modules of type saver and syncadaptor and manages the saving/syncing process.

Deserializer

Modules with module-type: tiddlerdeserializer can provide functions to create tiddlers out of any textual representation. Each function must be associated with a type like application/json or text/html. They get the textual representation of the tiddlers, some default field values and the type of the text block as arguments. They return an array of JavaScript objects representing the resulting tiddlers.

Deserializers are not managed by the syncer module. Instead the concept of deserializers is in fact part of the microkernel. This is necessary because the microkernel needs a way of loading tiddlers before it can load the core plug-in and execute it's startup modules. (Due to the fact that the core plug-in and it's modules are represented as tiddlers.)

The load-modules startup module loads additional deserializers and pushes them into the store. The core plug-in for example contains a few deserializers which can read a whole TiddlyWiki 5 or classic HTML file and load the individual tiddlers into the store.

Saver

Modules with module-type: saver provide functionality to save the whole wiki. There are three methods a saver can support:

save
This method is used, when the user requests a save, for example by clicking the save button in the sidebar.
autosave
This method is used automatically by TW when tiddlers are changed, created or deleted by the user.
download
This message is used when the wiki or a single tiddler should explicitly be downloaded. The control panel for example uses this method to provide a button which saves the wiki as a static HTML file.

A saver module has to export two functions. canSave(wiki) returning true if this module is capable of working and create(wiki) returning an instance of a saver object. This saver object has to provide an info property containing a name, a priority, an array of methods it supports and a method save(text,method,callback). This method is called from TW with the actual text which should be saved, the method which is used and a callback function to report errors: callback("Error while saving") or to notify that saving went well: callback(null, "Saving went well :)"). If the saver method successfully saved the file it has to return true, or false otherwise. Saves are triggered by messages from the UI. The syncer module uses the saver with the highest priority capable of the requested method to save the file.

The core plug-in contains a saver capable of saving the current state of the wiki to the local hard drive by using a special Firefox extension called Tiddlyfox. If this extension is not available, the savers canSave method would return false. A saver with a lower priority would then ask the user to save the current state as a new HTML file.

Syncadaptor

A module with module-type: syncadaptor provides functionality to get a list of tiddlers (this list is provided as SkinnyTiddlers, which are normal tiddlers without the text field) and to load, save and delete single tiddlers. A syncadaptor can also provide functions to login and logout so that syncadaptor modules can be used to synchronise tiddlers with a remote server.

The syncer module only uses one syncadaptor and honours a special system tiddler $:/config/SyncFilter containing a filter string. Tiddlers matching this filter string are saved to the server with a syncadapter. It uses the WebServer API to load modified tiddlers from the server, which returns only non-system tiddlers.

UI and Rendering Pipeline

The microkernel provides basic functionality to store tiddlers and manage modules and plugins. The Startup Process then loads the core plug-in including extensions to the store providing a Event Mechanism, Caching, Tags and a Filter Mechanism giving the ability to query for specific tiddlers.

Using some of this techniques the core plug-in also adds some functionalities to load and save single tiddlers or the whole wiki, described in Extended Persistence.

This next chapter will focus on the parts of the core plug-in that provide the UI of TiddlyWiki.

The rendering pipeline [https://tiddlywiki.com/talkytalky, 17.07.2014]

Parser

The first stage of WikiText processing is the parser. A Parser is provided by a module with module-type: parser and is responsible to transform block of text to a parse-tree. The parse-tree consists of nested nodes like

{type: "element", tag: <string>, attributes: {}, children: []} - an HTML element
{type: "text", text: <string>} - a text node
{type: "entity", entity: <string>} - an HTML entity like &copy; for a copyright symbol
{type: "raw", html: <string>} - raw HTML

The core plug-in provides a recursive descent WikiText parser which loads it's individual rules from individual modules. Thus a developer can provide additional rules by using module-type: wikirule. Each rule can produce a list of parse-tree nodes. A simple example for a wikirule producing a <hr> from --- can be found in horizrule.js

HTML tags can be embedded into WikiText because of the html rule. This rule matches HTML tag syntax and creates type: "element" nodes. But the html-rule has another special purpose. By parsing the HTML tag syntax it implicitly parses WikiText widgets. It the recognises them by the $ character at the beginning of the tag name and instead of producing "element" nodes it uses the tag name for the type:

{type: "list", tag: "$list", attributes: {}, children: []} - a list element

The Widgets part will reveal why this makes sense and how each node is transformed into a widget. Another special characteristic of the html-rule or the parse nodes in general is the attributes property. Attributes in the parse-tree are not stored as simple strings but they are nodes of its own to make indirect text references available as attributes as described in Widgets:

{type: "string", value: <string>} - literal string
{type: "indirect", textReference: <textReference>} - indirect through a text reference

Widgets

When the WikiText has been transformed into a parse-tree the next step is to transform this parse-tree into a widget-tree. We talked about widgets as parts of the WikiText markup but in fact each node of the parse-tree is transformed to a widget object. The core plug-in provides a basic widget object which gets the parse node it should represent and a DOM node. The widget then must create the DOM structure which represents the parse node and add it to the provided DOM node. A LinkWidget for example would create the DOM node for a <a>...</a> tag and put it in the provided DOM node. When a widget gets a parse node with child nodes it must create the corresponding child widgets.

All this functionality is basically provided with the base widget. But when creating the widget for a parse node it loads additional modules with module-type: widget. These modules can export multiple widgets which extend the base widget and can override it's methods like the rendering and refresh functions. As described in Parser each parse node contains a "type" property. This type directly determines which widget module is used.

{type: "text", text: <string>} - a text node

Would be transformed to a TextWidget. (Note that the TextWidget module exports a property "text".)

So in fact when talking about widgets in WikiText, they are not a feature added to WikiText but the ability to directly use a specific widget instead of the parser choosing a widget type.

In the beginning we talked about widgets and how they enable dynamic behaviour. But up until now we only described how widgets create DOM nodes from parse nodes. Widgets add dynamic behaviour in two ways.

Messages

Messages are events that are triggered by the user. They are generated by widgets for example when the user clicks on a ButtonWidget. Each message has a type property like "tm-delete-tiddler" and a parameter.

{type: "tm-delete-tiddler", param: "MyOldTiddler"}

When such a message is created by a widget it sends the message to it's parent widget which sends it to it's own parent widget and so on. On this way each widget can try to dispatch the message. This concept is realised in the base widget object. It provides a function dispatchEvent(message) which is called by the children to bubble the message up the widget tree. Another function addEventListener(type,listener) can be used to bind a function to a specific message type. If the listener returns false, the message is send to the parent widget.

The TiddlyWiki core plug-in handles a lot of messages in the NavigatorWidget.

Selective Update

With Messages a widget is able to put some kind of events into the TiddlyWiki application which is one part of the dynamic behaviour of widgets. The other part is selective updating. Widgets are often dependant on tiddler states. The ListWidget for example can be configured to list all tiddlers which are tagged with "important". Now, when such a tiddler is changed and it's "important" tag is removed, this change should reflect in the ListWidget. To allow widgets to react on such changes, each widget can provide a function refresh(changedTiddlers). The RootWidget is registered to the wiki store, using the Event Mechanism. When an change event occurs it starts to call the refresh function of its children with a list of the changed tiddlers. Each widget can then decide if it has to change or re-render its DOM representation and call the refresh function of its own children. This way every time a tiddler or the wiki store itself changes, each widget can instantly react on these changes or ignore them.

Another way of updating are text reference attributes (text references explained in Transclusion and TextReference):

{type: "indirect", textReference: <textReference>}

When a widget got a attribute which is a text reference and the refresh function is called, it can check if the text reference references a changed tiddler and update accordingly.

Nearly every state of the UI is stored in tiddlers. A search mechanism for example would use a EditTextWidget which shows an text input field for the search string and a ListWidget to show the results. The EditTextWidget is bound to a tiddler $:/temp/search. Meaning when editing the text in the input field the tiddlers content is changed to this text. On the other hand when the tiddler changes the RootWidget is notified. It then starts calling its childrens refresh methods. Eventually the refresh method of the EditTextWidget is called and it can re-render itself if necessary. This way TiddlyWiki can re-use an already existing data structure which is not only convenient because we don't need to introduce an additional structure but tiddlers managed in the wiki store are already a pretty powerful data structure, supporting an Event Mechanism (so we don't need an additional observer pattern), Caching, Metadata by additional tiddler fields and a way to obtain specific tiddlers.

Transclusion and TextReference

The previous parts about Widgets and the Parser explained how a block of WikiText is transformed into a DOM representation and how this presentation can react on changes to the wiki store. The previous chapters also describe that WikiText is saved in individual tiddlers, including the WikiText describing the UI components. This raises the question, how these multiple tiddlers are build up to a single UI. But before answering this question we need to introduce text references and transclusion.

TextReference

A text reference describes a special notation to indirectly refer to the contents of a specified tiddler field. The syntax of a text reference is:

<tiddlertitle>
<tiddlertitle>!!<fieldname>
!!<fieldname> - specifies a field of the current tiddlers

To obtain the actual text, the core plug-in adds a function to the wiki store getTextReference(textRef,defaultText,currTiddlerTitle). The "currentTiddlerTitle" is the title of a tiddler which is used when no tiddlerTitle is specified in the text reference. What the currentTiddler should be, depends on where the text reference is used. If it is for example used as a widget attribute, the current tiddler is the tiddler which contains this widget. Text references are used by widgets (attributes), filteroperators (parameter) and in transclusions. These elements use the getTextReference(textRef,defaultText,currTiddlerTitle) function.

Transclusion

Transclusion means including the contents of a different tiddler. This is realized with a transclude widget which parses the tiddler to be included and adds the resulting nodes as children of its own parse node. The trick with transclusion is, that it shows the content of a different tiddler but by default it does not change the current tiddler. This enables us to create tiddlers with text references and use them as a templates.

For example: Tiddler MyTask having the fields and the content of:

important: "very"
assoc.person: "Hans Dampf"

<$transclude tiddler="TaskHeaderTemplate" />

Hans needs some more Dampf.

And Tiddler TaskHeaderTemplate with a content of:

<$view field="assoc.person"/> has a <$view field="important"/> important task for us:

When showing tiddler MyTask it would result in:

Hans Dampf has a very important task for us:

Hans needs some more Dampf.

Transclusion and templates is one of the most important concepts in TiddlyWiki. It allows us to include other tiddlers using the metadata for our current tiddler. It also allows us to transclude a template tiddler with a third tiddler set as the currentTiddler with the TiddlerWidget:

<$tiddler tiddler="MyTiddler">
<$transclude tiddler="EditTemplate" />
</$tiddler>

This way we can create a different view on a tiddler which does not only show it's title and it's content but shows it's content and metadata in editable text fields and allows us to edit this tiddler. Also the template concept is used by the ListWidget to show the tiddlers through a specified template. Finally, when wanting to download the wiki as a new html file, this html file is created by binding a list of all tiddlers to a template and sending the resulting text to the syncer module described in Extended Persistence to save the current wiki.

RootWidget and Rendering Startup

The previous parts of this chapter showed how WikiText is transformed to DOM nodes which dynamically react to tiddler changes and a way to compose tiddlers from other tiddlers. This last part describes how the TiddlyWiki core plug-in starts up a UI build from tiddlers and WikiText.

After the microkernel has loaded it starts executing Startup Modules. The core plug-in contains two startup modules which are responsible to kick off the UI:

rootwidget.js is a startup module and creates an instance of the base widget. This widget is globally accessible $tw.rootWidget. The DOM node associated to this widget is the current browser window's DOM (document).

At first, the root widget has no children but provides some basic event handlers (Messages) like:

  • tm-notify: Displays the message given in param as a notification.
  • tm-save-wiki: Triggered by a save button, the user can click. This handler uses the syncer module described in Extended Persistence to save the current wiki.
  • tm-auto-save-wiki: Similar to tm-save-wiki but not triggered directly by the user but automatically triggered when a wiki page is edited and saved. A Saver implementation which starts a download of the updated wiki file would not support the auto-save method and would only be used when the tm-save-wiki message is used.
  • tm-download-file: This message also uses the syncer module described in Extended Persistence but explicitly demands to choose a saver with the download-method to start downloading a single tiddler.

After the root widget is loaded another startup module $:/core/modules/startup/render.js creates a transclude widget which contains the contents of $:/core/ui/PageTemplate which is now bound to the browsers DOM document. The render function of the transclude widget is initially executed and a listener is registered at the store which executes the refresh function of the transclude widget to trigger the Selective Update process.

Techniques for including other tiddlers and Templates are finally used in $:/core/ui/PageTemplate to build the TiddlyWiki UI only from tiddlers written in WikiText (with widgets implemented in javascript):

For example to implement the list of open wiki pages the $:/core/ui/PageTemplate contains a navigator widget which maintains a list of open tiddlers in a field of $:/StoryList and handles events like tm-navigate by adding a tiddler specified as parameter to the top of the list in $:/StoryList. The story tiddler transcluded in $:/core/ui/PageTemplate then uses a ListWidget to transclude all tiddlers in $:/StoryList through a special template $:/core/ui/ViewTemplate. A event of the type tm-close-tiddler would remove a specified tiddler from $:/StoryList. The Event Mechanism would trigger a changed event which triggers a call of the ListWidget's refresh function which would remove the tiddler from the list, closing the tiddler.