Release 5.2.0

3rd October 2021 at 3:15pm

See GitHub for detailed change history of this release

Congratulations to Frank. B for his winning design for the banner for this release (here are the other entries).

Highlights

Unrestricted Fieldnames and the New JSON Store Area

The major change in this release is that the format used to store tiddlers in TiddlyWiki's HTML file has changed from an HTML DIV-based format to a new JSON-based format. The consequences of this change are far-reaching. They are thus the primary rationale for bumping the version number from v5.1.x to v5.2.0:

Firstly, the characters that can be used in field names are now unrestricted (just like tiddler titles), making it possible to use fieldnames like My^Field or ☃️. This has become possible because every other part of TiddlyWiki was already capable of dealing with unrestricted field names apart from saving as an HTML file.

Secondly, external tools that read, write or otherwise process TiddlyWiki HTML files will need modification to understand the new store area format. The details are documented in the /dev wiki.

For more details, see ticket #5708 on GitHub.

Dragging and Dropping Images into the Editor

It is now possible to drag and drop an image file directly into the tiddler text editor. A popup allows the incoming image tiddler to be renamed, and then [img[title]] is automatically inserted into the tiddler text at the insertion point.

added (and again here) ability to drag and drop images directly in the editor

Simplified Access to Tiddler Title with ActionCreateTiddlerWidget

It is now possible to embed actions within the body of the ActionCreateTiddlerWidget, and those actions can directly access the title of the newly created tidddler via the variable createTiddler-title, and the title of the corresponding draft in the variable createTiddler-draftTitle. The old $savetitle and $savedrafttitle attributes are now deprecated.

<$button>
  <$action-createtiddler $basetitle="Homemade Button" tags="$:/tags/PageControls" text={{$:/state/new-button-caption}}>
    <$action-navigate $to=<<createTiddler-title>>/>
  </$action-createtiddler>
  Create a new non-functional page control button and open the tiddler
</$button>

For more details:

extended ActionCreateTiddlerWidget to make the new title available as a variable

More Flexible Parsing of Macros

Two long-standing limitations of TiddlyWiki's macro syntax have been resolved.

First, it is now possible to nest macro calls within the parameters to other macros. For example:

<<mymacro arg:"""nested <<macro>> call""">>

Second, macro parameters can now be used within filter expressions. For example:

[<qualify "$:/state/mydata">addsuffix[-]addsuffix<now "YYYY MM DDD hh:mm:ss">]

improved macro calls to use the same parser as that used for widget attributes
added support for macro parameters within filters (see Filter Parameter)

Optimised Refreshing of Transclusions

This simple-sounding change has several important consequences:

The first consequence is that it makes the TranscludeWidget much more efficient, and improves the performance of the refresh cycle.

The second consequence resolves a long-standing difficulty for many users: it is now possible to use an EditWidget to edit another field of the same tiddler. Previously, the edit widget would not work properly in such cases because it would be refreshed on every keypress, leading to the current editing caret/selection position being lost.

As sometimes happens, the third consequence is that constructions that relied on the old, buggy behaviour may no longer work as expected. A common example of such a construction is an invocation of a JavaScript macro. The problem stems from the fact that JavaScript macros are designed to be as simple as possible from the developers perspective, being pure functions that have no side effects, and that always return the same value for the same parameters. In particular, JavaScript macros cannot participate in TiddlyWiki's refresh mechansim, so they cannot force themselves to be updated when underlying data changes (for situations where such processing is needed, JavaScript widgets should be used instead of JavaScript macros).

However, developers have sometimes worked around this limitation by using the TranscludeWidget to force refreshing of a macro invocation when a particular tiddler changes. The newer, more selective behaviour means that sometimes such widgets will avoid a refresh that was previously relied upon to update the JavaScript macro.

For end users, if an upgrade to v5.2.0 causes problems then consult the discussion Forums to see if other users have encountered the same problem, and make a new posting if needed.

improved the TranscludeWidget to avoid triggering a refresh unless the transcluded field changes (previously it was also triggered for changes to other fields)

Other Notable Improvements

extended (and here) the ListWidget with an optional counter attribute specifying a variable to contain the numeric index of each list item. This is useful for creating a list of numbered items, and for detecting the first and last entries in a list

extended action widget execution with an optional new mode that ensures that all widgets are refreshed before invocation – see ActionWidget Execution Modes. This resolves a long-standing inconsistency in the way that ActionWidgets are executed, making them behave much more like a conventional programming language

improved the HTML comment syntax to enable it to be used before and in between pragmas such as macro definitions. This makes it much easier to provide documentation within complex wikitext tiddlers

fixed size of dropzone when story river is empty. For a long time, TiddlyWiki confusingly wouldn't recognise a drag and drop operation if the story river was empty

Performance Improvements

improved (and again here) the efficiency of the linked list implementation used in filter processing
improved the all Operator and here the links Operator, and here the backlinks Operator to use the new linked list implementation
improved ETag-based browser-side caching for all resources and GZip or Deflate compression for all responses to GET requests
improved plugin internal formatting to remove unneeded whitespace, saving about 6% on the size of the $:/core plugin
improved performance of sort Operator through optimising common cases

Usability Improvements

fixed ability to import previously blocked system tiddlers
improved warning message when cancelling a pending import
updated the default font weights in the "Snow White" theme to be thicker and more legible
fixed modal mechanism to prevent page from scrolling while modals are displayed
fixed fill colour for "remove tag" button
fixed page title so that the separating em-dash is only used if the site subtitle is present
fixed broken aria-label in $:/PaletteManager
updated SVG optimiser script
added plus/minus SVG icons: and
added link to the existing tiddler when the warning "Target tiddler already exists" is displayed in the edit template
added class to make tag pills in the edit template look like those in the ViewTemplate
fixed overflow of long field values that do not contain whitespace
improved GitHub, Gitea and GitLab savers to wikify the commit message, enabling macros to be used in the message
improved the text editor stamp toolbar button dropdown to allow sorting by drag and drop

Widget Improvements

added (and here) new MessageCatcherWidget to catch arbitrary messages and execute action strings
modified FieldManglerWidget to ensure it doesn't propogate events that it traps
extended DropzoneWidget to optionally invoke actions after the tm-import-tiddlers message has been sent, and to specify an optional contentTypesFilter which determines which content types are accepted by the dropzone.
extended DropzoneWidget to also use the specified deserializer for strings either dropped or pasted on to the dropzone
fixed SelectWidget refreshing
fixed ActionListopsWidget bug by avoiding concatenating fragments of filter expressions
extended ActionPopupWidget to create floating popups that must be manually cleared
extended (and here) KeyboardWidget to provide additional information variables
extended DraggableWidget with option to hide the drag image

Filter improvements

added new :map filter run prefix
added new :sort filter run prefix
added average Operator, median Operator, variance Operator and standard-deviation Operator for calculating averages of a list of numbers
added deserializers filter Operator for obtaining a list of the available deserializers
added format:titlelist operator for selectively adding double square brackets around titles
added charcode Operator for easily accessing special characters like tab and linefeed
added support for suffixes to filter run prefixes
extended search-replace Operator with m flag for multiline mode
extended range Operator to allow parameters to be specified as separate operands
added moduleproperty Operator and extended modules Operator to provide access to internal JavaScript modules

Hackability Improvements

extended unusedtitle Macro to add separator and template parameters
added color-scheme field to all themes to differentiate between light and dark themes
added support for image/vnd.microsoft.icon content type
added support for throttling refreshes for changed tiddlers with the title prefix $:/temp/volatile/
added a hidden setting to control sandboxing of tiddlers of type text/html
added text operation insert-text
added data-tag-title attribute to tag pills
added support for dynamic toolbar buttons
added new SystemTag: $:/tags/ControlPanel/Plugins for customising the tabs shown under the "Plugins" tab of control panel
added support for DateFormat tokens for weekday number and day of year
added further transliteration pairs for the slugify Operator

Client-server Improvements

fixed crash running in client server configuration when 'ETag' header is missing
fixed blank favicon when using lazily loaded images
fixed web server issue with custom path prefix and basic authentication
fixed crash on Node.js with ActionSetFieldWidget when type field is given a value upon new tiddler creation
fixed issue with saving tiddler files with titles that already end in the required extension
fixed several consistency issues with the filesystem plugin
fixed issue with encoding of $:/config/OriginalTiddlerPaths outside the wiki folder
updated the TiddlySpot Saver settings form
added 401 and 403 error messages for PUT saver
extended the external JavaScript template to be easier to use and work more smoothly when offline
fixed incorrect usage of X-Requested-With header for simple HTTP requests
fixed security issue allowing access to files outside the ./files/ folder

Node.js Improvements

extended RenderCommand to allow multiple variables to be passed

Plugin Improvements

Markdown Plugin

added support for using the KaTeX Plugin in Markdown
fixed extraneous paragraph tags in lists

XLSX Utilities Plugin

fixed crash with the XLSX Utils plugin

KaTeX Plugin

updated to KaTeX v0.13.18
added custom macro editor
added automatic numbering reset for each displayed tiddler

Freelinks Plugin

extended the Freelinks plugin with a filter to determine which tiddlers can be the targets of freelinks

Menubar Plugin

fixed Menu plugin to support optional dropdown-position field

BibTeX Plugin

fixed BibTeX Plugin to report errors more sensibly
modified BibTex Plugin to force fieldnames to be lowercase

Developer Experience Improvements

improved the widget subclassing mechanism to work with widgets that add event listeners in their constructor
improved the Jasmine test suite output
extended wikitext parser with a subclassing mechanism
added added support for utils-browser modules
added th-before-importing hook to allow plugins to inspect or modify the importTiddler object before any tiddlers are imported
added th-closing-tiddler hook when tiddlers are closed

Translation improvements

added Polish translation
improved Catalan translation
improved Chinese translations
improved French translation
improved German translations
improved Spanish translation

Other Bug Fixes

fixed crash with malformed input to decodeURIComponent()
fixed issue with lookup Operator returning "undefined" under some circumstances
fixed crash with unterminated wikitext comments
fixed tiddler info area content bleeding on close animation
fixed inline/block widget parsing glitch
fixed runaway regexp when parsing filters
fixed right margin of tag pill when used outside of the tags wrapper
extended upload saver to optionally work without a username or password
fixed RadioWidget to refresh selectively, and to use the checked attribute correctly
fixed "invert" option of wiki.search() method
fixed ES5 compatibility issue
fixed exporting of tiddlers that begin and end with double quotes
improved accessibility of button widget when controlling a popup
fixed EditTextWidget to use default text for missing fields
fixed css-escape-polyfill to work under Node.js
fixed (and here) crash when sorting by non-string fields
fixed (and here) some bugs in the EventCatcherWidget, introduced new stopPropagation attribute and new syntax for specifying actions
fixed CurrentTiddler variable consistency in subfilters and prefixes
fixed crash when accessing variables in filters that don't have a widget context
fixed unnecessary triggering of reload warning when JavaScript tiddlers are not subsequently imported
fixed minor issue with import pragma
fixed leading and trailing whitespace in themes
fixed configuration list of HTML5 block elements
fixed shape and color for disabled button to work with tc-btn-invisible class
fixed inconsistent spacing of view toolbar items
fixed crash when assigning new value to file input controls
fixed text editor stamp button to work with prefixes and suffixes in shadow tiddlers
fixed Internet Explorer crash when pasting text using an HTA
fixed bug with [all[shadows]] sometimes returning overridden shadow tiddlers
fixed refresh bug with radio widget where the tc-radio-selected class is not correctly updated
fixed crash with reading invalid JSON files

Acknowledgements

@Jermolene would like to thank the contributors to this release who have generously given their time to help improve TiddlyWiki: