Building Websites with Mathematica, Part 2

Tue. Sep 19 2017
mathematica
b3m2a1

So in this website’s very first post I ran through how one can use pelican and Mathematica in conjunction to make a website. Now we’re gonna cut pelican out all together.

The basic process will look almost exactly the same, though:

  • Write content in Mathematica notebooks

  • Customize the website via themes and templates

  • Export to build directory

  • (Optional) Deploy to cloud

The main trick here, though, is in that innocuous “customize the website via themes and templates”, as it’s that “templates” that will really trip us up.

‌

XMLTemplates

Basic Overview

Mathematica’s had a templating system for a little while, now:

PacletFind["Templating"][[1]]["MathematicaVersion"]

(*Out:*)

"10.1+"

This system has a general templating language that runs through TemplateObject . The templating language is too clunky for most simple cases, though. Because of this the team that made the templating system added a collection of specialized templates, including StringTemplate , NotebookTemplate , FileTemplate , and most important for us, XMLTemplate .

It’s this XMLTemplate system that we’ll use to build our site. So the first order-of-business is to translate our old Jinja templates into XMLTemplate -ready form.

Here’s a quick example. First the Jinja template:

 
<title> {% block title %} {{ SITENAME }} {% endblock %} </title>
<link rel="canonical" href="{{ SITEURL }}/{{ output_file }}">

And now the XMLTemplate

 
<title><wolfram:slot id="Title"><wolfram:slot id = "SiteName"/></wolfram:slot></title>
<link rel="canonical" href="<wolfram:slot id='SiteURL'/>/<wolfram:slot id='OutputFile'/>">

In general there’s a pretty simple translation between the two, although the Wolfram version is, unsurprisingly, more verbose.

XMLTemplate Complications

Unfortunately there are cases when things get more complex where this breaks down. The core case is when you are extending templates or using passed variables as slot names—or whenever you need variables to cascade up or down, essentially.

As an example, in Jinja we might have this:

 
{% extends "index.html" %}

{% block title %}
# {{ tag }} Articles | {{ SITENAME }}
{% endblock %}

{% block page_header %}
# {{ tag }} Articles
{% endblock %}

Our naive XMLTemplate would look like this:

 
<wolfram:get path = "index.html">
   
<wolfram:slot id = "Title">
# <wolfram:slot id = "Tag"/> Articles | <wolfram:slot id = "SiteName"/>
</wolfram:slot>
         
<wolfram:slot id = "PageHeader">
# <wolfram:slot id = "Tag"/> Articles
</wolfram:slot>
    
</wolfram:get>

But we can run into issues here. The first might occur in passing "Title" to our template.

Say we apply this template like so:

temp =
  XMLTemplate@"
   <wolfram:comment>
   We're excluding the <wolfram:get>...</wolfram:get> for now
   </wolfram:comment>
   
   <wolfram:slot id='Title'>
   #<wolfram:slot id='Tag'/> Articles | <wolfram:slot id='SiteName'/>
   </wolfram:slot>
   
   <wolfram:slot id='PageHeader'>
   #<wolfram:slot id='Tag'/> Articles
   </wolfram:slot>
   ";
TemplateApply[
  temp, <|"Tag" -> "tag", "SiteName" -> "mycoolsite"|>] // StringTrim

 -----------Out-----------
#tag Articles | mycoolsite



#tag Articles

All good here. But what happens if we also pass a "Title" ?

TemplateApply[
  temp, <|"Tag" -> "tag", "SiteName" -> "mycoolsite", 
   "Title" -> "mytitle"|>] // StringTrim

 -----------Out-----------
mytitle


#tag Articles

The "Title" slot got wiped out. This shows how you can be burned if you’re expecting the "Title" to cascade up. The "Title" is a global replacement as are all passed variables. If you have a "Title" slot in a parent template, the "Title" will insert there. A somewhat better way to write this is like so:

temp =
  XMLTemplate@"
   <wolfram:comment>
   We're excluding the <wolfram:get>...</wolfram:get> for now
   </wolfram:comment>
   
   <wolfram:slot id='PassedTitle'>
   #<wolfram:slot id='Tag'/> Articles | <wolfram:slot id='SiteName'/> \
| <wolfram:slot id='Title'/>
   </wolfram:slot>
   
   <wolfram:slot id='PageHeader'>
   #<wolfram:slot id='Tag'/> Articles
   </wolfram:slot>
   ";
TemplateApply[
  temp, <|"Tag" -> "tag", "SiteName" -> "mycoolsite", 
   "Title" -> "mytitle"|>] // StringTrim

 -----------Out-----------
#tag Articles | mycoolsite | mytitle



#tag Articles

This way the "Title" will cascade up as "PassedTitle" .

This can also bite you when you are using "<wolfram:get>" blocks the other way. Say you have a "var" parameter that you’re generating and which you’d like to pass into a "<wolfram:get>" block, like so:

varExists =
  "<wolfram:which>
   <wolfram:if test='KeyMemberQ[#,#var]'>
    <wolfram:slot id='if'></wolfram:slot>
   </wolfram:if>
   <wolfram:else>
    <wolfram:slot id='else' />
   </wolfram:else>
  </wolfram:which>";

This seems pretty good. We have a test which tests if a parameter "var" is in our template variable list. Well, when we drop this in a file and test it like so:

dir = FileNameJoin@{$TemporaryDirectory, "xml_templates"};
CreateDirectory[dir];

Export[FileNameJoin@{dir, "varExists"},
   varExists,
   "Text"
   ];
 
temp = "
  <title>
   <wolfram:with longTitle='True'>
    <wolfram:get path='varExists'>
     <wolfram:slot id='var'>longTitle</wolfram:slot>
     <wolfram:slot id='if'>A very long title</wolfram:slot>
     <wolfram:slot id='else'>A more compact title</wolfram:slot>
    </wolfram:get>
   </wolfram:with>
  </title>";


Block[{
    $TemplatePath =
      Append[$TemplatePath, dir]
    },
   TemplateApply[XMLTemplate@temp, <||>]
   ]

 -----------Out-----------

<title>
 
  
  A more compact title
 
 
</title>

We see here that the Slot operator doesn’t work as one would expect. It is not an Association of all current stack arguments, but rather just those passed. There’s some syntactic sugar to support "#var" type names from the stack, but meta-programming of this nature is hard. Similarly there can be complex issues with variable passing and values which all derive from this core issue.

XMLTemplates Routine Libraries

An effective way to circumvent these issues is to fall back to the core Mathematica code. In particular there are parameters Templating`$TemplateArgumentStack and Templating`$TemplateArguments which provide access to stack values. We can define a new varExists routine like this:

With[{
     tempArgs =
        $$templateLib["getTemplateArguments"][#]
     },
    ! MatchQ[
        $$templateLib["templateArgumentLookup"][tempArgs, "var"],
        _Missing | False | None | _String?(StringMatchQ[Whitespace])
        ]
    ] &

Which takes advantage of two other routines, getTemplateArguments :

(Join @@
     Flatten@{
         #,
         Replace[Templating`$TemplateArgumentStack, {
               {___, a_} :> a,
               _ -> <||>
             }]
         }) &

And templateArgumentLookup :

Replace[
    #@
       Replace[#[#2],
          t_TemplateObject :>
             TemplateApply[t, #]
          ],
    TemplateObject[{
         Templating`Evaluator`PackagePrivate`apply[_,
            a_
            ]
         }] :>
         Block[
            {
               Templating`PackageScope`$TemplateEvaluate = True
               },
            a
            ]
    ] &

Then we place these in .m files and define a loading function like this:

With[{Templating`lib`Private`libdir = DirectoryName[$InputFileName]},
   Templating`lib`$$templateLib[f_] :=
      
  Templating`lib`$$templateLib[f] =
         (
            Begin["Templating`lib`Private`"];
            (End[]; #) &@
               
     Import@FileNameJoin[{Templating`lib`Private`libdir, f <> ".m"}]
            )
   ]

Then we can use these routines in XMLTemplate expressions, like this:

 
<!--varDefinedBlock.html-->

<wolfram:which>
  <wolfram:if
    test='$$templateLib["varDefined"][#]'>
    <wolfram:slot id='if' />
  </wolfram:if>
  <wolfram:else>
    <wolfram:slot id='else' />
  </wolfram:else>
</wolfram:which>

And then in our generator we prep and clear $$templateLib .

By doing this we can make the XMLTemplate framework a lot more flexible and extensible and make our code a lot less verbose.

‌

Building Content

Basic Idea

So with our XMLTemplate considerations out of the way we move on to the real work of content generation. The basic process idea is that we’ll have an ever growing stack of parameters which encapsulates all of our site data. We’ll populate it from the .md and .html files in our content directory in the following order:

  • Meta-information for all files

  • Helper-info like URL, Summary, etc. for posts and pages

  • Raw HTML for posts and pages

  • Aggregation pages, like the index.html

We’ll have this stack be an Association for convenient lookup, which allows us to provide helper functions like this:

"ContentData" ->
  Function[
    Fold[
      Lookup[#, #2, <||>] &,
      $ContentStack,
      {#, "Attributes"}
      ]
    ]

Or aggregations like this:

"Pages" :>
  Select[
    Values@
      $ContentStack[[All, "Attributes"]],
    MemberQ[#["Templates"], "page.html"] &
    ]

Then by exposing this $ContentStack to the templates, we can write even more powerful templates with even less work. For instance, here’s a snippet from my standard index.html template:

 
<wolfram:sequence
    values="Replace[#IndexListing,Except[_List]:>#Articles]"
    slot="IndexItem" delimiters="<hr>"
    >

    <div class="article-bubble bubble centered">
      <article class="teaser">
          <h4 class="article-title">
            <a href='<wolfram:slot id="SiteURL"/>/<wolfram:expr>#IndexItem["URL"]</wolfram:expr>'>
              <wolfram:expr>#IndexItem["Title"]</wolfram:expr>
            </a>
          </h4>
          <div class="content article-summary">
            <wolfram:expr>#IndexItem["Summary"]</wolfram:expr>
          </div>
      </article>
    </div>

We pass the entire $ContentStack to the TemplateApply call, so the template can take advantage of the "Articles" parameter it provides.

Markdown Parsing

The hard part of all this is that we have to have an effective way to go from Markdown to HTML. Since we want to work with pure Mathematica code we need a Markdown to HTML converter function—something which I was unable to find before starting work on the site builder.

I won’t go into all the details of how to write one of these now, but know that it’s a pretty tedious endeavor (and my version is not even entirely complete). Let it suffice to know that one now exists.

‌

Copying Themes

The second-to-last (or final if you aren’t deploying in the cloud) step is to copy over the theme. We want to be efficient about how we do this—it’s wasteful to copy it over every time. To make this more efficient we use a system where we check the FileDate of the previously copied version and the version in the core theme directory. If the previously copied one is older we copy over the new one.

‌

Deployment

Finally, with all this in place, we can move to actually deploying our page. We will do it in exactly the same way as described in the original post.

‌

Extensions

The power of having something like this in pure Mathematica code is that it significantly lowers the barriers to developing websites. Making a new site is as simple as creating a new collection of XMLTemplate s (or more likely adapting an old one), maybe adding some new $$templateLib functions.

This is very powerful, and I’ve already set up two systems that use this. My first is an extension of the built-in PacletManager , which creates a static site that lists what is on a paclet server. The second is a general documentation host for providing a nicer interface on the documentation I generate and web-deploy.

The first one lives here and the impetus behind its creation is described in this post

post-11-7041362475981202038

And the latter is here and the impetus behind its creation is described in this post

post-11-7379889623084279071

Even better, both of these can also live as their own packages without any dependencies on anything outside of Mathematica. Having a native site builder just makes things vastly quicker to set up and so much more convenient.

And with that, I think, we’re done.