Custom Music Notation Renderer

Last year I spent a month working on a notation rendering system for a previous iteration of the game. While it might not make it into Dragon Drummer, I thought it could make an interesting dev post 🎵

Overview

For the previous game I needed a system that transformed MusicXML files (a standardized notation format exported by programs like Finale and Sibelius) into animated objects I could control in-game. I was thinking of directly exporting scores from Dorico as images, but I was aiming for a stylized low-poly look and needed the ability to animate each note head. Furthermore the system has to be performant in the context of a real time video game, so I decided to make an in-house notation rendering system. The high level architecture looks something like:

1. Parse MusicXML
2. Create symbolic layout and spacing
3. Render

MusicXML

MusicXML is an XML format exported by all major notation softwares. It contains not only information about notes and rhythms, but also layout things like stem direction. MusicXML files can be hefty so I convert them offline to a custom notation language inspired by Lilypond.

Key:C
Time:4/4
| r1 
| r 
Clef:G
| r8 ( e f g ) f4 e 
| d ( e8 c# ) ( c# b ) _ e4 
| r8 ( ^ e f g ) f4 e 
| d ( e8 b - ) ( b b b c ) 
| r ( e f g ) f4 e 
| d ( e8 c# ) c#4 ( b8 _ e ) 
| r ( ^ e f g ) ( f e d c# - ) 
| c#2. r4 

This notation is then converted on the Javascript client to a symbolic representation of notes, rests, accidentals, barlines, time signatures, etc.

Layout

The art of music engraving is really cool and I highly recommend this essay on computational music engraving if you want a deep dive. The layout system I implemented is bare bones in comparison and can be divided into three stages:

1. Create bounding boxes and anchors
2. Create spacing dependencies
3. Solve dependencies

Bounding Boxes and Anchors

First we represent each element of the music notation as a bounding box. Elements are things like

• noteheads
• stems
• accidentals
• ties
• barlines
• staff lines

The goal of our algorithm is to position and size these elements correctly. For each bounding box we place anchors, landmark points that we can use to specify relationships between boxes.

Spacing dependencies

Once we’ve got our bounding boxes and anchors, we create dependencies that relate anchors together. We can create simple dependencies like “anchor A must have the same height as anchor B”, or more complex dependencies that involve multiple anchors. I represent dependencies as classes that take one child anchor, one or more parent anchors, and implement a method applyDependency which applies the spacing constraint. This snippet shows the X_Offset dependency that defines an x offset between two anchors.

export class X_Offset implements IDependency {
    delta: number;
    child: Anchor;
    parent: Anchor;

    constructor(child: Anchor, parent: Anchor, delta: number) {
        this.child = child;
        this.parent = parent;
        this.delta = delta;
    }

    apply_dependency() {
        this.child.x = this.parent.x + this.delta;
    }
}

Though most spacing dependencies are fairly straightforward to setup, spacing notes horizontally can be a bit tricky. We could make the distance between notes proportional to the note duration, i.e. a half note would be visually spaced twice as long as an quarter note. This looks pretty bad in practice! Full blown layout algorithms found in professional engraving software do some fancy calculations to create the most visually pleasing spacing possible, but to keep things simple and fast I just have the spacing grow logarithmically with duration.

One gotcha when constructing dependencies: no cycles allowed! This can be a little tricky from a developer perspective since we can’t make the API enforce this. Probably should have written a runtime check for cycles but who’s got time for that 😁

Solving dependencies

Once we’ve constructed the proper dependencies between all our musical elements, we end up with our favorite data structure: a directed acyclic graph! We run a topological sort to make sure the dependencies are in order and then execute our applyDependency functions one by one. All of us this is in O(n) time, nice!

Rendering

Now that all the bounding boxes for our musical elements are positioned correctly, we just have to replace each element with the corresponding glyph. For a low-poly look that could be as simple as rendering a rectangle. Since each musical element is its own individual entity, we can control how it’s rendered to our hearts content. I ended up doing some (probably premature) optimizations to reduce draw calls via batching but I’ll save that for another post.

So there’s my high level overview of the notation rendering system I spent countless hours on that might never see the light of day! Or maybe I’ll try and sneak some notation in for our Dragon Drummers somewhere…

Credits

Ruler and camera icon in diagram made by Freepik from www.flaticon.com