Jolly Classroom Content Pipeline

Mission

Our goal is to build content that delivers user value. Our users are teachers, and pupils, who need effective and engaging lesson content.

We have a big mission, with lessons that will teach children literacy over 7 school years. One of the fundamental problems is managing all that information.

Postgres

Postgres seemed obvious, coupled with an HTML UI. What we rapidly found, was that it introduced friction to the iteration. As we iterated on the designs and activities, schema migrations

After realising Postgres was unsuitable, we set out to find a solution that was adaptable and easy to integrate. We chose AirTable.

AirTable

Airtable worked great, we could rapidly iterate on our schema, all the images, audio and text could be easily added.

Here’s an example of how the data looks in AirTable and how it looks on our user interface:

This user interface has six words, each with a missing sound. The pupils have to drag the missing sound into each word

To build this interface by querying directly, on this page alone, we would have to query Airtable for six words, their images and sound files.

Direct queries would also mean:

• Our UI implementation is tied to the AirTable structure.
• Our perfomance is dependent on Airtable performace.
• Our application cannot function offline.

Content Packager

So, we need an intermediate layer, that solves these problems. We call it content packager.

First, let’s break content packager down into the general steps it does:

• Fetching data from Airtable.
• Restructuring the data
• Storing the data

We have already expanded on Airtable, so let’s jump into the next topic.

GraphQL and Apollo

We never want our users to experience an error due to a change in Airtable. Let’s say a rename of a column or a change in data type. We also want to find these errors as quickly as possible.

To fulfil this, we chose GraphQL, combined with Apollo to fetch our data.

We write our GraphQL queries based off the Airtable GraphQL schema. If a field changes, an error is picked up at compile time, when Apollo runs.

Our Kotlin classes are generated from the query, so, our code is statically linked to the structure of the Airtable tables.

Transformation

Our UI needs data, when writing beautiful UI, you don’t want any additional complexity. Data needs to be provided in exactly the form you need.

Rather than simply caching the data, we need to design it for consumption, and to completely hide any details of our CMS.

Protobuf

Our UI is written in Dart, our backend in Kotlin, we need a shared format.

Enter Protobuf.

In Protobuf we declare a single shared format:

message Word {
  string text = 1;
  string audioPath = 2;
  string imagePath = 3;
  repeated Sound sounds = 4;
}

Model code is generated on the Kotlin, and Dart side, giving us:

• A single declaration of data structures
• Hyper Efficient data storage file sizes, 75% improvement on JSON
• Generated serialisation and deserialization code.

Our UI simply needs to deserialise the content definition from the protobuf files, and get to work on the content.

Result

The end result, is a system that is:

• Typesafe, from the database through to the UI. This gives us confidence it will work, whilst being adaptable.
• Robust, when errors occur, they break before deployment
• Offline, our content files are packed efficiently onto a users device, allowing it to work offline.

We are exceptionally happy with our content pipeline, it’s a key example of how our dedication to technical excellence delivers value to our customers, while enabling us to work rapidly and effectively.