CL Complete LMS

Every term, around assignment-marking week, we get the same shape of email. It usually opens with something like: "The teachers can mark some PDFs but not others, and a few are coming up as squares and boxes — is the server broken?"

The server is rarely broken. PDF annotation in Moodle™, and in most LMSes that bolt on a similar feature, is one of those quiet little subsystems that everything depends on for two weeks of the year and nobody thinks about for the other fifty. When it goes wrong, it goes wrong in three distinct ways, in roughly this order:

  1. The annotation tool refuses to open a submission at all, or shows a blank page.
  2. Letters in the rendered PDF turn into squares, boxes, or random Cyrillic for no reason anyone can see.
  3. The moodledata partition fills up over a couple of weeks and nobody can work out why.

These are all the same problem, dressed in different clothes. Almost all of it comes back to the toolchain Moodle™ uses behind the scenes to flatten and re-render the submitted PDFs so they can be drawn on. So let's walk through it the way we walk through it on a real client call.

What actually happens when a teacher clicks "annotate"

When a student uploads a PDF for an assignment and a teacher opens it in the marker, Moodle™ doesn't show them the original file. It converts each page of the PDF into a flat image, lays that image into the annotator canvas, and stores any pen strokes, highlights and comments as a separate overlay. When the teacher is done, Moodle™ stitches the overlay back onto the flattened pages and produces a new annotated PDF for the student.

That conversion — PDF in, page-images out, PDF back in — is where everything interesting happens. Out of the box, Moodle™ uses Ghostscript for it, via the pathtogs setting in Site administration → Server → System paths. Some sites have additionally installed Poppler's pdftoppm alongside it, either by hand or via a plugin, because Poppler tends to do a better job on certain classes of file.

If you've never looked, go and check that setting now. We've seen institutions where pathtogs is empty, points at a binary that no longer exists after a server migration, or points at a Ghostscript so old its security advisories are no longer maintained. Any of those will produce one of the three failure modes above.

Failure mode 1: blank pages and the silent fail

The dullest of the three. The PDF reaches Moodle™, Moodle™ hands it to Ghostscript, Ghostscript falls over, and the teacher sees either a blank annotation canvas or a polite "this submission couldn't be converted" message in the gradebook.

Three things to look at, in order:

Almost all of these are environmental, not Moodle™'s fault. The fix is usually a five-minute investigation followed by a one-line config change.

Failure mode 2: boxes, squares and the wrong language

This is the one that ruins teachers' Saturdays. The PDF opens, the annotator works, but the text on the page has become unreadable — usually as □ glyphs (the famous "tofu"), sometimes as the wrong characters entirely, sometimes as a single repeated letter where there should be a paragraph.

This is almost always a font problem, and almost always Ghostscript-related.

A PDF doesn't have to embed the fonts it uses. The spec is reasonable about this: if the document uses one of the 14 "standard" fonts (Helvetica, Times, Courier and a few others), the renderer is expected to have something equivalent. If the PDF uses anything else — and almost every PDF generated by Word, Pages, LibreOffice, or an institutional template these days does — the fonts have to be embedded inside the file. They usually are, but they're typically subsetted: only the glyphs actually used in the document are included, often under a renamed font name like AAAAAA+CalibriBold.

Ghostscript reads that subset and renders it. Usually. The pathological cases are:

On Debian-family systems, the immediate-fix toolkit is:

apt install fonts-noto-core fonts-noto-cjk fonts-noto-cjk-extra \
            fonts-dejavu fonts-liberation ttf-mscorefonts-installer

That last one needs you to accept the EULA on install. ttf-mscorefonts-installer is the one that quietly fixes 90% of "boxes where Calibri should be" cases, because so many institutional PDFs were originally Word documents.

After installing fonts, rebuild the fontconfig cache (fc-cache -fv) and — this catches a lot of people — restart the web server so PHP-FPM picks up the new font list. Ghostscript reads /usr/share/fonts at process start, not on demand.

If the fonts are there but the boxes persist, it's worth trying Poppler. That's the next section.

Ghostscript vs Poppler: when to switch, and what changes

Ghostscript is the venerable workhorse. It's a full PostScript interpreter that happens to also read PDF. It can do almost anything you can throw at it, including PDFs from 2003 that some institutions are still circulating. Its weaknesses are weight (it's a big process, slow to start, memory-hungry on large documents), occasional security CVEs that need patching the day they drop, and the font issue described above.

Poppler is a much narrower tool. The binary we care about is pdftoppm, which converts PDF pages to PPM/PNG/JPEG. It does that one thing, fast, and tends to do it well. Its strengths:

It loses to Ghostscript on:

For institutions where the assignment workflow is "students upload Word-or-Pages-exported PDFs, teachers annotate, PDF goes back" — which is almost everyone — Poppler wins. The font cases work better, the annotation queue moves faster, and the disk usage is lower because conversion is cleaner.

Moodle™ accepts a Poppler-based conversion path either through a maintained third-party plugin or — the cleaner route — by routing assignfeedback_editpdf through the Document converter service. We're happy to walk you through the switch (a few hours of work plus a careful regression test). The rough cost-benefit, in our experience: "Ghostscript-grade compatibility is rarely needed; Poppler-grade speed and font handling almost always is."

Failure mode 3: the moodledata partition that won't stop growing

Now the fun one.

A client called us in October because their moodledata partition had gone from 380 GB to 420 GB in two weeks, with no new courses, no new students, and no obvious uploads. They'd already doubled the volume once that year and weren't keen to do it again.

The growth was almost entirely in moodledata/temp/assignfeedback_editpdf/.

When Moodle™ converts a PDF for annotation, it produces:

Most of those are supposed to be temporary. Moodle™'s scheduled tasks tidy them up. The trouble starts when:

The fix is rarely "buy more storage". The right shape is:

  1. Confirm cron is running and the editpdf cleanup tasks have run recently (Site admin → Reports → Scheduled tasks).
  2. Find the orphans. A safe pass is anything in temp/assignfeedback_editpdf/ older than the last successful run of the cleanup task and not referenced in the mdl_assignfeedback_editpdf_* tables. We have a small script for this we hand to clients on engagements.
  3. Drop the conversion DPI back to 100 unless there's a genuine accessibility need higher. Most teachers can't tell the difference and the disk certainly can.
  4. Consider switching to Poppler, which produces cleaner intermediate output and tends to leave less mess.

In the case above we recovered 38 GB on the first night and put the institution on monitoring for the rest of the term. Disk pressure didn't reappear.

A note on Moodle vs other LMSes

We've described all of this in Moodle™ terms because that's where most of our work happens, but the same problems show up in Open edX (whose inline PDF features use their own converter chain), Canvas LMS (which uses server-side renderers for inline grading), and even self-hosted Blackboard installations that use a similar Ghostscript-based pipeline behind the inline grader. The directory names change. The pattern is the same: a PDF toolchain that sits behind a "graders' favourite feature", that nobody owns, that quietly fills disks and produces tofu when fonts are missing.

Seeing any of the three failure modes — annotations that won't open, boxes instead of letters, or a temp/ that won't stop growing? Talk to an engineer — we'll dig into the logs with you, free first hour. We do this on Moodle™ every week, and on most other LMSes too. See our emergency recovery and upgrades & maintenance services for the longer engagements.

Roughly half of the new clients who come to us already have “backups”. About a third of those have backups that work. The other two thirds find out the hard way — usually during a real incident, when the file they thought was a backup turns out to be something else entirely.

This is not a story about lazy admins or bad providers. It’s a story about a class of failure that’s almost invisible by design: backups that appear to be running, appear to be complete, and appear to be safely stored — until you try to restore one.

Here’s how to find out where you actually stand, in about 45 minutes.

The simple test that everyone skips

Set a calendar reminder for one hour next week. In that hour:

  1. Pick your most recent backup.
  2. Spin up a throwaway VM somewhere — a $5/month VPS is plenty.
  3. Restore the backup onto it.
  4. Log in as admin.
  5. Click into a recent course.
  6. Open a recent quiz attempt or assignment.

If all six steps work, your backup is real. If any of them fail, you’ve just learned something important — for free — before it matters.

That’s it. That’s the test.

Most teams have never done it.

The five ways we see backups silently fail

We’ve now done this exercise with dozens of new clients. The same five patterns show up over and over.

1. The database dumps, but moodledata is missing

Probably the single most common failure mode. The backup job carefully exports the database — every quiz attempt, every grade, every forum post — and then doesn’t back up the moodledata directory.

Result: you can technically restore the database, and Moodle™ will appear to load. But every uploaded file, every drag-and-drop assignment submission, every profile picture, every file in the question bank, every file resource in every course — all of those live in moodledata, not the database. A “DB-only” backup is roughly half a backup.

How to spot it: look at the size of your backup. If your moodledata directory is 20 GB and your backup file is 200 MB, the backup isn’t backing up your files.

2. The backup is on the same machine

We saw a heartbreaking version of this last year: a college with a meticulously-configured nightly backup job, writing tarballs to /var/backups on the same VM that ran Moodle™. The VM was deleted by the hosting provider after a billing dispute. Both the live site and 18 months of backups went with it.

A backup that lives on the same disk, the same VM, or even the same cloud account as your live site is not a backup. It’s a copy. The difference matters when the underlying thing fails.

How to spot it: look at where your backups are stored. If the answer is anything that contains the words “the server” or “same hosting account”, that’s the problem.

3. The backup ran successfully — for the wrong site

This sounds absurd, but we’ve seen it three times. A backup job was configured years ago for an older Moodle™ install, the install was migrated to a new location, but the backup job kept running against the old one. The “successful” green-tick emails kept coming. The actual current site had no backups at all.

How to spot it: check the timestamp and size of the last backup file. Does the size match the current site? Did it grow at the rate your site is growing? If your site is 40 GB and your backups are 4 GB and have been for two years, something’s wrong.

4. The backup is encrypted with a key nobody has

A modern variant: backups are pushed to S3 with server-side encryption using a KMS key, or the backup tool encrypts them with a passphrase. The site goes down. Someone tries to restore. The passphrase is in the head of a person who left in 2022. The KMS key is in an AWS account that was decommissioned during a cloud-cost rationalisation.

How to spot it: stop your hypothetical disaster-recovery test on step 3 above. Can you decrypt the backup right now, from a new machine, with the credentials available to your current team? If you have to “find the password somewhere”, that’s the failure.

5. The backup is fine, but the restore procedure is fiction

This is the subtlest one. The backup files are present, complete, in the right location, with the right encryption. The restore would work. But nobody on the team has ever actually performed the restore steps. The “documented runbook” is a wiki page last edited four years ago, listing commands for a different Moodle™ version, on an OS we no longer use, referring to a pg_dump flag that was deprecated in PostgreSQL 13.

You’ll find out about this one on the night you need it most.

How to spot it: the test above. Run the restore from a cold start. Time it.

Why this happens

None of this is anyone’s fault, really. Backups exist on the boundary between two teams — the people who run the site and the people who run the infrastructure — and that boundary is where things rot. Both teams reasonably assume the other has it covered.

It’s also boring. Restoring backups isn’t a sprint goal. Nobody gets a promotion for proving that backups work. So nobody does it, until someone has to do it under pressure.

The fix isn’t complicated, it’s just procedural: schedule it. Once a quarter, ideally once a month. Restore a real backup to a sandbox. Log in. Click around. Throw the sandbox away.

That’s the whole programme. It works because it forces the system to either prove itself or fail loudly in a safe environment.

What we do for our backup clients

Because this is so much our job, here’s how we do it for the sites we look after — copy any of it that’s useful:

The interesting bit isn’t any single piece of this. It’s that the verification step is non-negotiable. If a restore would fail, we want to learn that in a sandbox at 11 AM on a Tuesday, not at 2 AM on the night before exams.

If you remember one thing

A backup that has never been restored is not a backup. It’s a folder with the right name on it.

Schedule a restore test for sometime in the next two weeks. Block out an hour. Spin up a sandbox. Do the test.

If everything works, you’ll feel briefly excellent. If something fails, you’ll have found out at a time of your choosing — which is the entire point.

Need help with this? If you’d rather not do this yourself, our Moodle™ backup & DR service handles all of it — including a free “audit your current backup” engagement on day one. We’ll tell you honestly whether what you have works.

This is a real incident from a client we helped last year, anonymised. The numbers and timings are accurate. We’ve changed identifying details and kept it deliberately practical — if you’re ever in this position, the order things happen in is the part that matters.

The call

2:11 AM UK time, Tuesday in mid-November. A WhatsApp message:

“Moodle is down. Showing ‘Error reading from database’. We have final exams starting Thursday at 09:00. Please help.”

The client was a vocational college in continental Europe, roughly 1,400 students, running Moodle™ 4.1 LTS on a single VPS — Apache, PHP-FPM, MariaDB 10.6 — at a generic hosting provider. They were not, at this point, our managed-hosting client. We did their off-site backups, that was all.

The on-call engineer (me) replied within four minutes — fast for that hour because the WhatsApp ringer had been on for a different reason — and we got to work.

What we knew at 2:15 AM

The exam window mattered because Moodle™ was the delivery mechanism: the exam was an in-person, on-paper assessment, but the question papers, the rubrics, the timing, and crucially the assignment-submission window for the take-home component all lived in Moodle™ courses. Losing Moodle™ for 30 hours would have been bad but recoverable. Losing it on Thursday morning would have been a serious incident for the college.

02:15–02:35 — Triage

I asked the client to do three things and not log into anything else:

  1. Give me read-only SSH access to the VPS.
  2. Forward me the last 200 lines of /var/log/mysql/error.log and /var/log/apache2/error.log.
  3. Take a photo of the actual error page on the site so I had the literal wording.

Step 3 sounds trivial. It’s actually one of the most useful things you can ask for under pressure: the exact text of the visible error often tells you which subsystem failed first, and clients often paraphrase (“the database is broken”) in a way that loses information.

While they did that, I pulled up our backup status. The last verified restore for this client was 11 days earlier, the standard monthly check. It had passed. That meant the backup we had on the shelf was almost certainly restorable — useful to know before we needed it.

The Apache error log was full of:

PHP Fatal error: Uncaught dml_connection_exception:
Error connecting to database

The MariaDB error log was much more interesting:

[ERROR] [FATAL] InnoDB: Table mdl_question_attempt_step_data in
file ./moodle/mdl_question_attempt_step_data.ibd is encrypted but
encryption service or used key_id 1 is not available.

That was the smoking gun. The MariaDB encryption key file had become unreadable. The database was up, but some tables couldn’t be opened. Moodle™‘s connection check tries to query a system table that, on this site, lived in the same tablespace — so every page returned the connection error.

This is not a Moodle™ bug. It’s a MariaDB-level thing that can happen for several reasons: a botched OS upgrade, a /etc/mysql/encryption/ permissions change, a stray chmod -R from a backup script, or — as it turned out here — an apt unattended-upgrades run that had silently updated mariadb-server-core two hours earlier and changed the ownership of the key directory.

02:35–02:55 — Decide between repair and restore

We had two paths:

The temptation in a crisis is to try Path A first because it’s faster if it works. But the cost of failing Path A and then falling back to Path B is the time you already spent on Path A. So we did both in parallel.

I asked the client to authorise spinning up a recovery VM with the same provider in the same region. Path B started running on a new box while I worked on Path A on the live one.

02:55–03:20 — Path A: the fix

The encryption key file lived at /etc/mysql/encryption/keyfile. Owner had been changed during the package update from mysql:mysql to root:root. MariaDB couldn’t read it.

$ ls -la /etc/mysql/encryption/
total 12
drwxr-xr-x 2 root root 4096 Nov 12 00:13 .
drwxr-xr-x 4 root root 4096 Nov 12 00:13 ..
-rw------- 1 root root  768 Aug  4 11:22 keyfile

Compare to a server where it works:

-rw------- 1 mysql mysql 768 Aug  4 11:22 keyfile

The fix was one line:

sudo chown mysql:mysql /etc/mysql/encryption/keyfile
sudo systemctl restart mariadb

I did not run that yet. The temptation in a crisis is to act fast. Acting fast on a broken database is how you turn a recoverable incident into an unrecoverable one. Instead:

  1. I took a filesystem snapshot of /var/lib/mysql first — rsync -a to a sibling directory. About six minutes for 18 GB.
  2. I made a copy of the encryption keyfile to a path the recovery VM could pull from. Critically, the keyfile is needed to read the encrypted data later. If you lose it, your encrypted tables are unreadable forever. Backups that don’t include the key are useless against this specific failure.
  3. Only then did I run the chown.

I restarted MariaDB. It came up. I checked the error log for new lines — clean. I mysql --execute "SELECT COUNT(*) FROM mdl_question_attempt_step_data;" and got a number. The site loaded. Login worked. Course list rendered.

03:21 AM. We were back.

03:20–04:00 — Don’t trust the win

Coming back from an incident is the moment you most want to declare victory and go to bed. It’s also the moment you’re most likely to miss the second, smaller problem hiding under the first.

I asked the on-call engineer at our end — the one who’d been pulled in as a second pair of eyes at 02:35 — to do a different test in parallel: take the off-site backup from 03:00 UTC the previous day and start restoring it onto Path B’s recovery VM. Not to use it, but to have it ready, fully smoke-tested, in case anything else surfaced before Thursday.

This is the part most teams skip. They get the site back and stop. We treated the situation as still-fragile until the exam window had passed.

While that ran, I worked through a checklist on the live site:

The recovery VM finished its restore at 03:48. Smoke tests on the recovery VM also passed. We had two working sites: the live one, plus a 23-hour-old backup ready to swap in if anything else broke.

I left both up. Costs the client about $30 to keep the recovery VM running until Thursday — a small price for the option value.

What actually caused it

The post-mortem is more useful than the war story.

Root cause: unattended-upgrades had run a mariadb-server-core update at 00:13. As part of the package update, a postinst script had chown root:root /etc/mysql/encryption/keyfile — almost certainly a bug or an interaction with a non-standard config the client’s previous admin had set up.

Contributing factors:

The fixes we put in place that week:

What the client paid

Three things.

A flat emergency fee for the night — quoted in the WhatsApp thread before I logged in. The client agreed in writing.

A monthly maintenance retainer afterwards, which they hadn’t been paying before but signed up to within a week.

A small once-off fee to write the post-mortem and put the preventative measures in place.

We don’t bill hourly. We never have. The flat-fee model means we have no incentive to drag an incident out, and the client knows their downside before authorising us to start.

If you remember one thing

Two things, actually.

One: in a crisis, work two paths in parallel. The fast path and the safe path. The fast path saves the day when it works; the safe path saves your career when it doesn’t.

Two: the moment you get a site back is the moment you most want to declare victory. Don’t. Smoke-test, leave the recovery option warm, and write the post-mortem the next morning while it’s fresh. The next outage is being prepared right now by something nobody’s looking at.

If you’d like a similar level of “we keep your backup warm and rehearsed” without it being your job, that’s our backup & DR service. Or if your Moodle™ is on fire right now, email [email protected] and we’ll be on it within 30 minutes.

We upgrade a lot of LMS sites. Across Moodle™, Open edX, Totara, Canvas and a few others, the pattern is the same: point-release upgrades within the same major version are mostly boring (which is exactly how we like them). Major-version upgrades are where the pain lives.

This article is for anyone planning a major upgrade. It covers three things, in order: what can actually go wrong, why backing up to multiple locations matters before you commit, and the short pre-flight checklist we run before any upgrade. It is deliberately version-agnostic — the principles are the same whether you’re going from Moodle™ 3.x to 4.x, jumping a Totara release, upgrading Open edX across two named cuts, or moving a Canvas self-hosted install forward.

What actually goes wrong

In a decade of major-version upgrades, the failures are not exotic. They cluster into about six modes. If you know them in advance, you can design the pre-flight to catch them.

1. A third-party plugin breaks on the new version. Core LMS code is usually upgraded cleanly by its maintainers — they’ve test-rigged it. Third-party plugins are not. A plugin that’s no longer maintained, or whose maintainer hasn’t released a target-version build, can throw fatal errors on first page load. We see this on roughly half of large-site upgrades.

2. The active theme renders wrong, or doesn’t render. Themes depend on underlying frameworks (Bootstrap versions, Mustache templates, renderer hooks). Major-version jumps often change those. A theme that “just worked” before can show broken layouts, missing buttons, or unstyled forms after.

3. A custom local plugin or core patch uses a deprecated API. If your developer team has written code against the LMS’s internal APIs, capability names, database helpers, or course format hooks — that code can stop compiling. Particularly painful if the original author is unreachable and the code is undocumented.

4. A schema migration takes longer than your maintenance window. Some upgrades rewrite large tables (question banks, gradebooks, course-completion state). On a small site this takes seconds. On a 25,000-learner site it can take hours. If you booked a 30-minute window and the migration runs 90 minutes, your users come back to a half-upgraded site.

5. An integration’s token, endpoint or auth contract changes. LTI tool providers, SSO/SAML configurations, OAuth2 issuers, plagiarism detectors, video-conference connectors — any of these can quietly stop working when the LMS’s behaviour around them shifts.

6. The PHP or database version under the LMS is no longer supported. Major LMS releases regularly raise minimums. If you’re on PHP 7.4 and the target requires 8.1, the upgrade itself succeeds but the application fails to boot.

Each of these is preventable. The pre-flight checklist further down catches them.

Why backups in multiple locations matter — before you commit

The number-one rule of upgrades is: you should be able to fully revert to the pre-upgrade state. That sounds obvious. In practice, most teams discover it isn’t true at the moment they need it.

The reason is single points of failure in your backup arrangement. Almost every “we have backups” story has at least one:

The defensive posture before a major upgrade is straightforward: the data needs to exist in at least two independent locations, and both copies need to be verified as restorable in the past week. We use three locations for hosted clients — two clouds in different regions, plus a third immutable copy — because the marginal cost is small and the upside is enormous.

If your current arrangement is “we have one backup somewhere”, fix backups before you upgrade. A failed upgrade is a maintenance window. A failed upgrade with no working backup is an incident — and you don’t want to discover that one at 2 AM.

A workable minimum for an upgrade weekend:

That’s four artefacts, in three locations, with one of them rehearsed. If any of those is missing, postpone the upgrade until it isn’t.

The pre-flight checklist

This is the short version of what we actually run. It groups the 28 fine-grained items from our internal runbook into ten decisions. Each one is a go / no-go gate.

1. Verified, recent, multi-location backups

See the section above. If you cannot tick this honestly, stop here.

2. The exact source and target versions

Not “we’re on the LTS-ish version”. The exact build number of the source. The exact target. You’ll need both to find out what changed, and whether your plugins, theme and integrations will follow.

3. The target’s release notes, read end-to-end

Specifically the upgrade section. Note every breaking change, deprecation, capability rename, and any minimum-version bumps for PHP, the database engine, or any extension. Read the yellow flags, not just the red ones.

4. PHP, database, and OS versions confirmed supported

If any of these need a separate upgrade, do that first, on its own, with its own rehearsal and its own rollback. Never combine a PHP/DB upgrade with an LMS major upgrade — the failure surface multiplies.

5. Plugin and theme compatibility audit

For every third-party plugin and every theme, find out if a target-compatible version exists. Build a small spreadsheet: plugin name, current version, latest available, target-compatible version. Anything without a target-compatible version is a decision: replace, port, drop, or postpone.

6. Custom code reviewed against the changelog

Any local plugins, custom blocks, core patches, or theme overrides need a developer to walk through the upgrade notes and confirm none of the APIs the custom code uses have changed. Budget more time if the original author is unreachable.

7. Sandbox that’s a true clone

Same version, same plugins, same theme, same data shape, same PHP/DB versions, same OS. Restore from the same backup you’d use in a disaster. If the sandbox doesn’t match production, the rehearsal isn’t meaningful.

8. Full upgrade rehearsed end-to-end in the sandbox

Actually run it. The whole thing. Time it wall-clock from start to “Success”. If it doesn’t fit your maintenance window, change the plan — don’t try to be heroic on the night.

9. Smoke-test script run, and every integration re-checked

A defined list, written down, ticked off: admin login, course render, quiz attempt, assignment submission, file download, gradebook view, course backup/restore, learner enrolment, forum post, completion report. Plus every integration — LTI, SSO/SAML, OAuth2, payments, plagiarism, video, repositories — clicked through end-to-end.

10. Rollback plan, abort criteria, and a 24-hour change freeze

The rollback plan is normally: restore the pre-upgrade backup. Time the restore in the sandbox too — your maintenance window must be long enough for a rollback if needed, not just an upgrade. Write down the exact criteria for aborting (e.g. smoke-test fails on more than two items; p95 page-load worse than baseline by 30%; cron queue not draining within 30 minutes). Freeze production changes 24 hours before — no new courses, no plugin tweaks, no setting changes.

If you remember one thing

The upgrade itself is rarely the hardest part. The hardest part is being honest, before the night, about whether every item above is genuinely green. We have postponed upgrades — sometimes more than once on the same site — because something in this list wasn’t true. Every time, the team thanked us afterwards. Postponing is cheap. Failing live is not.

If you’d like us to drive an upgrade for you — Moodle™, Open edX, Totara, Canvas, Sakai, or another platform — the upgrades & maintenance service is this checklist, executed by us. Or if you’d just like a second pair of eyes on an upgrade you’re running yourself, say hello.