Background Processing with Quirrel

Most web applications need some kind of background processing at some point:

reminding a user of an upcoming event
sending out invoices at the beginning of the month
removing old data
processing an upload in the background
...

In other frameworks, you'd reach for solutions like Sidekiq or Celery.

For Blitz.js, Quirrel is a good choice. It's OSS and has a hosted version, and works by calling back to your Next.js API routes.

Quirrel is developed by Simon Knott, a Blitz community member. Don't hesitate to reach out to him if there are any questions!

Getting Started

Install Quirrel by running blitz install quirrel.

To speed up development, Quirrel allows you to monitor pending jobs in the Development UI. In there, you can also manually invoke jobs, so you don't have to wait everytime you test something. To use it, simply run quirrel ui or open ui.quirrel.dev in your browser.

For any Quirrel questions not covered by this document, check out the Quirrel Docs.

For deploying an application that uses Quirrel, check out this guide: Deploying Quirrel

Recipes

These recipes are designed to give you a general idea of how to use Quirrel.

Reminding a user of an upcoming event

This recipe schedules a booking reminder to be sent 30 minutes before show-time.

First, you define your Queue:

// app/api/booking-reminder
import db from "db"
import {Queue} from "quirrel/blitz"
import sms from "some-sms-provider"

// it's important to export it as default
export default Queue(
  "api/booking-reminder", // 👈 the route that it's reachable on
  async (bookingId: number) => {
    const booking = await db.booking.findUnique({
      where: {id: bookingId},
      include: {user: true, event: true},
    })

    await sms.send({
      to: booking.user.phoneNumber,
      content: `Put on your dancing shoes for ${booking.event.title} 🕺`,
    })
  },
)

Import the above file somewhere else and call .enqueue to schedule a new job:

// app/mutations/createBooking
import db from "db"
import bookingReminder from "app/api/booking-reminder" // 👈 the above file
import { subMinutes } from "date-fns"

export default async function createBooking(eventId, ctx) {
  const booking = await db.booking.create({ ... })
  await bookingReminder.enqueue(booking.id, {
    runAt: subMinutes(booking.event.date, 30),

    // allows us to address this job later for deletion
    id: booking.id
  })
}

That's all we need! Your customers will now be reminded 30 minutes before their booking begins.

If a booking is canceled, we can also delete the reminder job:

// app/mutations/cancelBooking
import db from "db"
import bookingReminder from "app/api/booking-reminder"

export default async function cancelBooking(bookingId, ctx) {
  ...
  await bookingReminder.delete(
    bookingId // this is the same ID we set above
  )
}

If your SMS provider is flaky, specify a retry schedule:

export default Queue(
  ...,
  ...,
  {
    // if execution fails, it will be retried
    // 10s, 1min and 2mins after the scheduled date
    retry: [ "10s", "1min", "2min" ]
  }
)

Sending out Invoices at the beginning of the Month

For this, Quirrel's CronJob is the perfect fit.

// app/api/monthly-invoice
import db from "db"
import {CronJob} from "quirrel/blitz"
import stripe from "stripe"

export default CronJob(
  "api/monthly-invoice", // 👈 the route that it's reachable on
  "0 0 1 * *", // same as @monthly (see https://crontab.guru/)
  async () => {
    const customers = await db.customers.findAll()
    await Promise.all(
      customers.map(async (customer) => {
        await stripe.finalizeInvoice(customer.stripeId)
      }),
    )
  },
)

Removing old data

Again, CronJob is a great fit.

// app/api/remove-old-data
import db from "db"
import {CronJob} from "quirrel/blitz"
import {subDays} from "date-fns"

export default CronJob(
  "api/remove-old-data", // 👈 the route that it's reachable on
  "0 * * * *", // same as @hourly (see https://crontab.guru/)
  async () => {
    await db.logs.deleteMany({
      where: {
        customer: {
          isPremium: false,
        },
        date: {
          lt: subDays(Date.now(), 3),
        },
      },
    })
  },
)

Processing an Upload in the Background

Uploaded data shouldn't be sent to Quirrel, but be stored in your own database. Add a new DB entity for it:

model UploadedCSV {
  id          Number @id @default(autoincrement())
  data        String
}

Then when a user uploads something, you insert it into the database and enqueue the resulting record's ID into Quirrel.

// app/mutations/uploadCsvForProcessing
import db from "db"
import csvProcessingQueue from "app/api/process-csv"

export default async function uploadCsvForProcessing(data: string) {
  const record = await db.uploadedCsv.create({
    data: {data},
  })

  await csvProcessingQueue.enqueue(record.id)

  return record.id
}

Our Quirrel Queue then fetches the corresponding data from the database and does the required processing. After that's done, it deletes the database record (alternative: add a flag called "finishedProcessing" and set it to true).

// app/api/process-csv
import db from "db"
import {Queue} from "quirrel/blitz"

export default Queue("api/process-csv", async (uploadId: number) => {
  const upload = await db.uplodadedCsv.findUnique({
    where: {id: uploadId},
  })

  await doYourProcessing(upload.data)

  await db.uplodadedCsv.delete({where: {id: uploadId}})
})

Now when you want to know wether an upload has already been processed, you can look it up in your own database:

// app/queries/hasFinishedProcessing
import db from "db"

export default async function hasFinishedProcessing(uploadId: number) {
  const count = await db.uploadedCsv.count({where: {uploadId}})
  return count === 0
}