Speeding up tint() in p5.js

September 3, 2022

Recently, some of my code was merged into the p5.js source to speed up the tint() method. There was an issue open about it causing sketches to drop frames, and indeed, it's pretty easy to reproduce the issue. I decided to look into it and see what I could do.

An example of tinting, using a test image found in the p5 repo

The problem

The tint() method is simple: When you call tint(r, g, b, a) and then draw an image, the color you specify gets multiplied with the colors of the image. It's basically doing this, in pseudocode:

function tint(r, g, b, a) {
  for (pixel of pixels) {
    pixel.r = pixel.r * (r / 255)
    pixel.g = pixel.g * (g / 255)
    pixel.b = pixel.b * (b / 255)
    pixel.a = pixel.a * (a / 255)
  }
}

It turns out, this is implemented by looping through all the pixels of the image, and multiplying each channel of each pixel. This is understandably pretty slow when there are a lot of pixels.

Thankfully, applying the same operation to every pixel is what GPUs are built for! Let's see what we can do to hardware accelerate this method.

Surely there's a blend mode for this

The tinted image is currently created by loading the image into a 2D canvas, updating the pixels, and then drawing the result. If we want to use the GPU, we could make a WebGL context and use a shader, but that feels like a pretty heavy solution. 2D canvases already come with some different options for blending colors! In JavaScript, these are defined by the globalCompositeOperation, but it is known more commonly as the blend mode.

Blending a solid color onto an image using some common blend mode options

The operation done by tint() is just a multiplication of each channel of the source color with each of the destination, and conveniently, there is a blend mode called multiply for just that! Blend modes are hardware accelerated on most machines these days, so we don't need to write our own shader to get a speedup.

Awesome, we're done! Let's try it with an image that has transparency too, just as a sanity check. I've cut some holes in the cat image from the p5 source repo.

Oh no

It almost works.

If I were to tint it white with tint(255, 255, 255, 255), that should be equivalent to multiplying the red, green, blue, and alpha channels by 1. Opacity is defined by the alpha channel, so when you multiply that by 1, nothing should change, right?

Unfortunately, when we draw a white rectangle on top with the multiply blend mode, it seems to overwrite the alpha channel and make it opaque.

I thought that maybe, instead of multiplying the alpha channels, it was simply replacing the alpha channel of the destination with the alpha channel of the source. I was, after all, blending a fully opaque white rectangle on top of the image. Could this be the pseudocode of what the multiply blend mode does?

function multiply(r, g, b, a) {
  for (pixel of pixels) {
    pixel.r = pixel.r * (r / 255)
    pixel.g = pixel.g * (g / 255)
    pixel.b = pixel.b * (b / 255)

    // Replace the alpha instead of multiplying it
    pixel.a = a / 255
  }
}

If this is true, then maybe I need to make the white image have the target opacity before blending it. Let's try that out! There's a blend mode called destination-in which uses the existing colors on the destination canvas, but replaces the opacity with the opacity of the layer being blended. We can first apply that to our tint color, and then multiply the result with the main image.

That's better! The fully transparent part remains transparent! But unfortunately, the semi-transparent section is not quite right. It seems like it's gotten whiter as it gets semi-transparent.

My previous hypothesis was slightly off. It seems that what multiply does is even weirder: it first mixes the destination image onto an opaque white background, and then replaces the opacity. In pseudocode:

function multiply(r, g, b, a) {
  for (pixel of pixels) {
    const opacity = pixel.a / 255

    // Mix the colors with white based on how opaque they are
    pixel.r = mix(255, pixel.r, opacity) * (r / 255)
    pixel.g = mix(255, pixel.g, opacity) * (g / 255)
    pixel.b = mix(255, pixel.b, opacity) * (b / 255)

    // Replace the alpha instead of multiplying it
    pixel.a = a / 255
  }
}

So this means, if we want to avoid our semi-transparent colors getting mixed with white, we have to make them fully opaque before multiplying.

Accommodating multiply's weird demands

It turns out, lots of blend modes destroy the alpha channel when you use them! Maybe we can use that to our advantage if we're trying to reconstruct a fully-opaque image and blend the original image with itself. Looking through the list of blend modes on MDN, two stand out:

color: Preserves the luma of the bottom layer, while adopting the hue and chroma of the top layer.

luminosity: Preserves the hue and chroma of the bottom layer, while adopting the luma of the top layer.

An image is represented in memory as red, green, and blue values, but these blend modes convert the image to hue, chroma, and luminosity. It sounds like color sets two of these three properties, hue and chroma, and luminosity sets the third. So If we blend once with each of these modes, will we turn the semi-transparent parts fully opaque?

The answer is yes!*

 

 

*With the caveat that we've lost a little bit of data in the process. Internally, canvases don't store color data as [r, g, b, a], they actually store it in premultiplied alpha format, which is [r*(a/255), g*(a/255), b*(a/255), a]. This is to make it slightly more efficient to mix colours for the default source-over blend mode. When you query color information, it divides again before showing you. However, because it's storing each value as an integer from 0-255 after you multiply by alpha, when you divide to get back, the result might not be the same. Imagine you are storing the value 231 with opacity 0.3:

That warning aside, it's close enough to be functional!

Assembling Frankenstein's monster

Now that we can reconstruct a fully opaque version, we can finally multiply it with a color and re-apply the alpha to get a tinted image!

It's ugly, but it works!

aaAAAAAaahhhHHH!!!

That's a lot

This took 4 passes of blending in order to tint properly. It only took one pass if we didn't have any opacity! It's still faster than iterating through pixels on the CPU, but it's 4x as slow as I was hoping. It would be great if we could use the one-pass version if we knew ahead of time that we have no transparency. Unfortunately, the p5 tint API has no way of knowing this without looping over all the pixels, which is what we were initially trying to avoid. But maybe there are other cases we can still speed up!

One more common use of tint is to tint with semi-transparent white to fade out an image. For example:

function draw() {
  background(255)
  push()
  const alpha = map(
    cos((millis() % 2000) / 2000),
    -1, 1,
    0, 255
  )
  tint(255, alpha)
  image(cat, 0, 0)
  pop()
}

I bet you thought you were done looking at this cat. So did I. Think again.

Thankfully, although tint knows nothing about the contents of the image, it can easily check the tint color to see if it's partially transparent white!

If this is the case, then we can use the canvas globalAlpha property to quickly and efficiently draw an image with a given opacity:

if (this._tint[0] < 255 || this._tint[1] < 255 || this._tint[2] < 255) {
  // An actual tint color has been applied. Put all the code for the big
  // four-pass render here.
} else {
  // Wow! So fast!
  ctx.globalAlpha = this._tint[3] / 255;
  ctx.drawImage(img.canvas, 0, 0);
}

Now you can use tint to fade just opacity without any performance issues!

So should you still use this?

This will be landing in the next version of p5 (at the time of writing we're at 1.4.2, so presumably this will be in 1.4.3.) But do I recommend that you actually use tint?

The answer, like the answer to many things, is it depends.

Start with it!

You should 100% use it for all your tinting needs at first. Part of p5's whole philosophy is to be easy to learn, and new programmers should not be worrying themselves with squeezing the most performance possible out of their sketches. In my opinion, it is perfectly fine to trade performance for less developer time, especially when learning. The goal of this tint update is to make it a tool 90% of people can use without having to think about it.

Use something else if you really need to

You should only start thinking about it if you're doing so much each frame that you're dropping frames, you've profiled your code, and you've identified tint as being the main bottleneck. Only then would I suggest one of the following alternatives:

Do you want to get involved too?

p5 has a really friendly community! I encourage you to join in. Some options on how to do that: