Writing and using fragment shaders

Custom shaders can be used to provide rich graphical effects beyond those provided by the Flutter SDK. A shader is a program authored in a small, Dart-like language, known as GLSL, and executed on the user's GPU.

Custom shaders are added to a Flutter project by listing them in the pubspec.yaml file, and obtained using the FragmentProgram API.

Shaders, in the form of GLSL files with the .frag extension, must be declared in the shaders section of your project's pubspec.yaml file. The Flutter command-line tool compiles the shader to its appropriate backend format, and generates its necessary runtime metadata. The compiled shader is then included in the application just like an asset.

flutter:
  shaders:
    - shaders/myshader.frag

When running in debug mode, changes to a shader program trigger recompilation and update the shader during hot reload or hot restart.

Shaders from packages are added to a project with packages/$pkgname prefixed to the shader program's name (where $pkgname is the name of the package).

To load a shader into a FragmentProgram object at runtime, use the FragmentProgram.fromAsset constructor. The asset's name is the same as the path to the shader given in the pubspec.yaml file.

void loadMyShader() async {
  var program = await FragmentProgram.fromAsset('shaders/myshader.frag');
}

The FragmentProgram object can be used to create one or more FragmentShader instances. A FragmentShader object represents a fragment program along with a particular set of uniforms (configuration parameters). The available uniforms depends on how the shader was defined.

void updateShader(Canvas canvas, Rect rect, FragmentProgram program) {
  var shader = program.fragmentShader();
  shader.setFloat(0, 42.0);
  canvas.drawRect(rect, Paint()..shader = shader);
}

Fragment shaders can be used with most Canvas APIs by setting Paint.shader. For example, when using Canvas.drawRect the shader is evaluated for all fragments within the rectangle. For an API like Canvas.drawPath with a stroked path, the shader is evaluated for all fragments within the stroked line. Some APIs, such as Canvas.drawImage, ignore the value of the shader.

void paint(Canvas canvas, Size size, FragmentShader shader) {
  // Draws a rectangle with the shader used as a color source.
  canvas.drawRect(
    Rect.fromLTWH(0, 0, size.width, size.height),
    Paint()..shader = shader,
  );
​
  // Draws a stroked rectangle with the shader only applied to the fragments
  // that lie within the stroke.
  canvas.drawRect(
    Rect.fromLTWH(0, 0, size.width, size.height),
    Paint()
      ..style = PaintingStyle.stroke
      ..shader = shader,
  )
}

Fragment shaders can also be used with the ImageFilter API. This allows using custom fragment shaders with the ImageFiltered class or the BackdropFilter class to apply shaders to already rendered content. ImageFilter provides a constructor, ImageFilter.shader, for creating an ImageFilter with a custom fragment shader.

Widget build(BuildContext context, FragmentShader shader) {
  return ClipRect(
    child: SizedBox(
      width: 300,
      height: 300,
      child: BackdropFilter(
        filter: ImageFilter.shader(shader),
        child: Container(
          color: Colors.transparent,
        ),
      ),
    ),
  );
}

When using ImageFilter with BackdropFilter, a ClipRect can be used to limit the area that is affected by the ImageFilter. Without a ClipRect the BackdropFilter will be applied to the whole screen.

ImageFilter fragment shaders receive some uniforms automatically from the engine. The sampler2D value at index 0 is set to the filter input image, and the float values at indices 0 and 1 are set to the image's width and height. Your shader must specify this constructor to accept these values (for example, a sampler2D and a vec2), but you should not set them from your Dart code.

When targeting OpenGLES the y-coordinates of the texture will be flipped so the fragment shader should un-flip the UVs when sampling from textures provided by the engine.

#version 460 core
#include <flutter/runtime_effect.glsl>

out vec4 fragColor;

// These uniforms are automatically set by the engine.
uniform vec2 u_size;
uniform sampler2D u_texture;

void main() {
  vec2 uv = FlutterFragCoord().xy / u_size;
#ifdef IMPELLER_TARGET_OPENGLES
  // When sampling from u_texture on OpenGLES the y-coordinates will be flipped.
  uv.y = 1.0 - uv.y;
#endif
  vec4 color = texture(u_texture, uv);
  float gray = dot(color.rgb, vec3(0.299, 0.587, 0.114));
  fragColor = vec4(vec3(gray), color.a);
}

Fragment shaders are authored as GLSL source files. By convention, these files have the .frag extension. (Flutter doesn't support vertex shaders, which would have the .vert extension.)

Any GLSL version from 460 down to 100 is supported, though some available features are restricted. The rest of the examples in this document use version 460 core.

Shaders are subject to the following limitations when used with Flutter:

• UBOs and SSBOs aren't supported
• sampler2D is the only supported sampler type
• Only the two-argument version of texture (sampler and uv) is supported
• No additional varying inputs can be declared
• All precision hints are ignored when targeting Skia
• Unsigned integers and booleans aren't supported

A fragment program can be configured by defining uniform values in the GLSL shader source and then setting these values in Dart for each fragment shader instance.

Floating point uniforms with the GLSL types float, vec2, vec3, and vec4 are set using the FragmentShader.setFloat method. GLSL sampler values, which use the sampler2D type, are set using the FragmentShader.setImageSampler method.

The correct index for each uniform value is determined by the order that the uniform values are defined in the fragment program. For data types composed of multiple floats, such as a vec4, you must call FragmentShader.setFloat once for each value.

For example, given the following uniforms declarations in a GLSL fragment program:

uniform float uScale;
uniform sampler2D uTexture;
uniform vec2 uMagnitude;
uniform vec4 uColor;

The corresponding Dart code to initialize these uniform values is as follows:

void updateShader(FragmentShader shader, Color color, Image image) {
  shader.setFloat(0, 23);  // uScale
  shader.setFloat(1, 114); // uMagnitude x
  shader.setFloat(2, 83);  // uMagnitude y
​
  // Convert color to premultiplied opacity.
  shader.setFloat(3, color.red / 255 * color.opacity);   // uColor r
  shader.setFloat(4, color.green / 255 * color.opacity); // uColor g
  shader.setFloat(5, color.blue / 255 * color.opacity);  // uColor b
  shader.setFloat(6, color.opacity);                     // uColor a
​
  // Initialize sampler uniform.
  shader.setImageSampler(0, image);
 }

Observe that the indices used with FragmentShader.setFloat do not count the sampler2D uniform. This uniform is set separately with FragmentShader.setImageSampler, with the index starting over at 0.

Any float uniforms that are left uninitialized will default to 0.0.

The reflection data generated by the Flutter's shader compiler can be audited with the following commands in order to see things like uniform offsets.

cd $FLUTTER
# Generate the .sl file.
`find bin/ -name impellerc` \
  --runtime-stage-metal \
  --iplr \
  --input=path/to/myshader.frag \
  --sl=foo.sl \
  --spirv=foo.spirv \
  --include=engine/src/flutter/impeller/compiler/shader_lib/ \
  --input-type=frag
# Convert the .sl file to .json
flatc \
  --json \
  ./engine/src/flutter/impeller/runtime_stage/runtime_stage.fbs \
  -- ./foo.sl
# View results
cat foo.json

The shader has access to a varying value that contains the local coordinates for the particular fragment being evaluated. Use this feature to compute effects that depend on the current position, which can be accessed by importing the flutter/runtime_effect.glsl library and calling the FlutterFragCoord function. For example:

#include <flutter/runtime_effect.glsl>

void main() {
  vec2 currentPos = FlutterFragCoord().xy;
}

The value returned from FlutterFragCoord is distinct from gl_FragCoord. gl_FragCoord provides the screen space coordinates and should generally be avoided to ensure that shaders are consistent across backends. When targeting a Skia backend, the calls to gl_FragCoord are rewritten to access local coordinates but this rewriting isn't possible with Impeller.

There isn't a built-in data type for colors. Instead they are commonly represented as a vec4 with each component corresponding to one of the RGBA color channels.

The single output fragColor expects that the color value is normalized to be in the range of 0.0 to 1.0 and that it has premultiplied alpha. This is different than typical Flutter colors which use a 0-255 value encoding and have unpremultipled alpha.

A sampler provides access to a dart:ui Image object. This image can be acquired either from a decoded image or from part of the application using Scene.toImageSync or Picture.toImageSync.

#include <flutter/runtime_effect.glsl>

uniform vec2 uSize;
uniform sampler2D uTexture;

out vec4 fragColor;

void main() {
  vec2 uv = FlutterFragCoord().xy / uSize;
  fragColor = texture(uTexture, uv);
}

By default, the image uses TileMode.clamp to determine how values outside of the range of [0, 1] behave. Customization of the tile mode is not supported and needs to be emulated in the shader.

When targeting the Skia backend, loading the shader might be expensive since it must be compiled to the appropriate platform-specific shader at runtime. If you intend to use one or more shaders during an animation, consider precaching the fragment program objects before starting the animation.

You can reuse a FragmentShader object across frames; this is more efficient than creating a new FragmentShader for each frame.

For a more detailed guide on writing performant shaders, check out Writing efficient shaders on GitHub.

For more information, here are a few resources.

• The Book of Shaders by Patricio Gonzalez Vivo and Jen Lowe
• Shader toy, a collaborative shader playground
• simple_shader, a simple Flutter fragment shaders sample project
• flutter_shaders, a package that simplifies using fragment shaders in Flutter