Alpha in Starling Filters and Basic Branching in AGAL

Back in September, a Chris posted a comment on this blog asking about a Starling filter that would create a circular mask over an image. Being the lazy sort I am, I never got back to the commenter (Sorry, Chris). Some stuff I was doing at work the other day reminded me of the question, though; and, although it’s a relatively simple thing to do, the task raises two interesting problems I thought deserved a whole post rather than just a quick response.

In plain English to create a circular mask we would want to do something like this: Find the distance of every pixel in an image from the center of our circle. If that distance is greater than the radius of our circle then we should set its alpha to zero. Well, to find distances we can turn to the good ol’ Pythagorean theorem – distance equals the square root of distance x times distance x plus distance y times distance y (I’m still amazed I use information I learned in high school in my day to day life. Who’d a thunk those teachers were on to something?). In a sort of pseudo/AS3 code, our operation may look something like this:

var distancex = pixel.x - center.x;
distancex = distancex * distancex;
var distancey = pixel.y - center.y;
distancey = distancey * distancey;
var distance = distancex + distancey;
distance = Math.sqrt(distance);
if (distance > radius)
	pixel.alpha = 0;

We can start to port that pseudocode over to AGAL like so:

; Assume v0 contains original pixel position
; Assume fc0 contains our circle definition:
; fc0 = [centerX, centerY, radius, 1]
sub ft0.x, 	v0.x, 	fc0.x
mul ft0.x, 	ft0.x, 	ft0.x
sub ft0.y, 	v0.y, 	fc0.y
mul ft0.y, 	ft0.y, 	ft0.y
add ft0.z, 	ft0.x, 	ft0.y
sqt ft0.z, 	ft0.z
tex ft1,	v0, 	fs0<2d, clamp, linear, mipnone>
; ...

But there lies the rub… We now have our distance value in ft0.z and our texture information (red, green, blue, and alpha) in register ft1.xyzw, but how do we do the if conditional? Ideally, we would like to write something like:

if (ft0.z > fc0.z) mov ft1.w, 0

Unfortunately, AGAL doesn’t provide such explicit branching statements. It does, however, provide four nifty little ‘set’ operations which, with a little ingenuity, can handle such conditionals: SGE (“set if greater than or equal”), SLT (“set if less than”), SEQ (“set if equal”), and SNE (“set if not equal”). These operations will set a register’s component to either 1 or 0 depending on whether or not the conditional passes. Let’s take a look at SLT for a moment. SLT (like the other 3 set operations) takes two arguments. If the first argument is less than the second the result will be 1, otherwise the result will be 0. In Actionscript, the operation would look something like this:

var result:int = arg1 < arg2 ? 1 : 0;

So, how is that helpful? Well, we can use that 1 or 0 in a multiplication operation to get either 0 or the original result. If we go back to our pseudocode and re-write it using a ternary statement similar to the one above, it could look like this:

var distancex = pixel.x - center.x;
distancex = distancex * distancex;
var distancey = pixel.y - center.y;
distancey = distancey * distancey;
var distance = distancex + distancey;
distance = Math.sqrt(distance);
var conditional = (distance < radius) ? 1 : 0;
// alpha is now either its original value or 0 if the distance is greater than the circle's radius
pixel.alpha = pixel.alpha * conditional;

We know we can do that in AGAL, so now our complete fragment shader can look like this:

sub ft0.x, 	v0.x, 	fc0.x
mul ft0.x, 	ft0.x, 	ft0.x
sub ft0.y, 	v0.y, 	fc0.y
mul ft0.y, 	ft0.y, 	ft0.y
add ft0.z, 	ft0.x, 	ft0.y
sqt ft0.z, 	ft0.z
tex ft1,	v0, 	fs0<2d, clamp, linear, mipnone>
slt ft0.w, 	ft0.z, 	fc0.z
mul ft1.w, 	ft1.w, 	ft0.w
mov oc, 	ft1

So, the main point here:

If you need to do some basic branching in AGAL, look for a way to use a “Set” operation and multiply the result with another value to get either a 0 or the original value.

If we plunk that AGAL into a Starling filter now, it might look like this:

 *	Copyright (c) 2012 Devon O. Wolfgang
 *	Permission is hereby granted, free of charge, to any person obtaining a copy
 *	of this software and associated documentation files (the "Software"), to deal
 *	in the Software without restriction, including without limitation the rights
 *	to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 *	copies of the Software, and to permit persons to whom the Software is
 *	furnished to do so, subject to the following conditions:
 *	The above copyright notice and this permission notice shall be included in
 *	all copies or substantial portions of the Software.
package starling.filters
	import flash.display3D.Context3D;
	import flash.display3D.Context3DBlendFactor;
	import flash.display3D.Context3DProgramType;
	import flash.display3D.Program3D;
	import starling.textures.Texture;
    public class CircleMaskFilter extends FragmentFilter
        private var mShaderProgram:Program3D;
		private var mVars:Vector.<Number> = new <Number>[1, 1, 1, 1];
		private var mCenterX:Number;
		private var mCenterY:Number;
		private var mRadius:Number;
        public function CircleMaskFilter(radius:Number = 100.0, cx:Number = 0.0, cy:Number = 0.0)
			mCenterX = cx;
			mCenterY = cy;
			mRadius = radius;
        public override function dispose():void
            if (mShaderProgram) mShaderProgram.dispose();
        protected override function createPrograms():void
            var fragmentProgramCode:String =
				"sub ft0.x, v0.x, 	fc0.x							\n" +
				"mul ft0.x, ft0.x, 	ft0.x							\n" +
				"sub ft0.y, v0.y, 	fc0.y							\n" +
				"mul ft0.y, ft0.y, 	ft0.y							\n" +
				"add ft0.z, ft0.x, 	ft0.y							\n" +
				"sqt ft0.x, ft0.z									\n" +
				"tex ft1, 	v0, 	fs0<2d, clamp, linear, mipnone>	\n" +
				"slt ft0.w, ft0.x, 	fc0.z							\n" +
				"mul ft1.w, ft1.w, 	ft0.w							\n" +
				"mov oc, 	ft1"
            mShaderProgram = assembleAgal(fragmentProgramCode);
        protected override function activate(pass:int, context:Context3D, texture:Texture):void
			mVars[0] = mCenterX / texture.width;
			mVars[1] = mCenterY / texture.height;
			mVars[2] = mRadius	/ ((texture.width + texture.height) * .5);
			context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, mVars, 1);
		public function set centerX(value:Number):void { mCenterX = value; }
		public function get centerX():Number { return mCenterX; }
		public function set centerY(value:Number):void { mCenterY = value; }
		public function get centerY():Number { return mCenterY; }
		public function set radius(value:Number):void { mRadius = value; }
		public function get radius():Number { return mRadius; }

If you try that filter though, you’ll notice something very odd. The circle is there all right, but instead of the outside being transparent, it has kind of a ghosted screen look. An interesting effect, maybe, but not what we wanted. That’s because when dealing with shaders in Stage3D, in order to get alpha values, you have to set the blend factors of the Context3D instance. If you check out the Adobe documentation, they even, very helpfully, tell you what blend factors to use. Since, in our case, we want to to use Alpha, back in the activate method of our CircleMaskFilter, just before we set the program of the context, set its blend factors like so:

context.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);

A little gotcha though – as soon as we’re done running our filter, we need to reset the context3D’s blend factors back to No Blending. Luckily, the good folks who put together the Starling framework made that simple enough. Just as there is an activate method in the base FragmentFilter, there is also a deactivate method. Simply override that method and reset the blend factors to no Blending:

override protected function deactivate(pass:int, context:Context3D, texture:Texture):void 
	context.setBlendFactors(Context3DBlendFactor.ONE, Context3DBlendFactor.ZERO);

Finally, you should get a result like this.

And that’s really all there is to it to handle alpha in a Starling filter. Once you get that down, you can do all sorts o’ stuff. With a second texture you can easily create complex masks. Or instead of finding the distance between two pixels as we just did, you can calculate the distance between two colors. Why (you ask)? Well, imagine you have a target color and you calculate the distance between it and each pixel in your texture. If the distance is below a give threshold, you set its alpha to 0, and suddenly you have a basic greenscreen application.

So, Chris, I hope that helps out. Better late than never….


  1. Ben W says:

    Nice one!

    Just a heads up though:

    ife Jump if source1 is equal to source2
    ine Jump if source1 is not equal to source2
    ifg Jump if source1 is greater or equal than source2
    ilt Jump if source1 is less than source2
    els Else
    eif Close an if or else block

    these have been added to AGAL(2?), woooo branching, making shaders like this even easier

  2. Devon O. says:

    Oh that’s fantastic!

    That’s funny. Just Friday I was reading through getting annoyed at all the opcodes there that weren’t actually used by AGAL.. I guess now they are.. I wish Adobe would keep up the documentation on AGAL – or maybe I just completely missed it..

    In any case, thanks for the heads up, Ben.

  3. Devon O. says:

    A quick follow up:

    the opcodes listed by Ben W above are available in Flash Player 11.6 (still on labs). There’s some info about them in the release notes:

    I’m sure I’ll write more about them when I’ve had some time to play…

RSS feed for comments on this post. / TrackBack URI

Leave a Reply

Devon O. Wolfgang

Technical Reviewer of “The Essential Guide to Flash CS4 AIR Development”

Contributing Author of “Flash AS3 for Interactive Agencies”

Senior Software Engineer at Meez

Starling Particle Editor


Droste / Escher Effect in AGAL

A New Year’s Eve bonus[…]


Ending the Year with Dynamic Multi-Pass Shaders in Starling

I think there’s some law which states you have to make at least two blog posts per year to legally say you maintain a blog[…]

Adobe ANEs

Base Mobile Tasks .ane and a Few Tips

Admittedly (and rather pathetically), it’s been so long since I’ve posted¬†anything on this blog[…]


Some Photographic Shaders

A couple weeks ago I was contacted and asked to develop some filters for Starling that could be used in a mobile camera app[…]


Animated Particles, Books, and Code – It’s a New Year

It’s a new year[…]


Logical Or Assignment Bug in ASC2

So, here’s something to keep an eye on if switching to AIR 3.8[…]

Starling Filter Collection

Starling Filter Collection

Building up a collection of filters for the Starling framework[…]

Starling Warp

Warp Filter for Starling

Another filter for the Starling framework[…]


Promises Promises

What it boils down to is handling asynchronous tasks […] in a clean, intelligent fashion […]


Starling ‘God Ray’ Filter

While cruising the internet today looking for interesting things to try out I ran across this fun little GPU Gem[…]

Alpha in Starling Filters and Basic Branching in AGAL

In plain English to create a circular mask we would want to do something like this[…]

Starling, Nape Physics, and PhysicsEditor

A look at using the PhysicsEditor tool for Nape and Starling […]

Playing With a Couple Game Ideas

Just a couple game fragments, really[…]

One More Filter For Starling

Another filter for the Starling framework[…]

Filters in Starling

Using and writing filters in the Starling framework[…]

Towards a Better Scratch

Perhaps it was too much Meat Beat Manifesto in the late 80’s […]