Environment Mapping with ARCore and Unity3D

So here’s a little trick for mapping a device camera image onto a 3D model when using Unity3D and Google’s ARCore in order to create some realistic environment reflections and further blend augmented reality with plain old reality. I originally got the idea from this Unity blog post. Before digging into John Sietsma’s solution, though, I implemented my own and figured there were big enough differences to warrant describing my own approach.

Before jumping into some code, here’s a little look at the effect in action. You can see how the floating torus reflects both the color of the candle as well as my hand waving around it. In this video, the reflection amount is set to .85 and the blur amount to 7 (those properties will be made clear later):

Overview

From a high perspective, the effect is achieved by taking the camera image, blurring it, then mapping it onto a 3d model using radial coordinates based on the world normals. Is the effect perfect? Not at all. Really the entire thing is faked. First, because we’re using the camera as an environment map, it’s entirely possible to see an item behind the 3d model reflected in front of it. Also, the UV’s of the camera image will never match the model UV’s – and, even if they somehow did, the camera image texture is not tileable so there will always be some fairly nasty visible seams. That said, though, if using a relatively complex model (the drawbacks are especially apparent on simple primitives), adding a decent blur, and blending the reflections with the original model texture, the results are, as we used to say back when I worked for the government, good enough for government work.

The basic steps, then, are these:

1. Capture the camera image into a Texture2D instance
2. Blit that texture into a RenderTexture instance using a blur shader
3. Apply that blurred RenderTexture to the 3d object’s material via shader

Shaders

The first shader we’ll need is the shader applied to 3D model that is based on a basic surface shader using a BlinnPhong lighting model for nice highlights and reflection via Emission. This is really the core of the effect and the shader that does all the magic. This shader uses a Map Texture property (_MapTex), samples it from UV’s created by the ‘radialized’ world normal. That sample is used for the output Emission and becomes the reflection. This shader also does a bit of jiggery-pokery to adjust the map UV’s according to some adjusted UV properties. The reason for this is because the image captured from the device camera may be oriented in a way that doesn’t make sense for our reflections. You can see another example of this in the Google ARCore computer vision example project  EdgeDetectionBackground.shader file (which is where I, admittedly, yanked a good deal of implementation). This shader also contains a _ReflectionAmount property which ranges from 0 to 1. Obviously 0 will have no reflections, whereas 1 will show only reflections. As mentioned above, the example in the video uses a _ReflectionAmount of .85.

The only other necessary shader is one that creates blurs. The implementation I went with is kind of interesting. Ordinarily, to create blurs, a shader samples neighboring pixels then averages the output by dividing by the number of samples taken. Well, this one does exactly that but does it in a nice neat succinct fashion including a nifty little function that (roughly) calculates Gaussian distribution. I found this algorithm in a Unity forum post and it is, in turn, based on a Shadertoy post. So I’ll leave it to the lawyers to decide if it’s usable or not – but I’ve added my own MIT license anyway. Notice this shader has a _BlurAmount property to increase/decrease the blur effect. As mentioned, the example video uses a blur amount of 7.

Code

Finally we get to some actual set up and code. First, in order to get the device camera image, I used the Google TextureReader component included the ARCore computer vision example and attached it to a ‘controller’ game object in my Unity3D scene. Because these reflections will be blurred and do not need to be very detailed, it’s not necessary to use a large image width and height in the Inspector panel settings. I went with 800×480, myself, but you can play around with that. I also used ‘Cover Full Viewport’ for the Image Sample Mode property and ‘Image Format Color’ for the Image Format property.

In our controller code, we’ll need to cache an instance of that TextureReader and add a listener for the OnImageAvailableCallback event. Something like this:

Now, the OnImageAvailable method is really the meat and potatoes of this effect. This is where we draw the captured image into a cached texture, blur the image using a material utilizing the above BlurShader, then apply it to our AR model’s material (using the above ARCoreEnvironmentMap shader). I wanted to keep this method neat so wrote it in a (kinda, sorta) functional way. I’ll include the extension functions I used down below. In any case, the OnImageAvailable method looks like this:

And, aside from some helper stuff, that is really it.

Helper Stuff

In order to get the updated UV coordinates of the camera image based on device orientation, I basically yanked a number of methods from the ARCore computer vision example and placed them into an ARCoreHelper class (you can see it used above in the line ARCoreHelper.GetUVsFromTexture). The ARCoreHelper class looks like this:

And finally here are some generic extension methods I’ve been slowly building up for my own use that may come in handy:

And that is that. I hope this helps some folks out. Let me know, if you’d like.