This blog entry describes the project realized for the second module of the MA 2D for VFX, a clean-up project and a CG comp, as well as the different challenges faced during the process and the choice of techniques to solve them and their effectiveness.
CHOOSING THE SHOT
I choose this shot after the 3D presentation the students had at the end of their 12 weeks course. During that presentation, the teacher discussed with the class the different shots that they were working on, and made some recommendations for improvement.
I was specially interested in one of the works, it was an alley with many different 3D elements and lights, but it was far from being completed. After talking with the 3D artist, I decided to comp his shot, doing a clean-up of the original footage, that will consist in removing 3 motorbikes and doing integration of 1 or 2 of the CG elements. With this idea, we had a project to work on in the short term, and we could also consider to do a more complicated version in the future, adding more elements, lightning, and to maybe change the environment from daytime to night, as an option for my final project on the Module 5. His idea and work in progress at that moment can be seen below:
As there were a lot of different CG elements, we decided to choose 1 or 2 that he would be able to finish in the following weeks: the bin that can be seen at the left, and potentially the pipe that can be seen in the right. If decided to transform it into a night time shot, the pipe could have been changed for a street light, illuminating the right side of the shot.
After deciding the idea, I started to work in the footage (See below).
The footage that I received was undistorted, and I started to study it and the different techniques that I will use for every part, while waited to received the original footage. Unfortunately, due to different problems, I wasn’t able to receive the original footage or those CG elements.
Still with this problems, I decided to continue with the project and this version of the shot, trying to accomplish the original idea: doing a clean-up (removing the 3 motorcycles) and adding a CG bin into it. The reason for doing this, is that in the case of having to do this comp in a working environment, as part of a bigger project, I would have to deliver it regardless of any possible problems.
CLEAN-UP IN 2D
During the beginning of the project, I was using only 2D techniques, and moved to projections later. Still, for most of the project I used 2D techniques, the main reason in doing so, was that I was not totally comfortable yet with 2D techniques, and I wanted to improve in them and get comfortable with the basics techniques before starting with projections.
The areas that I finished using 2D techniques can be seen below:
For those parts, I did a patch, that worked on a specific frame (reference frame), recreating the walls or floor to cover the motorbikes using in most of the cases, Planar Tracker.
As the VFX Compositor Conrad Olson (Conrad Olson IMDB), recommends:
“In a rig removal shot you ideally want to clean up one frame and then use that frame as a patch to fix the other frames in the shot. You can create more than one patch for a shot and mix between the clean frames if the shot changes too much for one patch too work. There are some situations that would need so many different patches you’re better off just breaking out the clone tool and painting up each frame. Usually you can combine the two techniques.” (Conrad Olson, 2011).
In this shot I did not use frame by frame painting in any case, however, it was my idea for two small areas, the second window on the left (which is slightly covered by the first motorcycle). and for the floor on the right side of the footage, below the graffiti, which can seen in just 4 frames. However, I was still able to use Planar Tracker on the right side, and projections on the left window.
When approaching how to do the patches, I started to track different points with different Trackers, creating an average between them, however, soon I realized that what was working better was the Planar Tracker.
“The PlanarTracker follows flat surfaces where most points on the surface lie in the same plane. Things like walls, ceilings, and sides of cars are good examples of planar surfaces (…) the more planar the surface, the better. (…) The important thing to remember is that planar tracking tracks regions as opposed to small image patches. This means you get a much more stable and accurate track compared to traditional 2D pattern trackers. For things like inserting images and set extension, it is nearly always faster and easier to use a planar tracker as opposed to some sort of projection in a camera solve, for example”. (The Foundry, 2011).
The main problem that I had while doing the project, it was due to the distortion of the footage, as the undistortion was not correct, and was not equal on the left edges than in the right edges, and the image is distorted unequally in the center of the image as well.
This caused the need to use different Planar Trackers for almost every part of the footage with the need of adjusting it depending on the movement, being this different in the center of the shot than closer to the edges, and more distorted on the left corner than on the right one. For this, I was often adding Keys on my Corner Pins during the different movements of the patches.
CLEAN UP IN 3D
Even that most of the clean-up was made with 2D techniques, some areas were made with Projections, specially the reflections in the windows. The areas where I used 3D Projections are:
In order to do the projections, the first part was to track the camera movement. This was done on the side of the script, to be sure that I was able to have a back up of it, if it was needed needed, and to keep the 2D and 3D techniques separate.
The part where I spent more time, was cleaning the windows of the second motorcycle in the left, and recreating the reflections on them:
I used Time Offset to use the frames of the reflections that were cleaned, and invested more time in the 4th window, with 3 different projections:
1- one projection for the first frames
2 – another projection for the second part of the frames
3 – a third projection using a partial real time reflection of the same window to merge the 2 previous reflections.
Another tricky part, was covering the mirror of the motorcycle on the right side. For it, I started doing a Paint work that would be projected on a Card on the floor:
After that, I tried to do a paintwork on the bricks, and use later Planar Tracker. However, a good result was difficult to achieve as the wall on that side is not really a plane. Instead, I used Geo, trying to recreate the shapes of those corners in the bricks with different cubes. But, as the bricks on that part were not straight either, I found really hard to do those curves properly. As a result, a decided to do 3 different patches, one for every face of the bricks that I needed to cover, Merge them, and projected into a card, creating roughly just the corner of the bricks on it with Model Builder, and adjust it with corner pins later:
RENDER FOR MAYA
In order to have a CG element, I exported my Tracked Camera and Point Cloud into a fbx file. I did 3 different versions so the 3D artist could find it easier to add the element into the scene. One version was just the tracked camera and the point cloud, another version was with a card on the scene, on the place where the floor is, and the third version had a Cube on the place where the bin would be located.
The 3D artist found it hard to work from a scene exported from Nuke, what created multiple problems with the different versions of the CG bin.
On the first CG, the first problem that I had was that the format of the passes of the bin were 1066×565 while I was working in 4K. Even once reformat at 2K, the result was this:
The position of the bin on the scene wasn’t the expected either, and none of the passes had an alpha. This last thing it could have being easily fixed, however, the movement was also wrong. This is the beauty as it was given (before Merging it with the background).
And this is the Diffuse pass Merged with the Background (with the Background reformat to the pass format):
As working in 4K resulted in a really slow process in Maya for the 3D artist, for the second CG, I exported a version of my footage in 2K, before the cleanup.
After doing another Render, in which the movement was wrong, the 3D artist used 3D Equalizer to track the camera in the shot. After 2 attempts, this was the result (scaled):
I assume that the problems in tracking the shot are due to the fact that the shot has a wrong undistortion work on it. However, I decided to work with this version.
I did some work in the alpha and RGB to fill some areas that were not covered. (See below)
To fix the movement, I tried using a Planar Tracker of the floor, as well as doing a point tracker of the bin, and make a match-move to another point tracked in the floor. None of this techniques worked, so I projected the bin into a cube using model builder.
After doing this, I decided not to use the shadow passes that I was given, and to do my own shadow instead. To do this, I used a corner pin and blur to create a shadow, I tracked the movement of the bin, and I did a match-move to the shadow. In order to obtain a better result, I did 2 shadows, with one of them being less blurry and closer to the bin, in order to recreate the contact shadow.
With everything in place, I did the grading work, adjusting everything by viewing the channels by separate before doing any final adjustment viewing RGB.
After painting the RGB, adjusting the movement, creating a shadow, and grading, this is the result (scaled):
SCRIPT / ORGANIZATION
The fact of having such a clear script, made me easy to localize the different patches. I kept the trackers at the top, out of the main clean-up script, and the CG, pointcloud, camera render, and projections, on the right, that way it did not got mixed with the 2D clean-up and main script. The clean-up script has 3 main backdrops, two on the left, for the 2 bikes on the left of the footage, and one on the right, for the bike on the right side of the street. I used the a red/orange colour for the projections, as well as for the merge nodes with those projections on the main pipeline.
For the CG comp, I used a different script, with a render of the finished Clean-up Comp, to make the script lighter, faster, and more easy to identify and work on.
The final Comp of both the clean-up and the CG comping can be seen below:
- Probably this project would have been easier to finish if it was approach almost totally with projections, using cards for most of the parts, and different geo shapes to clean-up of the motorcycle on the right side. Even that this could have been a slower process at the beginning, ones the cards and geo are on the right position, the rest would have been faster.
- The CG comping was done in 2K, as a result, all the renders of the Rig Removal project, were done in 2K as well. However, during the Rig Removal project I worked in 4K, what means that my script was more heavy and slow, causing Nuke to crash often. In this case, if i would have chosen to work in 2K from the beginning, would have probably avoided the crashes in Nuke, to a certain extension.
- For the same reason, if I would have worked with renders in my cleanup script (before doing projections by example, or after every motorcycle cleanup), the script would have been lighter, and Nuke would have worked faster as there would have been less nodes to read.
Conrad Olson. Frame-by-frame Painting. January 6, 2011. http://conradolson.com/frame-by-frame-painting
The Foundry. Introduction to Planar Tracker Tutorial. Dan Ring, 2011. https://help.thefoundry.co.uk/nuke/content/tutorials/planar_tracking.html