One common algorithm to perform Face Morphing requires fitting a polygonal mesh on top of all input face images. This must be done in a way such that each vertex converge to the same physical spot of all faces – such as the eye contour, nose tip, chin, etc. Having such a mesh aligned to a face can also be useful for other purposes, from caricaturing to beautification, by performing non-rigid modifications to the original face.
In this post, I’ll describe how I implemented a simple face distortion tool using only JavaScript and HTML5.
Suppose you want to find a Homography transform that maps a set of 2D coordinates to another arbitrary set of 2D coordinates. This problem has many applications, such as image registration based on keypoint matching. This post will describe a few steps to make sure the homography estimation is numerically stable and can be performed using 32bit floating point coordinates.
Back in 2014, I wrote an article on how to implement 2D omnidirectional shadow mapping by only using ThreeJS and standard WebGL resources. That technique consisted of modeling each light object with three perspective cameras with a field of view of 120⁰ each. Similarly to the classical shadow mapping algorithm, the scene depth was projected into these cameras, and later I would render the scene, using the shadow maps to check whether or not each fragment should be lit.
That technique had two major downsides. Firstly, the decomposition of each light into three cameras meant the need for three render calls for each light, which is not ideal in terms of performance. Secondly, the projection planes of these cameras are combined into one equilateral triangle centered at the light position. Therefore, if a light is positioned in a place where its triangle intersects with an obstacle, then its field of view will be limited to the part of this triangle that is outside of the obstacle.
The previous method was far more susceptible to reduction of the light FOV when they were positioned close to an obstacle.
If we can rely on the extension EXT_frag_depth (which is was turned into a standard feature of WebGL2), then it is possible to improve this technique both in terms of quality and performance. Furthermore, by fixing the maximum amount of lights, it is also possible to greatly increase rendering performance by avoiding multiple render passes.
The purpose of this article is to describe both improvements in detail. If you are not familiar with this technique, you may find the previous article a good source for other details.
Aerial view of Belo Horizonte. The frames were taken from Google Maps.
The problem of image registration consists of aligning two images taken from different perspectives in order to obtain a larger visual representation of some large structure. For instance, one can stitch several overlapping aerial pictures to obtain the map of an entire city. Another example is the creation of panoramic pictures, which consists of stitching pictures taken by rotating the camera. The registration problem is also present in the medical fields. It is required to register medical imagery in order to accurately perform diagnostics as well as detect changes during several types of treatments.
With such a broad spectrum of applications, one can expect the registration problem to be presented with countless particularities. The signals being registered can have an arbitrary amount of dimensions; they might not even contain the same representation of the structure being mapped in the first place. The data from an MRI (representing soft tissues) can be registered with CT scans (more sensitive to high density objects); one may also attempt to register satellite imagery with cartographic maps.
The objective of this article is to gently introduce how the registration problem can be addressed with a non-linear optimization framework, from the Taylor series until the formulation of the Homography Jacobian and Hessian. In this article, we will focus on rigid registration, although the concepts herein can also be applied to the non-rigid case.
Algorithmic Pensieve 