Jan 25, 2023 • by Benjamin Gallois

Automation with Intent: A Practical Philosophy for Artistic Image Processing

Introduction

After collaborating with several artists in iris art and other related photography niches, I realized that precise, fully automated tools are not always the optimal solution for these creative applications.

Initially, I developed complete solutions that automated much of the tedious, manual work typically done using standard image processing tools like GIMP or Photoshop. However, the user feedback was often negative—and, in many cases, not entirely justified. For instance, it is well known in image processing that a small subset of difficult images in any dataset may require special treatment. Yet users frequently focused on these exceptions to argue that the tool was fundamentally flawed.

Another common critique involved the detection of nearly imperceptible defects that the tool either introduced or failed to correct. Upon closer inspection, these flaws were also found in results generated through manual, careful work, the very process the tool was designed to streamline.

After reflecting on this feedback and gaining more experience working alongside artists, I came to a deeper understanding: when applied to tasks traditionally done by hand, automation can be perceived as erasing or devaluing the artist's contribution. This led me to a new philosophy when designing tools for creative domains:

By adhering to these principles, I collaborated more successfully with artists and art galleries in various photographic niches. In the following section, I will present a tool I developed to assist with image standardization—a task that is both crucial and notoriously difficult to perform consistently by hand.

Iris Art

Introduction

Iris art is a niche in photography that involves taking a close-up picture of the human eye to produce an outstanding view of the iris, enhancing the colors and textures unique to each person. The artist generally produces an image following either a multi-image or single-image approach. The muti-images approach starts with the photographer taking several pictures of the eye with subtle changes in lighting and eyelid position. The artist combines these images to remove eyelid occlusions, glares, and eyelash reflections and recover the entire iris with high fidelity. With the single-image approach, the photographer takes only one image. Then, the artist uses inpainting techniques to recover the iris. These two methods produce outstanding results, but each has its challenges.

The first challenge is to get precise iris segmentation to single out the iris (colored part) and eliminate the sclera (white part). To produce an esthetic image, the artist has to round the iris and center the pupil (black part). He also reduces the pupil to keep the same ratio between iris and pupil in all the pictures, facilitating subsequent iris composition. Finally, he must remove lighting glare, eyelid occlusions, and eyelash reflections to recover the entire iris.

Multi-Image

The multi-image approach will solve this challenge by using image combinations. Each image has to be precisely adjusted according to the others; the eye being spherical, a non-rigid registration has to be performed to get a pixel accuracy alignment. The precise color tones and brightness must be matched between images to produce a coherent result. Then, the defects must be detected, and images must be combined to remove them. Non-rigid registration is technically complicated, as it requires working with high-resolution images in a reasonable amount of time. Automatically finding an algorithm to combine images to eliminate all the defects is tricky; for this reason, the multi-image method is not the most common in iris art studios.

Single-Image

The single-image approach uses image inpainting to correct all defects, basically copying other parts of the iris and seamless-pasting them to remove defects and recover (and imagine) missing parts of the iris. It is easier and faster than the multi-image approach but at the expense of iris fidelity. This technique is the most encountered in iris art studios mostly because it is easy to implement and does not necessitate any deep knowledge in image processing, and can be performed on regular software like Gimp or Photoshop.

Rounding

After reconstructing the iris using either of these methods, it is necessary to round the iris and pupil to achieve an aesthetic result and, most importantly, reduce the pupil's size to allow a good view of the iris. It's important to note that the final quality of the artwork depends on the photographer because if the picture is taken with a very dilated pupil, the iris will be mostly invisible, and recovering the iris pattern will be very difficult and inaccurate.

DeepIris

After working with several art galleries to automatize the analysis process, both for multi- and single-image, using classical computer vision, I developed a new approach using deep learning to speed up and simplify the iris segmentation and rounding, as well as the pupil reduction and centering, allowing high fidelity iris reconstruction.

The most common technique in iris art studios is to approximate the iris and pupil as perfect circles, vaguely detect them as circles, and then shrink and crop them. This technique is fast but introduces several artifacts in the final result. The most common one is that some parts of the iris will be missing, mostly some edges and parts close to the pupil. These missing parts will produce art that seems asymmetrical and unnatural. The same can be said when the pupil is shrunk using an approximated iris and pupil centers; the intricate iris pattern will be deformed unnaturally.

To solve this problem, I developed a suite of tools that automatically detects and segments the iris and pupil with great accuracy using deep learning. The algorithm can then deform and preserve patterns and symmetry. It can work with a wide variety of images, from whole-face pictures to eye close-ups, at any image resolution and the size of the iris compared to the image size. These tools can be (and have been) tailored for iris art studios using transfer learning on proprietary datasets and specific requirements to be integrated into custom software or analysis pipelines.

At its core, the tool integrates two segmentation deep learning models to segment the iris and the pupil with pixel accuracy. From this segmentation, it can find the precise contours of the iris and pupil that will not be perfect circles and find their true centers that will not be the same. Using the tool detailed in this other blog post, we can then morph the image to:

In the end, the produced eye is perfectly symmetrical and balanced, and the intricate iris pattern is preserved, making an outstanding and aesthetically pleasing artwork. As the relative ratio of iris/pupil is standardized, several eyes can be combined to produce artwork.