Current MS Students/Stacy O'Malley

From CSWiki

CS590

Project Prospectus

Splitting Image: Foreground Object Extraction and Background Image Inpainting

Contents 1 What I am planning to accomplish 1.1 Brief Project specification 1.2 High level architecture 2 Why this is academically interesting 2.1 Intended audience 2.2 Why this is MS-level work 3 Previous work 3.1 Platforms to be used and what I will add to each 3.2 Available platforms I am not using and why 3.3 Literature review 4 Anticipated challenge(s) 4.1 Challenge(s) 4.2 Anticipated approach to each challenge 5 What I bring to this work 5.1 My relevant background and experience (CS590) 5.2 What I find interesting about this work 5.3 How this work goes beyond my experience and course work

[edit] What I am planning to accomplish

[edit] Brief Project specification

I plan to implement a system to extract objects from an image and fill the gaps that remain in the image after the object is extracted. The input is an image on which the user will select a region of interest (ROI). The region of interest will be segmented in attempt to separate the foreground object from the background. The user will be permitted to select areas to be included (or not) as parts of the foreground object. The areas selected as the parts of the foreground object will be grouped together and extracted from the image, creating two new images, one of just the foreground object and the other of the region of interest with gaps left by the removal of the foreground object. The second image will then be processed to remove the gaps left by the foreground object with the goal of having the object removal as undetectable as possible.

This system is to eventually be integrated with Jose Amenero’s JTIP program.

As image segmentation was a large topic of focus in CS560, I do know that research in the area has yielded several methods for segmenting images including clustering, region growing, graph partitioning, histogram-based methods, neural networks, and other more interactive approaches such as Intelligent Scissors and more recently, Soft Scissors. Of course, I already know that simple algorithms to segment the image into a known number of objects will be insufficient for this project since a large part of the user experience of the JTIP program requires that the result of the segmentation spoil the illusion as little as possible. Being that the JTIP program is so highly interactive (the user is navigating through the image), an interactive image segmentation approach seems the best direction as long as it is easy to use. Soft Scissors is a recent development in interactive segmentation that is easy to use and yields excellent results. Currently, this is the algorithm I am planning on implementing though I have not studied it enough yet to know how at this time.

Image inpainting is a new area of study for me as it was only briefly discussed in CS560. In the short time I have spent researching it, I have learned of a few methods for solving the problem of filling in missing information in images. Some interactive methods require the user to select pieces of the image (color or texture pieces) to use to fill in the empty spaces. This approach seems tedious and frustrating to use with results that vary according to user skill and patience. Other techniques involve filling in gaps in one image of a series by using information from the other images in the series that come before and after it. Though it produces nice results, this technique is mostly used in film restoration and therefore it is obviously not appropriate for the problem I am trying to solve. For this problem, I feel that a non-interactive approach would yield better user experience and results. The method of filling missing information in the Image Inpainting paper referenced below involves processing each pixel in the "gap" of the image starting at the boundary and working inward, propagating information along lines of constant intensity. The algorithm makes no requirements on the shape of the area to be filled or the type of images to be used and the results in the paper are impressive. Currently, this is the algorithm I am planning on implementing though I have not studied it enough yet to know how at this time.

[edit] High level architecture

Note: The shaded boxes denote existing architecture.

[edit] Why this is academically interesting

[edit] Intended audience

The intended audience is anyone interested in Computer Vision, image processing and/or JTIP.

[edit] Why this is MS-level work

This project involves the research and implementation of non-trivial algorithms based on complex mathematics. Additionally, segmentation and gap-filling (or image inpainting) are problems in the Computer Vision field for which there is no general solution, that is, there is not one best way to solve the problem. Part of my project involves studying the current research (the algortithms, types of images used and the results obtained) to select a method to implement for my project that will solve the problem in the best way for my purposes and be achievable in a reasonable amount of time.

[edit] Previous work

[edit] Platforms to be used and what I will add to each

• OpenCV is an open source library of functions related to Computer Vision. This library is well known and widely used in the Computer Vision field; it has been used on such projects as SwisTrack and Stanley, Stanford University's autonomous vehicle. Being that OpenCV is open source, there is potential to contribute some of my code to the library.

[edit] Available platforms I am not using and why

There are a few other Computer Vision related platforms I am aware of but have not yet researched fully. Therefore, at this time I am not committed to using or not using the following:

• ITK
• VXL
• CLIP
• Blepo

[edit] Literature review

• Bertalmio, M., Sapiro, G., Caselles, V. and Ballester C., "Image Inpainting," SIGGRAPH 2000, pp. 417-424.
• Horry, Y., Anjyo, K. and Arai, K., "Tour Into Picture," SIGGRAPH 97 Proceedings, pp. 225-233.
• Ohta, Y., Kanade, T. and Sakai, T., "Color Information for Region Segmentation," Computer Graphics and Image Processing, Vol. 13, No. 3, July 1980, pp. 222-241.
• Wang, J. and Cohen, M., "Optimized Color Sampling for Robust Matting," Proceedings of IEEE CVPR, 2007.
• Wang, J., Agrawala, M. and Cohen, M., "Soft Scissors: An Interactive Tool for Realtime High Quality Matting," ACM Transactions on Graphics, Vol. 26, No. 3, July 2007.

[edit] Anticipated challenge(s)

[edit] Challenge(s)

Some challenges I expect to face are:

• Understanding and implementing the complex mathematics involved in an efficient manner.
• Re-learning basic C/C++ programming skills and acquiring new (more advanced) ones.
• Developing a clear understanding of the technical challenges involved in each problem and the pros and cons of proposed solutions.
• Working with large amounts of data (multi-channeled images) in an efficient manner.

[edit] Anticipated approach to each challenge

To meet each challenge I plan to:

• Practice.
• Research papers published on these topics and discuss issues with my advisor.
• Work step by step, solving smaller issues individually before becoming buried beneath slow sloppy code with memory leaks.

[edit] What I bring to this work

[edit] My relevant background and experience (CS590)

• CS350 Computer Graphics - This class provided my introduction to Computer Graphics and OpenGL which may prove useful as background for understanding the general technology behind the JTIP system.

• CS560 Advanced Artificial Intelligence (Computer Vision) - This class provided my introduction to Computer Vision, image processing and the OpenCV library; here I learned about issues faced in the field (i.e. image segmentation and object recognition) and wrote programs to implement simple algorithms devised to solve some of these issues.

[edit] What I find interesting about this work

I find Computer Vision interesting because it is still a relatively new field of study that can be applied to a diverse set of disciplines. Computer Vision problems exist in such areas as artificial intelligence, machine learning, robotics, mathematics, neurobiology, image processing and others. Given that I am a person with many varied skills and interests I feel that the Computer Vision field and I are a good match.

Particularly, I find image processing interesting because I have long held an interest in photography and the power images have to quickly convey information, tell a story and evoke strong emotional responses all on their own.

[edit] How this work goes beyond my experience and course work

This work goes beyond my experience and coursework due to the amount of research and complexity of algorithms and mathematics necessary to solve the problems. This work will incorporate a great deal of my past life and programming experience and provide a means for me to contribute to a current research field.