Current MS Students/Richard Lim

From CSWiki

Diary | Definitions and Acronyms

Project: Non-Photorealistic Renderer

Contents 1 What I am planning to accomplish 1.1 Brief Project specification 1.2 Brief Project details 1.3 High level architecture 1.3.1 Overview 1.3.1.1 Figure 1 1.3.1.2 Figure 2 1.3.1.3 Figure 3 1.3.2 Details 1.3.3 Definitions and Acronyms 2 Why this is academically interesting 2.1 Intended audience 2.2 Why this is MS-level work 3 Previous work 3.1 Platforms to be used 4 Anticipated challenge(s) 4.1 Challenge(s) 4.2 Anticipated approach to each challenge 4.3 Literature review 5 What I bring to this work 5.1 My relevant background and experience (CS 590) 5.2 What I find interesting about this work 5.3 How this work goes beyond my experience and course work

[edit] What I am planning to accomplish

[edit] Brief Project specification

With the improvements in OpenGL language, OpenGL does not have to used like a state machine; OpenGL can be programmed at the graphic driver level to develop custom rendering context. Using the OpenGL Shader Language, this project will create a non-photorealistic render engine that allows custom shading styles, such as cross-hatched patterns or custom half-tone styling. This project overrides the pipeline given workflow with a non-photorealistic filter.

[edit] Brief Project details

Standing on the shoulder of giants, whatever, I still cannot see through the eyes of the genius. Elements of narrative bring depth and characterization to games as the development practice shifts from flat files to dimensional emoting with the characters. The role and importance of the visualization of the conceptual nature of the media played on the imagination of player. As the technology and the increased cycles became available, the development has brought many genres to a growing age range of consumers. What has not been pushed is the artistic nature of vision. What if Van Gogh was a video game designer; what would world would he develop; what would the “Starry Night” be like? The artistic nature of interpreting the nature of the world from particles to objects, video game development next logical step is to explore building artistic entertainment. To take one-step nearer to the conceptual option artistic interactive video entertainment a need for non-photorealistic renderer becomes clear.

OpenGL® is a software interface for graphic hardware; OpenGL is intended to take advantage of the computer display hardware for 2D/3D graphic manipulation. If the hardware does not support OpenGL feature, the software-only implementations of OpenGL will take over. Having the software implementation means that your program can still operate on computer systems that do not have 3D accelerated graphics cards ensuring that the application is widely distributable. With the hardware acceleration and a fast microprocessors, 3D graphics are moving beyond typical components use for consumer and business applications, such as games and scientific applications.

OpenGL® is highly flexible, but it is still a static interface for putting graphics on the display device. OpenGL is composed from sets of geometric sequential operations for image data manipulations on graphics hardware. The application may work within the traditional rendering model, but if need to control and define area lights, performing light calculations per-vertex, or facing one of the many rendering limitations of OpenGL rendering model, a custom OpenGL shader could be implemented.

OpenGL® shading language is a high-level procedural language; it extends the OpenGL API exposing the processing pipeline. Developers can manipulate the underlying graphics hardware to create a greater range of rendering effects. Dedicating the graphic hardware for the manipulation of the vertex gives more visual power while off-loading the need of the CPU for other tasks, such as artificial intelligence or a physic engine. To build a non-photorealistic render engine, OpenGL could not be used alone. OpenGL would override the shading with its lighting and blending interpretations; using OpenGL shader, the non-photorealistic render engine could calculate the lighting and shadowing needs of the image within the graphic hardware.

[edit] High level architecture

[edit] Overview

The user has vertex information defining some typical rendering of content desired, such as a teapot. The teapot goes through the graphic pipeline and the teapot will get rasterized. However, the user may want to render in a non-photorealistic style for the teapot. To accomplish this, the user links to this project, non-photorealistic renderer. The non-photorealistic render contains the shader language needed to modify the vertex information provided by the user application. The non-photorealistic render overrides the pipeline workflow to implement its own interpretation of the vertex information, referring to the Shader compilation and execution of the GLSL. The resulting modification of the vertex data, the teapot, will be passed to the OpenGL hardware rasterization.

[edit] Figure 1

The traditional render, Figure 1: OpenGL Pipeline Architecture, depicts the graphic pipeline for all versions of OpenGL, refer to the Definitions and Acronyms section. Figure 1 the user passes vertices and a rastered image comes out. The project hides the GLSL code needed for the non-photorealistic rendering. The user would need to just enable the render style and define a custom half-tone texture, refer to the Details.

[edit] Figure 2

Figure 2: Shader Flow is the three boxes Pre-vertex operations, rasterization, and Per-Fragment operations from Figure 1. The ARB designed the blue squares as the functions that developers need to override to make custom rendering content and the ARB designed the yellow squares to remain the same static behavior from the previous versions. What we see here is the new programmable model still interacts with the same static model. The project, the blue squares written in GLSL, resolves the vertices needs by solving the coloring and edge finding for the blue sections resulting in the image seen. The project needs to override all three of these defined functions to create the non-photorealistic image.

[edit] Figure 3

Figure 3: Shader compilation and execution of GLSL is the structure of project. This depicts the execution flow of the implementations within the OpenGL 2.x driver. The project is comprised of three filters, the GLSL code that takes the user vertex, solve the edges, coloring, lighting, transform, and rasterize into a non-photorealistic image. This image is the overall project that will hide the GLSL from the user and override all three of the filter steps in the pipeline.

[edit] Details

The non-photorealistic render requires the developer to solve the lighting interpretations and defining the edges, since the developer has overridden the static workflow from Figure 1. If the developer wants to create a custom look and feel, the developer has to take the responsibility of managing the vertex, geometry, and/or fragments. This management is the programmed workflow of the pipeline with the code written by this project. Solving the shading needs of this project is complex, such as silhouette edge detection and crease edge detection needed for the egde-based inking style. GLSL offers a way beyond state based rendering, but the development of the rendering, math, and algorithms required to accomplish this style is on the shoulders of the developer. This project shall present a method to render in this style by solving these requirements.

This project will provide a few APIs to construct a custom non-photorealistic rendering. An interface to enable/disable the non-photorealistic rendering; the enabling interface shall turn on or off the feature. An optional interface to define a custom half-tone brush style; the half-tone interface gives the developer an optional custom look. If the interface is not defined, it will default to a general cel-based brushing style. The plan is to have the user define a texture using the standard OpenGL calls. The project shall provide a simple interface to define the custom half-tone indice, the index name of the texture. The project plugins, the blue parts, will take the texture information and into the calculations for the rasterasteration.

The non-photorealistic renderer shall be built on OpenGL and using the OpenGL vertex shaders, leveraging on the Figure 2: Shader Flow. The OpenGL Shading Language is a high-level procedural language; the language can be used in applications to design and implement behavior within the parallel graphics hardware. The language constructs provide natural algorithms; the OpenGL Shading Language is designed to support scalar, vector, and matrix types; structures and arrays; sampler types that access textures; data type qualifiers that define shader input and output; constructors for initialization and type conversion; and operators and flow-control statements, referring to Figure 3.

• Shader execution, referring to Figure 2: Shader compilation and execution of GLSL

1. Create shader object(s).
2. Load shader object(s).
3. Compile source.
4. Create program object.
5. Attach shader object(s).
6. Link program object.
7. Use program object.
8. Attach shader to application.

[edit] Definitions and Acronyms

Term	Meaning
2D	Two dimensions
3D	Three dimensions
ACM	Association for Computing Machinery
API	Application programming interface
ARB	Architecture Review Board
CPU	Central processing unit
Frame Buffer	Video output device driving video display from a memory buffer containing a complete frame of data. The information in the buffer consists of color values for every vertex which may or may not be mappable on the screen.
GLEW	OpenGL Extension Wrangler library
GLUT	OpenGL Utility Toolkit
GLSL	Open Graphics Library Shading Language
GPU	Graphics processing unit
GUI	Graphical User Interface
OpenGL	Open Graphics Library
Per-Fragment Operations	The process of converting fragments to pixels on the frame buffer. The texel generation begins with generating texture elements from the texture memory applied to each fragment. Next step, the fog calculations are applied. Follow begins the testing steps, the fragments shall be tested for scissor, alpha, stencil, and depth. Finally filter steps, the fragments shall be blended, dithered, masking. The results of the actual pixel data are stored in frame buffer.
Per-Vertex Operations	Each vertex and normal coordinates are transformed within the model-view matrix where the lighting calculation per vertex is performed using the transformed vertex and normal data.
Pictogram	Symbol representing a concept, object, activity, place, or event by illustration.
Pipeline	State-based vertex processing workflow for graphic processing.
Pixel Operations	Pixels from memory are unpacked. Next step scaling, data is scaled, biased, and processed into the pixel map.
Programmable Fragment Processor	Performs traditional graphics operations on interpolated values, texture access, texture application, fog, and color sum.
Programmable Geometry Processor	Access the geometry, lines, triangles, quads, etc., to create and modify new geometry.
Programmable Vertex Processor	Programmable unit that operates on incoming vertex attributes, such as position, color, texture coordinates, and so on. The vertex processor is intended to perform traditional graphics operations such as vertex transformation, normal transformation/normalization, texture coordinate generation, and texture coordinate transformation.
Raterization	Conversion of geometric and pixel data into fragments, the fragments are arrays containing color, depth, line width, point size, and anti-aliasing calculations. If shading mode fill is enabled, the interior pixels of polygon shall be filled at this stage.
SIGGRAPH	Special Interest Group on Graphics and Interactive Techniques
Texel	Texture element, also known as a texture pixel, is a fundamental unit of texture space used in computer graphics. Textures are derived from arrays of texels.
Texture Memory	Type of digital storage for texture data needed for GPU.

[edit] Why this is academically interesting

[edit] Intended audience

This render engine is a proof of concept that demonstrates the possibility of image rendering content. With the investment of developing the filters necessary for anyone who is interested in developing custom rendering content. Developers can implement and override the visual content beyond state settings and texture fakes.

[edit] Why this is MS-level work

This project implements a graphics hardware solution for non-photorealistic rendering by replacing the static functionality with programmability, such as vertex processing and fragment processing, refer to the architecture. OpenGL Shading Language has been designed to allow application developers overriding capabilities of the processing that occurs at those programmable points of the OpenGL pipeline.

[edit] Previous work

[edit] Platforms to be used

• FLTK is a cross-platform C++ GUI toolkit for UNIX®/Linux® (X11), Microsoft® Windows®, and MacOS® X and it provides modern GUI functionality without the bloat and supports 3D graphics via OpenGL® and its built-in GLUT emulation. (note: used as a test harness for the non-photorealistic renderer, it is not part of the project)
• GLEW is a cross-platform open-source C/C++ extension loading library where it provides efficient run-time mechanisms for determining which OpenGL® extensions are supported on the target platform.
• OpenGL® is a standard specification defining a cross-language cross-platform interface for writing graphical applications.
• OpenGL® Shading Language is a high level shading language simular to C programming language; GLSL was derived from OpenGL Architecture Review Board(ARB), an interface specification allowing developers more direct control and manipulations of the graphics pipeline without using assembly language or hardware-specific languages.

[edit] Anticipated challenge(s)

[edit] Challenge(s)

The non-photorealistic implementation differs from the traditional rendering. The static nature of the defined image manipulations results in a 2D/3D image from the defined colors and attributes defined. The problem happens when the intended image is not smooth. What if the render technique desire is a rough shadow? The traditional model is immobile, to leverage the hardware power of the graphic hardware and to support dynamic context, the OpenGL shader would override the lighting and shadow. Following the data flow diagram variation of the render will override the coloring, opacity of the volume interpretation, and remove ray tracing.

Following the traditional design flow, the non-photorealistic implementation could be implemented with a mesh over the primitive. The illusion would give the user the impression a hard shadow has been wrapped over the object shadow part. This can be planned and created prior to rendering. However this customization does not work if the light source moves, the lighting will not map to the mesh location. Understanding the typical render pathway, if the lighting and shadowing were enabled, the render completion state would not map to the fake implementation. If the mesh shadowing was implemented the rendering would be a static implementation.

Hardware vendor support for the OpenGL ARB standards has been adopted and implemented in many of the graghic cards, but what is the software penalty for the software rendering of this feature. What is the cost for older cards attempting to use the non-photorealistic renderer? This does not impact the implementation, but it is a consideration that should be addressed and measured.

[edit] Anticipated approach to each challenge

With the understanding of the traditional rendering data flow, the non-photorealistic render engine would need to handle its color and opacity with the ray tracing disabled. To accomplish those needs and keeping the hardware acceleration, OpenGL shader can be developed to calculate and implement the needed texturing to simulate the desired artistic interpretation. OpenGL shaders run within the OpenGL API, referring to the architecture, and is applied within the graphic pipeline.

[edit] Literature review

1. Buchanan, J. and M. Sousa. The edge buffer: A data structure for easy silhouette rendering. NPAR 2000: Symposium on Non-Photorealistic Animation and Rendering, pages 39–42, June 2000.
2. Fein, Andi. and Morgan McGuire. Real-Time Rendering of Cartoon Smoke and Clouds. Proceedings of the 4th international Symposium on Non-Photorealistic Animation and Rendering. pages 21 - 26. June 2006.
3. Hertzmann, A. and D. Zorin. Illustrating smooth surfaces. Proceedings of SIGGRAPH 2000, pages 517–526, 2000.
4. Lake, A., C. Marshall, M. Harris, and M. Blackstein. Stylized rendering techniques for scalable real-time 3d animation. NPAR 2000: Symposium on Non-Photorealistic Animation and Rendering, pages 13–20, June 2000.
5. Markosian, L., M. Kowalski, S. Trychin, L. Bourdev, D. Goldstein, and J. Hughes. Real-time non-photorealistic rendering. Proceedings of SIGGRAPH ’97, pages 415–420, August 1997.
6. Martz, Paul. OpenGL® Distilled. First Edition. Boston: Addison-Wesley, 2006.
7. Raskar, R. and M. Cohen. Image precision silhouette edges. Proceedings of ACM Symposium on Interactive 3D Graphics, pages 135–140, 1999.
8. Rost, Randi. OpenGL® Shading Language, Second Edition. Boston: Addison-Wesley, 2006.

[edit] What I bring to this work

[edit] My relevant background and experience (CS 590)

I have been developing user interfaces and common components for over a decade. I have developed 3D pictograms used for Ultrasound machines. Rendering content has been a hobby of mine for all my life, as I was always interested in graphical representation.

[edit] What I find interesting about this work

About seven years ago, when I was working in Ultrasound, I was assigned the Pictogram feature. Pictograms are iconic images representing the content captured by the Sonographer. I decided to implement the pictograms as 3D representations of the image, where the transducer and positional information can be captured. This implementation was done using OpenGL loading 3ds models of anatomy I created, such as heart, fetus, hand, etc.

[edit] How this work goes beyond my experience and course work

This work goes beyond my experience with OpenGL, I am required to learn 2.x specifications and the new shader language. OpenGL version 2.1 was released in Aug 2006, the seventh revision from the original 1.0. Although OpenGL was backwards compatible, OpenGL enhances the power of programmable pipeline.

• Programmable Shading - OpenGL Shading Language creates, manages, and uses programmable shaders promoting the core features in OpenGL 2.x. The complete list of features related to programmable shading includes:
  • Shader Objects provides mechanisms necessary to manage shader and program objects.
  • Shader Programs - Vertex and fragment shader programs may be written in the high-level OpenGL Shading Language, replacing fixed-functionality vertex and fragment processing.
  • OpenGL Shading Language - a high-level program controlling the vertex and fragment pipelines. The Shading Language Specification defines how OpenGL features control vertex and fragment programs interact with the fixed-function OpenGL pipeline and how applications manage those programs.
• Multiple Render Targets - Programmable shaders may write different colors to multiple output color buffers in a single pass.
• Non-Power-Of-Two Textures - defines a restriction at the power-of-two dimensions has been relaxed for all texture targets, where the non-power-of-two textures may be specified without generating errors. Non-power-of-two textures was promoted from the ARB texture non power of two extension.
• Point Sprites - replaces point texture coordinates with texture coordinates interpolated across the point. This allows drawing points as customized textures, useful for particle systems.
• Separate Stencil - defines the functionality of the front and back faces of primitives, improving performance of shadow volume and Constructive Solid Geometry rendering algorithms.

OpenGL has matured to a point where the non-realistic rendering filter can be implemented without causing a performance degradation. To take advantage of new filters and versions, OpenGL 2.x APIs is required.