User:Ianroessle/Project Page
From CSWiki
Here is a link to the main course page of which this Project is under advisement: CS 491a page .
This project now is housed at sourceforget here
[edit] Brief project description
A Google Earth plug-in that provides an interface to Image analysis and data classification algorithms. Edge detection algorithms will assist in identifying shapes,which will be combined with Weka to classify homogeneous textures. Users will establish training data for the classifiers by drawing polygons using built-in functionality provided by Google Earth. Once classification is performed over the feature space, the resultant classifier is then applied to fresh data as it appears in Google Earth, as a semi-transparent image overlay.
[edit] Anticipated users
The users will be amateur Google Earth explorers who could use this plug-in too track changes in and find planetary phenomena.
[edit] Main conceptual (i.e., user-level) objects
The data set consists of the location, image, and altitude data. Along with this, there are a few higher level User objects: Samples, Classifiers, and Experiments. The Samples contain the raw data with the timestamp and a classification if this is a training sample. Above this is a Classifier which is the result of the data mining algorithm on a set of samples which can be used to take a unlabeled sample and classify it. Above this is an Experiment, which contains the samples taken at varying times, and the set of classifiers that have been generated at varying times.
[edit] Primary conceptual (i.e., user-level) operations
The conceptual flow of the application start with booting the plug-in. After this an experiment is created or loaded. Once this is done. A set of Folders appears inside of Google Earth. The user draws polygons in Google Earth and stores them inside of the plug-in generated folders. An example folder is "Positive Forest Sample", if you were conducting an experiment to classify forests. Once samples are collected you can switch tabs in the plug-in, configure settings and activate Weka to build a classifier. Once there are classifiers in the stored experiment, you can highlight it and "Apply classifier to view region", and it will apply the classifier to the current view screen. A test classifier was built to train a classifier on forested and deforested regions in the Rain Forest with a 99.5879% accuracy on a training set consisting of two samples consisting of 2700 data points with a 10-fold cross validation.
[edit] Why I am interested in this project
I chose this project because I am an explorer. My interest in Computer Science grew out of the investigation of Cellular Automata which I consider an analog for natural phenomena. I believe this plug-in will put in the hands of amateur's like myself a power tool to aid in their exploration efforts.
[edit] Status
All of the core proof-of-concepts have been worked through. This includes interfacing Java to Google Earth to control navigation by Win32 COM, interfacing to it's polygonal overlays and Raw Data using a mixture of Win32 GDI by JNI, COM and an XML Parser. The framework of a Java GUI for the plug-in is constructed.
The work now is in building a fully functional GUI, that allows one to create, save, open, and perform experiments, with a database to store all of the information. Also some of the communication between the applications is done via intermediary files. It would be much faster to pass pointers to references to objects in memory, between JNI and Java wrt Image data, as well as between Weka and Java wrt ARFF data.
Below you will find a log of my progress.
If you have any questions, comments, or suggestions please email me at ian at ianroessle.org,
Thanks,
Ian
Official Roadmap
0. [x] Includes all the stuff I did prior to building this roadmap. Learned Google Earth COM API, wrote routines for
data acquisition as proof of concept.
1. [x] Problem of waiting for streaming to finish.
2. [x] Port Application to Java from C#
3. [x] Write a function to automatically capture the data within a geographic region, with (left [latitude],right [latitude],
top [longitude],bottom [longitude]) parameters.. At a given scale in distances per pixel (measurement) in meters).
This includes marrying the picture data which is stored as a bitmap and the latitude/longitude/altitude data which is
CSV, all into a format for use with weka.
4. [x] Build Docked UI to do automatic Capture with boundary settings input from text fields. (The experiment Window)
5. [x] Write Script to acquire KML (Polygon Information) from Google Earth
6. [x] Modify/Fork Capture script to only capture data within Polygonal masks and incorporate Class Name into output dataset.
We can actually start working with Weka now..
7. [x] Add options in Docked UI to include settings for Weka, and button to apply data aquired from the training data into Weka.
8. [x] Add feature to load/save the resultant classification results, including confusion matrix, etc...
9. [x] Create button to apply/deapply classification results as KML experiment overlay in realtime. (Only the 50% Box)
10. [x] Add ability to load save the resultant KML Overlay over the entire experiment window given a specified scale, this should
include the classification result that led to this result as well.
11. [ ] Add ability to compare loaded experiment overlay to a previously saved KML experiment overlay. This will show two overlays
overlapping, also a popup window will report both the change in total area of all classifications, and the change in
the number of distinct elements fitting the classification with a threshold value of minimum area that constitutes an
element).
Project Done.. We now have a Machine Learning Plug-in for Google Earth that should be friendly enough for K-12 enjoyment.
Anything after this is superfluous to the deliverable but maybe should be considered for addition on a later date..
12. [ ] Consider issue of the "mosaic tiling of satellite photos", and if reapplying a classification result on a new data set,
how robust is the classifier on data from a potentially different season.
Add the option to save the previous training data and have a confirmation box to determine whether that training data still
fits and can be added to the previous data set ( to help to train a year round classifier)....
A potential forked application from here, assuming reasonable seasonally stable classifiers can be constructed
13. [ ] Create Windows Service to automate this procedure of reclassification once a month emailing the resultant deltas to a
specified address
14. [ ] Can the render window data be acquired from a minimized project window (Can this be ran in the background entirely).
15. [ ] Fork the project into a Distributed version which can be booted into server or Client mode for distributing the work across
a cluster, as a windows service
16. [ ] Create a website called Project Earth Watch, for advertising current experiments, that people can dynamically join in, help
and watch the planet change.
[edit] Week 12 (Spring Break.. supposedly, hah! and week 1 of Spring Term) - March 16th-March 28th, 2008
OK.. So there is one more nut to crack for this project, and it involves something called, the Universal Transverse Mercator Grid, which is the projection system used to project the globe onto a rectangular grid (for points relatively far from the polar regions.. another system is usually used there)
if I am to devise a device that can scan across an arbitrary portion of the globe, I need to know when I move the camera to a specific altitude at a specific location what i am looking at in the render window only by know the latitude and longitude of arbitrary points in the render window... I did a number of experiments.. which didn't come out all that great...Converting the points to UTM.. made things consistent with respect to the rate in which the latitude changed over the render window at a set altitude, so I ts possible to determine the aperture angle.
Is is worth nothing that the aperture is fixed on x and varies on y according to the aspect ratio....
The early experiments pre UTM.. had problems where the aspect ratio of lat/long varried according to geographical location.. because lat long isn't rectangular expect at the equator... but this is normalized enough in UTM..
So...
here are the results of the aperture experiments:
At one location @ 5000m altitude
f(0,0): lat(-11.5264), long(-59.0437) f(1,1): lat(-11.49241043952179), long(-58.99069554240289) Device Aspect Ratio w/h: 1.5283842794759825 Easting/Northing Aspect Ratio w/h: 1.514566884289667 Aperture angle on Easting: 0.522370229322127 Aperture angle on Northing (dependant on Aspect Ratio): 0.363249674108748
At another @ 10,000m altitude
f(0,0): lat(34.1), long(-118.2) f(1,1): lat(34.16793111051022), long(-118.07439796134139) Device Aspect Ratio w/h: 1.5283842794759825 Easting/Northing Aspect Ratio w/h: 1.575626077252505 Aperture angle on Easting: 0.5280343527095376 Aperture angle on Northing (dependant on Aspect Ratio): 0.3545462359882892
Same point as experiment 2 but with a different device aspect ratio, illustrating fixed easting angle.
f(0,0): lat(34.1), long(-118.2) f(1,1): lat(34.2046526683447), long(-118.07430459194445) Device Aspect Ratio w/h: 0.992721979621543 Easting/Northing Aspect Ratio w/h: 1.0210365314042145 Aperture angle on Easting: 0.5299539585877998 Aperture angle on Northing (dependant on Aspect Ratio): 0.5209204898854602
Success!..
The last experiment was modified based on information gathered in the previous three to produce this:
f(0,0): lat(34.1), long(-118.2) f(1,1): lat(34.16793111051022), long(-118.07439796134139) Device Aspect Ratio w/h: 1.5283842794759825 Easting/Northing Aspect Ratio w/h: 1.575626077252505 Experimental Aperture angle on Easting, alpha: 0.5280343527095376 Experimental Aperture angle on Northing , beta: 0.3545462359882892 Expected Aperture angle on Northing by device aspect ratio and experimentally determined alpha: 0.36457188069386093
What this shows.. is that in my scanner I can hold alpha as a global constant. and determine beta by:
beta = atan( tan(alpha)/device aspect ratio).
This is more accurate if I used the experimental easting/northing aspect ratio.. so I might just have a calibration routine when I start a scan although it's a toss up.
here is the code used on this last experiment.. for historical purposes (the code is turning into the the code that will complete the next milestone.
IPointOnTerrainGE loc_ur = ge_glue.GE_interface.getPointOnTerrainFromScreenCoords( 1.0, 1.0);
LatLong latlong_ur = LatLong.valueOf(loc_ur.latitude(), loc_ur.longitude(), DEGREE_ANGLE);
UTM utm_ur = latLongToUTM.convert(latlong_ur);
double w=Math.abs(utm_ur.eastingValue(METER)-utmcenter.eastingValue(METER));
double h=Math.abs(utm_ur.northingValue(METER)-utmcenter.northingValue(METER));
double grid_ar = w/h;
double alpha = Math.atan((w/2)/alt);
double beta = Math.atan((h/2)/alt);
writer.write("f(0,0): lat(" +lat + "), long(" + lon+ ")");
writer.write("f(1,1): lat(" +loc_ur.latitude() + "), long(" + loc_ur.longitude() + ")");
writer.newLine();
writer.write("Device Aspect Ratio w/h: " + device_ar);
writer.newLine();
writer.write("Easting/Northing Aspect Ratio w/h: " + grid_ar);
writer.newLine();
writer.write("Experimental Aperture angle on Easting, alpha: " + alpha );
writer.newLine();
writer.write("Experimental Aperture angle on Northing , beta: " + beta);
writer.newLine();
writer.write("Expected Aperture angle on Northing by device aspect ratio and experimentally determined alpha: " +
Math.atan(Math.tan(alpha)/device_ar));
writer.newLine();
Onto building the scanner and populating a database.
[edit] Week 11 (Winter Finals Week) - March 8th-March 15th, 2008
Revised report and put it at the top of this page.
[edit] Week 9,10 - February 23rd-March 7th, 2008
Wrote half time Report here: 491 Report
Investigated KML Parser here: gekmllib
[edit] Week 7,8 - February 9-22th, 2008
- As per instructions on Brant's Blog I got a working implementation of the Google Earth COM from Java... The key to interfacing Win32 COM and Java was a tool called com4j. This utility is really cool.. It builds a set of .java files that map to the com interfaces of a given program!
java -jar tlbimp.jar -o ge_code -p googleearth "c:\program files\google\google earth\googleearth.exe"
And you have your java classes that map to the com interface.
- This forked a sub task that I almost forgot about. While I have a working COM interface to Google Earth, I need access to GDI32 and User32 to get the Render Window data. So I learned Java Native Interfaces....
The documentation for JNI can be found here It includes a fairly decent hello world tutorial... Compiling the DLL can be done from the commandline with visual studio 2005 by something like this..
natives.cpp
#include <jni.h>
#include "googleearthscanner_HelloWorld.h"
#include <stdio.h>
JNIEXPORT void JNICALL Java_googleearthscanner_HelloWorld_displayHelloWorld
(JNIEnv *, jobject)
{
printf("Hello world!\n");
return;
}
googleearthscann_HelloWorld.h
#include <jni.h>
#ifndef _Included_googleearthscanner_HelloWorld
#define _Included_googleearthscanner_HelloWorld
#ifdef __cplusplus
extern "C" {
#endif
JNIEXPORT void JNICALL Java_googleearthscanner_HelloWorld_displayHelloWorld
(JNIEnv *, jobject);
ifdef __cplusplus
}
#endif
#endif
cl -I"c:\Program Files\Java\jdk1.6.0_03\include" -I"c:\Program Files\Java\jdk1.6.0_03\include\win32" -LD natives.cpp -Fenatives.dll
Next up writing the GDI32 and USER32 C++ Wrapper code in natives.cpp. That will have to wait until later in the week -Ian Friday Feb 8th, from 6-11pm
JNI Interface to System Calls working...
modified natives.cpp that performs system call:
//System APIs
//#include <wingdi.h>
#include <windows.h>
#include <winuser.h>
//Java Natives
#include "googleearthscanner_HelloWorld.h"
#include <jni.h>
//Other
#include <stdio.h>
JNIEXPORT void JNICALL Java_googleearthscanner_HelloWorld_gdi_1MaximizeHwnd
(JNIEnv *, jobject, jint hwnd)
{
int bla=hwnd;
SetForegroundWindow((HWND) bla);
printf("Hello world number %d", bla);
return;
}
New command for compiling DLL needs to link with user32.lib and gdi32.lib
cl -I"c:\Program Files\Java\jdk1.6.0_03\include" -I"c:\Program Files\Java\jdk1.6.0_03\include\win32" -LD natives.cpp user32.lib gdi32.lib -Fenatives.dll
Next up.. finish the port with all technical difficulties mapped out, finishing next milestone, also the code is spread out over may langugages.. and commands.. needs to be organized better..
here is the order of operations between test builds.
1) Write the code in Java, describing Native interfaces as necessary 2) Compile Native Header file using javah 3) Implement native functions in natives.cpp 4) Compile DLL 5) Test project..
and the C++ code needs to be integrated into the java application better.. --- -Ian Wednesday afternoon 2 hours logged 4-6pm.
Java Program has a qualified bootup sequence. port and milestone 2 complete.
Last minute I found out that JNA might be a slightly easier approach than JNI.. -ian Wednesday from 7-10pm
Got a working JPanel, that is persistent, it has tabs..one of which is a tester for goto xy function. This is a big step toward milestone 4.
Figured out how to send simulated keystrokes to the main google earth program. This give me the ability to access the KML Polygonal data. A step toward milestone 5, next up for this is to test out a good XML parser/builder. to interface the overlay I/O.
Debugged issue with Netbeans...
Found XML Parser...
Next Up.. build OO Model for classification.
[edit] Week 6 - February 8, 2008
Outlined Official Road map. First 11 milestones are what I expect to have in the finished project.
5(This quarter) +1 (Spring Break) + 11 (Spring Quarter) = 16 Weeks Remaining.
By the end of this quarter I plan to complete the first 6 milestones.
Spring break I will tackle the Milestone 7.
Spring Quarter Milestones 8-11, and whatever QA Testing/ Documentation that is necessary.
From here on I will use the numbers in the road map as a reference to indicate progress.
[edit] Week 5 - February 1, 2008
[edit] More with Google Earth
Have working code to boot google earth, maximize it fly to a specific location, capture color and altitude characteristics along with respective latitude/longitude.
A lot of it involves Win32 COM and Windows System Calls.
Next step is to write code to verify that google earth was done downloading the data, it will repeatedly take snapshots until the bottom of the map says that it is done streaming.
Additionally a external toolkit window needs to be built and docked to the left hand side of the screen to control parameters of the machine learning plug-in. Parameters are to include
2) scale in pixels / meter^2 to perform classification 3) Depth of neighbors to use in classification 4) Checkbox to use altitude info as well 3) Which algorithm, and its parameters.
4) A button to Perform the learning
The procedure is:
1. Boot the program which will boot Google earth and initialize the plug-in. 2. Training sets will be indicated by drawing polygons within Google Earth of a specific color for each class and can be loaded from kml files. 3. Tune the plugin to desired settings and click the button 4. wherever you are, it will apply the classification parameters to what you are viewing, although the neighbors data is interrelated to the scale.. so issues of zooming in need to be factored in to the application of the classifiers.
Input and output will be interfaced with features of Google Earth.
[edit] Week 4 - January 25, 2008
[edit] Google Earth COM API
There is documentation here to access the google earth client API via COM: http://earth.google.com/comapi/
It provides enough information to control the camera and with a little work pull image data from it as well.
I've got all but the image pulling working. although I know it's possible.. and will finish that later tonight or tomorrow...
Decided to build an overlay to classify which satellites took pictures where.. This is a necessary step to doing image analysis on the planet as unnormalized data can introduce less accurate results later on.
My feature set and choice of algorithm I will leave undisclosed at this time and may change with analysis....
An interesting side effect of this is I can also build a polygonal map of earth should I care too... Might be something cool to import into Croquet and use the actual pictures as surface maps.. Taking things down this route would lead to interesting considerations on scalability.. The earth is pretty big..
I might have to scale down my project to just one country..
-Ian
[edit] Week 3 - January 18, 2008
[edit] Croquet and Smalltalk Demo
Croquet is a 3d environment written in Smalltalk. Here is a small demo of Croquet Starting the environment from under the hood and getting a new object to appear in the world.
[edit] Demo Script
Smalltalk is an interesting langugage.... All of the source code is interpreted at runtime.. so methods can be changed dynamically. It's all about Instantiating objects and then communicating with these objects with messages.. As a programmer coming in.. you initiate a program by instantiating an object and sending a message too it, wich could cascade into something..
Start things going You need a Workspace, and a transcript. The System Browser is useful too...
Here is a tutorial for java programmers on how to learn the syntax and get around.
http://daitanmarks.sourceforge.net/or/squeak/squeak_tutorial-1.html
And then there is Croquet..
http://www.opencroquet.org - A link to the website http://www.opencroquet.org/index.php/Developer%27s_Guide - a link to the developer guide..
And ...
[edit] Script to load croquet from programmer interface and create cube
master := CroquetMaster new openInWorld. harness := master instVarNamed: #harness. spaceRef := harness avatar replica island future at: #masterSpace. cubeRef := spaceRef island future new: TCube. spaceRef future addChild: cubeRef.
[edit] Week 2 - January 11, 2008
[edit] Java Platform in Prolog
Java provides a rich platform for building language and doing symbolic arithmetic.
In CS 332L we explored this. We worked in SWI-Prolog, you can download it here, http://www.swi-prolog.org/.
Here is the current working version of the Prolog java Implementation.
http://www.ianroessle.org/eval_with_gc.zip
About the presentation:
I know I didn't exactly show you why there were 5 objects in memory during the peek.. but thinking it through there should be 5 objects.. at the point in time... that that peek occured. It was during the second constructor call at which point the original c object isn't destroyed yet. so there are 4 for the two c objects.. and the extra object that is locally created in the constructor at that time, yeilding 5 internal variables. Computers don't make mistakes...
Anyhow.. that kinda flustered me from giving the rest of the presentation... that and the time crunch...
So,I've successfully implemented linked lists and other higher level data structures with this language...
In doing so I learned the importance of the block operator.
Implementing the block operator is the next major important level of complexity in higher level language development.
From there maybe some operators to facilitate parallel processing..
and maybe mapping out different levels of complexity in language structure.. all of which are capable of producing Recursively Ennumerable outcomes.. but at different levels of ease to the programmer.
