PixelObjectList implementation - Duplicates by visual identification #79

chrishajduk84 · 2017-03-09T05:03:27Z

This first PR only looks at the visual portion of the analysis.
Note there are some preliminary GPS functions and colour functions for now. They are a result of planning. They are included in this PR, but are not in a functional state. Please ignore these mostly empty functions.

This PR should have been completed a while ago....here it is.

…tial list for matching later on.

…yzed on a pixelobject level.

…jectlist

…of fuzzy logic, but it will have to do. Eventually, these fuzzy logic sections should be user-programmable.

… replacing it with functions within a namespace.

…, write atest to see if PixelObjectList works.

… recognition using PixelObjectList is programmed. The fuzzy logic still needs to be fine tuned and modified. An integration test has been written for unique object detection.

…leted to reduce processing time. At the current rate, a full flight of objects will require over 8 billion minutes to process (and who knows how much memory). This algorithm may need to be optimized a bit more, however human vision interaction may be necessary to garbage collect objects which have no importance or significance to the overall mission. Also added target analysis functionality into main.cpp.

…ges to pixel_object_list and supporting functions. Expanded on fuzzy logic and hieretical analysis

…ed some less important functions, that currently served no purpose.

benjaminwinger

This looks pretty good in general, though it seems a little incomplete. You should probably focus on getting the GPS stuff working and testable, since I imagine we'll be mainly be relying on clustering by GPS co-ordinates.

benjaminwinger · 2017-03-10T03:12:51Z

modules/core/include/object.h

+     *
+     * @return The distance covered by each pixel of the image in the X and Y
+     * directions.
+     */


Objects wouldn't have a single distance since they are composed of multiple frames. If anything you might want to include this sort of information in a frame.

benjaminwinger · 2017-03-10T03:15:43Z

modules/core/include/target.h

+     * @param y The distance covered by each pixel of the image in the Y.
+     */
+    void set_pixel_distance(double x, double y);
+    void set_pixel_distance(cv::Point2d);


Same as the comment on Object, Target likewise isn't tied to a single frame

benjaminwinger · 2017-03-10T05:09:26Z

modules/targetanalysis/src/pixel_object_list.cpp

+    }
+    //Is the best match good enough - FUZZY LOGIC
+    //TODO: This should be a configurable value via a socket connection or a
+    //learning algorithm


Just a thought here, but we should create a settings singleton for this purpose. Then the settings object can handle exposing a way to be modified that we can use through the GUI.

Little slow on this PR. This TODO would be something I'll try and include in a separate PR.

benjaminwinger · 2017-03-12T02:13:19Z

modules/targetanalysis/include/target_analyzer.h

+         *        
+         * @return  vector containing the unique Targets which have been analyzed
+         */
+        void getGPSCorners(cv::Point2d cameraAlpha, double longitude, double latitude,


This should probably operate on a just a Frame. We would need to include to include a reference to camera information (including the alpha) in Frames. For now probably just add the cameraAlpha to the Frame class, we'll have to add support for camera profiles of some sort later.
You'd be able to get the lat/lon/alt/heading too from the frame's Metadata struct.

Also, unit tests would be great for the GPS calculations.

I don't know if we've talked about this before, but assume that at this point the image is rectified, later we'll have to rectify Frames when we first create them.
I'm not sure what we should do about perspective distortion though. The stock GoPro lens is really wide, so even with correcting for lens distortion, the perspective distortion will still mean that the edges of the frame are going to quite compressed compared to the centre, meaning that using the corners of the image to interpolate the position of a given point may be hit and miss (if that was what you were planning on using this for). I think going from the centre should give better results in general.

Good point about using the center rather than corners. Using something similar to "pixel distance" will be used to determine other points based on the center point.

benjaminwinger · 2017-03-12T18:05:49Z

modules/targetanalysis/src/pixel_object_list.cpp

+    return 0;
+}
+
+double PixelObjectList::compareContours(PixelObject* po1, Object* o2){


Given how slow clustering can be already, I'd sort of prefer to be comparing precomputed data. I think it should be sufficient to compare precomputed information such as the regularity of the contour, largest and smallest diameter, etc..
If anything we could start with comparing precomputed information and directly compare contours to double check clusters.

benjaminwinger · 2017-03-13T01:35:55Z

modules/core/src/object.cpp

+}
+
+void Object::add_pobject(PixelObject * po){
+    pixelObjects.push_back(po);


Note that there's a bunch of other things that need to be done here. For now just averaging the data from the PixelObjects in the Object should be fine, we can do more later.

…coordinate math, followed by threshold analysis.

… Added unit test to check calculations. For now, wide angle go-pro alpha angle values are being used. Exact calibration should be done manually.

chrishajduk84 · 2017-04-12T23:12:59Z

@benjaminwinger Issues fixed with respect to your comments.

Two things that are not in this PR and will be worked on in a future one:

Settings Singleton (Should this be for the whole warg-cv suite? or just for my module? I'm thinking the whole thing)
Option for pre-computed data (for contours), I'm still not quite sure how I will do this exactly. Once again, this is a separate PR.

-GPS code works in unit tests, I just need a proper camera calibration (alpha values) and to test it in the field or with videos.

…ects together is very iffy. It needs to be tuned using a GUI interface (AKA Human Vision).

…nd threshold_bias values for the pixel_object_list analysis.

chrishajduk84 · 2017-04-19T04:20:00Z

@benjaminwinger If you get the chance, can you review this PR?

benjaminwinger · 2017-04-19T23:33:48Z

Sorry, I've been a little busy (and sick).
Working on it.

chrishajduk84 · 2017-04-19T23:34:55Z

Sorry no worries. I'm going to try and get a live version of this running this weekend though...

benjaminwinger

I've added a few comments. I'm going to have to look at this more tomorrow.

benjaminwinger · 2017-04-20T23:31:42Z

modules/core/include/object.h

+     * positions/colours/areas with potential error
+     *
+     */ 
+    void update();


I note that you only use this in conjunction with add_pobject (and vice versa). Why not combine the two methods?

benjaminwinger · 2017-04-21T02:53:51Z

modules/targetanalysis/include/pixel_object_list.h

+     * @param o the pixelobject that is converted to an object and added to the
+     * list.
+     */
+    void addNode(PixelObject* o);


Is there a situation when this needs to be called externally? I would imagine that you only want to be calling addAndCompare. The same goes for the comparison functions, wouldn't they only be used internally?
I don't think there is any reason for them to be public.

benjaminwinger · 2017-04-21T04:07:59Z

modules/core/include/object.h

 * @file object.h
 * @author WARG
 *
+ * @brief Container class for storing information about


This describes a class not a file. I'm not really sure why this needed to be moved.

benjaminwinger · 2017-04-21T04:09:37Z

modules/targetanalysis/include/pixel_object_list.h

+ * possible error. As targets are processed unique targets will 
+ * be identified and the data combined into a single object.
+ *
+ * This class is a singleton.


Does this really need to be a singleton? I would think that you only want the TargetAnalyzer singleton to be accessible globally. You could just give the TargetAnalyzer singleton an instance of the PixelObjectList.

Also, why have a special data structure for the PixelObjects? Wouldn't it work to do the comparisons inside the TargetAnalyzer and use an STL data structure to store the objects?

This works right now. I'll make it a non-singleton, but I'm not going to go through and modify the List with the standard library.

I'll fix this later is what I am saying. (AKA low priority)

Understandable. Abstracting away the data structure used would make it easier to make it thread-safe, but I think the best thing to do for now is just to run this in a single thread anyway. That would mean buffering the pixel_objects to process and creating a new worker thread that reads from the buffer and handles them (adding to the buffer is technically still not entirely thread-safe since most STL data structures don't support concurrent writers, but a lot better than modifying the PixelObjectList from 8 threads at once. Boost has some good thread-safe containers that should be looked into at some point).

Agreed. The slow part is still the TargetId stuff, so a single thread should be okay...
You are right multiple threads in list comparisons could get a little bit tricky.

IE: Object 1 and object 2 are the same, but due to concurrency don't ever get properly compared and are both appended to the list as a unique object.

benjaminwinger · 2017-04-21T04:13:34Z

modules/targetanalysis/src/object_list.cpp

+    //not it is a duplicate will be known
+    int i = 0;
+    while(i < listLength){
+        tail->o->


This clearly doesn't compile. The only reason there isn't an error is because this file isn't included in the CMakeLists.
Then again, is the ObjectList really necessary since the PixelObjectList is handling Objects?

I will remove this file.

benjaminwinger

This looks pretty good overall from a code point of view.
Hopefully we can test it's effectiveness some time before the competition (I'll try to put the camera stuff up tomorrow so that we can get more accurate GPS co-ordinates)!

benjaminwinger · 2017-04-22T00:14:35Z

modules/core/src/object.cpp

+    cv::Scalar cSum;
+    for (PixelObject* po : pixelObjects){
+        this->centroid += po->get_gps_centroid();
+        this->area += po->get_gps_area();


This isn't being updated afterwards to make it an average instead of a summation

benjaminwinger · 2017-04-22T00:19:14Z

modules/core/src/object.cpp

+    this->centroid = cv::Point2d(0,0);
+    cv::Point2d maxDistance(0,0);
+    cv::Scalar cSum;
+    for (PixelObject* po : pixelObjects){


Noting my other comment about combining this with adding objects, when adding objects these values could be updated rather than re-calculated.

Okay I will rename this function to "recalculate" and have another called "update". Both will be private to the class.

benjaminwinger · 2017-04-22T01:25:06Z

modules/targetanalysis/src/pixel_object_list.cpp

+        return 0;
+    }
+    po1->set_gps_centroid(result);
+    double distance = TargetAnalyzer::getInstance()->getGPSDistance(result, o2->get_centroid());


I worry a bit about calculations like this in that the result is possibly going to vary quite a bit depending on the order that targets are added. Unfortunately it's probably the best we can do for now given that it would get a lot more complicated to do otherwise.

Agreed. A better solution would be to store all GPS coordinates and then use K-Means clustering....probably won't happen before competition.

benjaminwinger · 2017-04-22T01:41:45Z

modules/targetanalysis/src/pixel_object_list.cpp

+
+
+//TODO: ADD OPTION TO ONLY LOOK AT PREDEFINED SHAPES
+double PixelObjectList::compareContours(PixelObject* po1, Object* o2){


How slow is this actually? You had said before that it wasn't terribly fast, and it's just going to get a lot slower the more objects are in the list.
That is to say, after 20 minutes in the air, can we expect this to be running at a remotely usable speed? I think it's safe to assume we may have as many as several hundred pixelobjects detected during the flight and every time a new one is added it will be compared to each of the previous ones in turn. If it takes a few minutes to add a each one by the end then I don't think it's worth using this as it is at all.

I think I've suggested before just using precalculated statistics on the shape such as the shortest and longest real world distances across the contour through the centre, maybe some sort of regularity calculation too.

At the same time, if it is actually fast enough to be usable, it might be prudent to leave it be for now, given that we don't have much time.

On my laptop 75 Pixel Objects take 90 seconds to compute. It is slow. I will use precomputed statistics instead from images I got from last flight test.

I'm not sure how to hard code it. I'm thinking of using the target_loader, I partly made last year....to load a file with contour points.

What do you think? @benjaminwinger

No, not hard-coded shapes, if we want to be filtering based on shape we can do that in the identification phase.
Here the point would be to cluster based on shape, so you'd calculate certain statistics, as mentioned before, during the identification phase and then here we they can be compared similarly to how you're comparing colour or location.

benjaminwinger · 2017-04-22T01:48:29Z

modules/targetanalysis/test/unique_po_test.cpp

+    BOOST_LOG_TRIVIAL(debug) << "Images in test: " << numImage;
+    BOOST_LOG_TRIVIAL(debug) << "PixelObjects in test: " << numPixelObjects;
+
+    BOOST_WARN_MESSAGE(numPixelObjects == 75, "Unexpected number of pixel objects, has the target id algorithm changed? Are there more images than expected?"); //If the number of test photos has changed, this integration test may become invalid


Isn't unit testing fun...
You don't need to change this now, but it would be better to feed synthesized data into the TargetAnalyzer so that the results are more predictable. You don't want this test to be failing just because the identification algorithm has changed.

benjaminwinger · 2017-04-22T02:20:58Z

Also, regarding the duplicate comments from yesterday (now deleted), the first time I accidentally closed the page and they disappeared, then my graphics driver crashed and I had to reboot my computer.
Apparently my comments weren't lost after all, though they weren't showing up before I submitted the rewritten ones.

benjaminwinger · 2017-04-25T02:44:42Z

main.cpp

    boost::asio::io_service::work work(ioService);
    processors = boost::thread::hardware_concurrency();
-
+    


In all honesty, why do editors even allow you to create lines like this?

Woops. Any vim extension suggestions?

See http://vim.wikia.com/wiki/Remove_unwanted_spaces
What I use:

func! DeleteTrailingWS() exe "normal mz" %s/\s\+$//ge exe "normal `z" endfunc autocmd BufWrite *.{c,h,cpp,hpp,rs,java,py,etc...} :call DeleteTrailingWS()

The BufWrite glob can be modified depending on the files you use. Note that it isn't actually good to do this for all files. Try programming with Whitespace when doing so for example (why you would actually use Whitespace is an entirely different question).

It also means that everything you touch gets whitespace stripped, and depending on what codebase you're working on, people can get annoyed if you start checking in changes where every file that you touch has loads of whitespace changes. I generally just use git add -p and avoid other people's whitespace issues.

…tude faster and provides very similar results. Computation of length to width ratio, area to perimeter ratio, actual area to square area, etc.

Chris Hajduk added 16 commits September 27, 2016 22:41

New PixelObjectList Class. Used to store all pixelobjects in a sequen…

25b5eba

…tial list for matching later on.

Added haversine formula for calculating GPS coordinates

6b2ce3f

Finished GPS corner function

030044c

Possibly finished the GPS coordinate implementation

94566f0

Add GPS coordinates to Pixel Objects, uniqueness is observed and anal…

c15c80a

…yzed on a pixelobject level.

Reformatted targetanalysis code to use objectlist rather than pixelob…

5725401

…jectlist

Redefined TargetAnalysis code flow

02473c7

Added a compareAndAdd function. For now it uses statistics and a bit …

b0e2c06

…of fuzzy logic, but it will have to do. Eventually, these fuzzy logic sections should be user-programmable.

Doesn't need to be a class, because it would be a singleton... Simply…

2996458

… replacing it with functions within a namespace.

Code compiles. Whether it works or not is another question. Next step…

0b09a23

…, write atest to see if PixelObjectList works.

Started building uniqueness test for pixel objects

2d1caa6

Added test images where duplicates are present. Preliminary duplicate…

6a7b80d

… recognition using PixelObjectList is programmed. The fuzzy logic still needs to be fine tuned and modified. An integration test has been written for unique object detection.

Removed unecessary file for testdata specific to targetanalysis. Chan…

9cc79aa

…ges to pixel_object_list and supporting functions. Expanded on fuzzy logic and hieretical analysis

Documented the PixelObjectList Class

1f504f0

Added final comments and documentation to target_analyzer. Also remov…

d1ace28

…ed some less important functions, that currently served no purpose.

chrishajduk84 requested a review from benjaminwinger March 9, 2017 05:03

benjaminwinger suggested changes Mar 13, 2017

View reviewed changes

chrishajduk84 closed this Apr 8, 2017

Chris Hajduk added 4 commits April 7, 2017 22:41

Merge Conflict resolved

c82d14d

Beginning integration of GPS analysis. Next step, Unit tests for GPS …

2dbb76e

…coordinate math, followed by threshold analysis.

Added ability for contours to be GPS located (based on the centroid).…

3a7bbe4

… Added unit test to check calculations. For now, wide angle go-pro alpha angle values are being used. Exact calibration should be done manually.

Finished GPS integration. Added tests to test functionality.

63b6663

chrishajduk84 reopened this Apr 12, 2017

Chris Hajduk added 2 commits April 18, 2017 21:17

Colour Comparison Working. The fuzzy logic which groups duplicate obj…

9c05359

…ects together is very iffy. It needs to be tuned using a GUI interface (AKA Human Vision).

TargetAnalyzer is the settings singleton. It now contains threshold a…

bc1b157

…nd threshold_bias values for the pixel_object_list analysis.

benjaminwinger reviewed Apr 21, 2017

View reviewed changes

benjaminwinger suggested changes Apr 22, 2017

View reviewed changes

Optimized addition of nodes, fixed issues with private/public members

3e8bc64

benjaminwinger reviewed Apr 25, 2017

View reviewed changes

compareContour changed to be super efficient. It is an order of magni…

33df7f8

…tude faster and provides very similar results. Computation of length to width ratio, area to perimeter ratio, actual area to square area, etc.

chrishajduk84 force-pushed the master branch from 33c0663 to 33df7f8 Compare April 27, 2017 00:59

Chris Hajduk added 2 commits April 26, 2017 21:01

Merged Conflicts

99271e8

Fixed to incorporate new constructor for Frame.h

0fe008e

benjaminwinger mentioned this pull request Jun 20, 2017

Object serialization #91

Open

benjaminwinger pushed a commit to benjaminwinger/computer-vision that referenced this pull request Jan 12, 2018

Added geolocation code (adapted from Chris's PR UWARG#79)

cee34a5

benjaminwinger pushed a commit to benjaminwinger/computer-vision that referenced this pull request Jan 12, 2018

Added geolocation code (adapted from Chris's PR UWARG#79)

2c99eea

benjaminwinger mentioned this pull request Jan 15, 2018

Geolocation #96

Open

benjaminwinger pushed a commit to benjaminwinger/computer-vision that referenced this pull request Jan 21, 2018

Added geolocation code (adapted from Chris's PR UWARG#79)

e21e905



		//TODO: ADD OPTION TO ONLY LOOK AT PREDEFINED SHAPES
		double PixelObjectList::compareContours(PixelObject* po1, Object* o2){

		boost::asio::io_service::work work(ioService);
		processors = boost::thread::hardware_concurrency();

PixelObjectList implementation - Duplicates by visual identification #79

Are you sure you want to change the base?

PixelObjectList implementation - Duplicates by visual identification #79

Uh oh!

Conversation

chrishajduk84 commented Mar 9, 2017

Uh oh!

benjaminwinger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjaminwinger Mar 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrishajduk84 commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrishajduk84 commented Apr 19, 2017

Uh oh!

benjaminwinger commented Apr 19, 2017

Uh oh!

chrishajduk84 commented Apr 19, 2017

Uh oh!

benjaminwinger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjaminwinger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

benjaminwinger Mar 12, 2017 •

edited

Loading

chrishajduk84 commented Apr 12, 2017 •

edited

Loading

benjaminwinger commented Apr 22, 2017 •

edited

Loading