Issues related to geodesk-tiles

[pbsurf]

• FeatureUtils::isEmpty() - I assume this should be == nullptr, not != nullptr (or could do FeaturesBase::isEmpty() { return begin() == end(); } and FeaturesBase::bool() { return !isEmpty(); } and get rid of FeatureUtils::isEmpty() since I don't see any other uses?)

• bool(TagValue) returns false if tag value is "no" ... but for some tags such as access=no, the absence of the tag and a no value have opposite meanings. Maybe restrict bool() to existence testing only (or add a separate TagValue::exists() method)?

• double(TagValue) appears to give undefined result if parseDouble fails since val is not initialized. Maybe init to 0?

Issue with multiple threads:

There is a note in Query::requestTiles() that multiple consumer threads aren't supported. I just ran into this problem but it's not clear to me what is going wrong. I see both consumer threads stuck in Query::take() with pendingTiles_ > 0 and completedTiles_ == 0 but the ThreadPool<TileQueryTask> queue is empty. I'll try tomorrow with thread sanitizer.

Misc:

• Observed gol increase from 280MB to 1GB when accessing w/ libgeodesk ... not sure how this is possible since file is opened read-only. File returns to original size after running gol info.
• geodesk tool issue: building XXX.gol fails with confusing error if XXX-work/ folder remains from aborted run.

[pbsurf]

Change to fix thread sanitizer warnings (don't think any of these were a problem, but now actual issues won't get buried): pbsurf@4f8053e?w=1 (w=1 to hide whitespace changes since my editor is set to strip trailing spaces on save - I tested that the following removes all trailing whitespace: find . -name '*.h' -o -name '*.cpp' -exec sed --in-place 's/[[:space:]]\+$//' {} \+ ).

Also, it seems EMPTY is used before it's initialized here: https://github.com/clarisma/libgeodesk/blob/main/src/query/TileQueryTask.cpp#L11

[clarisma]

• FeatureUtils::isEmpty() is now fixed -- great catch, much appreciated!

• bool(TagValue) returning false for no is by design (to match the behavior of the Python and Java toolkits). To discover whether a feature or tag set has a given key with any value, you can use hasTag(). But maybe I should create a TagValue::isNull() method to explicitly check for an empty tag?

• double(TagValue): Still investigating

• Multi-threading: I will need to revise the queue management. The execution model is changing in V2, but I'll likely backport changes in the meantime. And yes, atomic refcounting is crucial as well. Will look into this next week.

• TileQueryTask.cpp -- EMPTY is used before it's initialized: Good catch, this is non-compliant and only works by the grace of the compiler. I'll revise this.

• Increase of GOL size: This is an artefact of the memory-mapping approach in the current Java-based GOL Tool (Only the logical file size changes; the GOL won't actually consume 1 GB of disk space in this case). The new GOL Tool avoids this.

• Confusing error after aborted run: Can you tell me the error, and at what stage the previous build aborted (if known)? May be related to build: Delete work folders in case of unrecoverable error gol-tool#87

Thanks again for your help!

[pbsurf]

TagValue::isNull() could be useful.

I haven't encountered any problems yet with multiple consumer threads since fixing the thread sanitizer warnings.

Because Key can't be created until a GOL is opened, it would be useful to support default construction of a null Key, as here: pbsurf@a4b849e?w=1

A minor issue: Feature::bounds() asserts when called for a relation because it does WayPtr(feature_.ptr).bounds(); here while xy() just does feature_.ptr.bounds().

[clarisma]

Thanks, very good points!

• Key now has a default constructor
• I resolved the assert in Feature::bounds()

As for multiple concurrent query consumers, I will need to make a change in the queue processing (It was originally designed to support the Python library, which prior to Python 3.13 was guaranteed to be single-threaded). Possible deadlock if queue is full (which isn't possible if only one consumer). Fix is coming.

[pbsurf]

Maybe I'm misusing the term "consumer thread" - in my case each thread is creating a separate Features object (from the shared global Features) for some tile bounding box. It seems like if the ThreadPool queue was full, threads would just be waiting on the notFull_ condition variable (with ThreadPool::mutex_ unlocked), with one thread released each time notFull_ is notified. I also don't see any interaction with a second mutex (such as Query::mutex_) that could result in a true "deadlock". I did a quick test setting the queue size to 1 for the FeatureStore::executor_ ThreadPool with two threads iterating over separate Features objects and still didn't get any hangs or thread sanitizer warnings. Anyway, feel free to ignore my rambling here since I'm obviously much less familiar with the code base.

[pbsurf]

Eliminating the dependency on boost makes it possible to build on debian/ubuntu without installing any additional headers (whereas the boost headers are over 160MB!): pbsurf@03f8c69

[pbsurf]

Also, any tips on speeding up gol build for the whole planet with limited RAM? Is it possible to build and then merge subsets? Trying on a VPS with 6 cores but only 8GB RAM and it slowed down to ~100K ways/sec in the sorting phase (so maybe 10 days to finish?)

[clarisma]

Quick thoughts, will check more closely over the weekend.

1. crc32: I do try to eliminate dependencies whenever possible, so thanks for the suggestion. Ideally, I'd like to find a portable version with intrinsics, but your LUT-based function is perfectly reasonable. Previously I used the crc32 function from zlib, but that created a link-time dependency. FetchContent should only bring in the header for Boost's crc module. Is it downloading all of Boost in your configuration?
2. A few years back, I ran gol build for a planet PBF (about 60 GB at the time) on a dual-core laptop with 8 GB RAM and a SATA SSD. It finished in just under 5 hours (with fans running at full blast). The sort phase relies on an index, which currently takes up about 20 GB. If this index can't be kept in memory, excessive swapping kicks in and tanks performance, so 100K ways per second seems reasonable. The other phases are less memory-sensitive and mostly CPU-bound. So I'd estimate 4 to 5 hours total for your setup. Unfortunately not possible to merge multiple GOLs built from smaller sets. In v2, there will be a way to save a GOL in compressed form, so you can build it on a fast workstation, transfer it to a server and then load it again (basically zip/unzip, but much faster, and with higher compression).

[pbsurf]

1. Whoops, nevermind. I'm not using cmake, so it was just using crc.hpp from the boost package I already had installed, which in turn uses other boost headers, whereas crc.hpp from github.com/boostorg/crc doesn't use any other boost headers.

2. The sort phase finished in 12 hours, but the compile phase was failing with a Java heap space error. I fixed this by rerunning with JDK_JAVA_OPTIONS="-Xmx8G" prepended to the command line (no need to edit the gol script). Perhaps the sort phase would have gone faster with this too. Final GOL size is 209GB (from 80GB planet.osm.pbf) - gol info -t says ~110GB of free pages while gol info -f says total free is 10MB.

[clarisma]

12h for sort seems awfully slow. Is this running off an SSD? (Thermal throttling may be kicking in since the index files would be furiously swapping on an 8GB system.)
As for heap space, I learned that 500MB per thread (i.e. virtual core), not exceeding 50% of physical RAM, appears to be the sweet spot. (The capriciousness of JVM heap tuning is one of the reasons the v2 GOL Tool is a C++ rewrite).
The more concerning issue is the size of the resulting GOL -- even with the latest planet, it should come in under 100 GB. Do you still have the output from gol build? Could you post the output from gol check and gol info -f -t?

[pbsurf]

Probably just because it's a cheap VPS ($10/mo) - 12h is fine, since I don't need to rebuild regularly. So far I haven't found any missing data building tiles and there is plenty of disk space, so I have no complaints! If you do want to investigate more, I can add your public key.

Complete timeline:

• gol build -> heap space error during compile stage
• increased swapfile size
• renamed the partial planet.gol and ran gol build again -> empty gol file error
• named the file back to planet.gol and ran gol build again -> same heap space error
• remembered about java heap space settings (should have just read your readme more carefully, ha ha)
• ran gol build with -Xmx8G -> success!

Only have the output from the final run, but I don't recall anything standing out from the earlier runs:

Building planet.gol from planet-250317.osm.pbf using default settings...                                                     
Compiled 35892 tiles in 1h 15m
07:04:14 No exports for tip 000002, can't resolve relation/15954793 at 000000/00000620                                      
07:04:14 No exports for tip 000002, can't resolve relation/15025423 at 000000/00000650                                       
07:04:14 No exports for tip 000002, can't resolve relation/16762597 at 000000/000006D0                                       
07:04:14 No exports for tip 000002, can't resolve relation/16236117 at 000000/000007A4                                       
07:04:14 No exports for tip 000002, can't resolve relation/16445284 at 000000/000007CC                                       
Linked 35892 tiles in 3m 54s
Built planet.gol in 1h 19m       

gol check:

0009E000: 261974 pages of unreferenced data
40000000: 262123 pages of unreferenced data
80000000: 262122 pages of unreferenced data
C0000000: 524285 pages of unreferenced data
140000000: 262106 pages of unreferenced data
180000000: 262106 pages of unreferenced data
1C0000000: 786416 pages of unreferenced data
280000000: 524287 pages of unreferenced data
300000000: 262105 pages of unreferenced data
340000000: 262127 pages of unreferenced data
380000000: 262113 pages of unreferenced data
3C0000000: 262128 pages of unreferenced data
400000000: 262113 pages of unreferenced data
440000000: 786386 pages of unreferenced data
500000000: 524250 pages of unreferenced data
580000000: 262142 pages of unreferenced data
5C0000000: 262127 pages of unreferenced data
600000000: 262116 pages of unreferenced data
640000000: 262143 pages of unreferenced data
680000000: 262136 pages of unreferenced data
6C0000000: 262113 pages of unreferenced data
700000000: 262139 pages of unreferenced data
740000000: 262135 pages of unreferenced data
780000000: 786401 pages of unreferenced data
840000000: 786429 pages of unreferenced data
900000000: 262143 pages of unreferenced data
940000000: 262142 pages of unreferenced data
980000000: 262142 pages of unreferenced data
9C0000000: 262143 pages of unreferenced data
A00000000: 262143 pages of unreferenced data
A40000000: 262136 pages of unreferenced data
A80000000: 262112 pages of unreferenced data
AC0000000: 262129 pages of unreferenced data
B00000000: 786429 pages of unreferenced data
BC0000000: 262129 pages of unreferenced data
C00000000: 524251 pages of unreferenced data
C80000000: 524267 pages of unreferenced data
D00000000: 262142 pages of unreferenced data
D40000000: 262109 pages of unreferenced data
D80000000: 524256 pages of unreferenced data
E00000000: 262131 pages of unreferenced data
E40000000: 262112 pages of unreferenced data
E80000000: 524287 pages of unreferenced data
F00000000: 262129 pages of unreferenced data
F40000000: 786408 pages of unreferenced data
1000000000: 524269 pages of unreferenced data
1080000000: 262113 pages of unreferenced data
10C0000000: 524279 pages of unreferenced data
1140000000: 262104 pages of unreferenced data
1180000000: 262110 pages of unreferenced data
11C0000000: 262136 pages of unreferenced data
1200000000: 262112 pages of unreferenced data
1240000000: 524283 pages of unreferenced data
12C0000000: 262120 pages of unreferenced data
1300000000: 262135 pages of unreferenced data
1340000000: 262098 pages of unreferenced data
1380000000: 262140 pages of unreferenced data
13C0000000: 262140 pages of unreferenced data
1400000000: 262136 pages of unreferenced data
1440000000: 262141 pages of unreferenced data
1480000000: 786410 pages of unreferenced data
1540000000: 262079 pages of unreferenced data
1580000000: 262117 pages of unreferenced data
15C0000000: 262142 pages of unreferenced data
1600000000: 262045 pages of unreferenced data
1640000000: 262143 pages of unreferenced data
1680000000: 262136 pages of unreferenced data
16C0000000: 262138 pages of unreferenced data
1700000000: 262131 pages of unreferenced data
1740000000: 262080 pages of unreferenced data
1780000000: 524282 pages of unreferenced data
1800000000: 524259 pages of unreferenced data
1880000000: 262131 pages of unreferenced data
18C0000000: 262129 pages of unreferenced data
1900000000: 262040 pages of unreferenced data
1940000000: 262116 pages of unreferenced data
1980000000: 262052 pages of unreferenced data
19C0000000: 262132 pages of unreferenced data
1A00000000: 262135 pages of unreferenced data
1A40000000: 262051 pages of unreferenced data
1A80000000: 262097 pages of unreferenced data
1AC0000000: 262122 pages of unreferenced data
1B00000000: 262096 pages of unreferenced data
1B40000000: 262130 pages of unreferenced data
1B80000000: 261989 pages of unreferenced data
1BC0000000: 262043 pages of unreferenced data
1C00000000: 261964 pages of unreferenced data
1C40000000: 67019 pages of unreferenced data

gol info:

Tile set ID: 72a02b43-2bc7-4b08-8b05-51b82754ca56
35,892 tiles (all loaded)
Indexes: aerialway, aeroway, amenity, boundary, building, building:part, communication, craft, emergency, geological, healthcare, highway, historic, industrial, landuse, leisure, man_made, military, natural, office, place, power, railway, route, shop, telecom, tourism, waterway

Free block  Count      Total
====================================
      4 KB     14      56 KB    0.5%
      8 KB      7      56 KB    0.5%
     12 KB     12     144 KB    1.4%
     20 KB      4      80 KB    0.8%
     28 KB      1      28 KB    0.3%
     32 KB      7     224 KB    2.2%
     36 KB      7     252 KB    2.4%
     40 KB      1      40 KB    0.4%
     44 KB      3     132 KB    1.3%
     48 KB      3     144 KB    1.4%
     52 KB      5     260 KB    2.5%
     56 KB      3     168 KB    1.6%
     60 KB      7     420 KB    4.1%
     64 KB      3     192 KB    1.9%
     68 KB      2     136 KB    1.3%
     72 KB      2     144 KB    1.4%
     76 KB      5     380 KB    3.7%
     80 KB      1      80 KB    0.8%
     84 KB      2     168 KB    1.6%
     88 KB      3     264 KB    2.6%
     92 KB      1      92 KB    0.9%
     96 KB      2     192 KB    1.9%
    108 KB      3     324 KB    3.1%
    112 KB      7     784 KB    7.6%
    124 KB      6     744 KB    7.2%
    128 KB      7     896 KB    8.7%
    132 KB      2     264 KB    2.6%
    136 KB      2     272 KB    2.6%
    140 KB      3     420 KB    4.1%
    144 KB      1     144 KB    1.4%
    148 KB      1     148 KB    1.4%
    152 KB      4     608 KB    5.9%
    168 KB      1     168 KB    1.6%
    180 KB      1     180 KB    1.7%
    184 KB      2     368 KB    3.6%
    192 KB      1     192 KB    1.9%
    196 KB      1     196 KB    1.9%
    220 KB      1     220 KB    2.1%
    240 KB      2     480 KB    4.7%
    244 KB      1     244 KB    2.4%
------------------------------------
Total free    141  10,304 KB  100.0%

              Loaded   Count    Min KB      Max KB    Avg KB        Total KB
=====================================================================================
Tiles         35,892  35,892      4 KB  125,084 KB  2,780 KB   99,803,896 KB   45.66%
  Zoom 0           1       1  3,264 KB    3,264 KB  3,264 KB        3,264 KB    0.00%
  Zoom 3          53      53     16 KB   15,036 KB  2,148 KB      113,944 KB    0.05%
  Zoom 6       1,050   1,050      4 KB   20,532 KB  4,024 KB    4,225,284 KB    1.93%
  Zoom 9      16,764  16,764     16 KB   58,152 KB  3,544 KB   59,472,480 KB   27.21%
  Zoom 12     18,023  18,023    236 KB  125,084 KB  1,996 KB   35,988,920 KB   16.47%
  Purgatory        1       1      4 KB        4 KB      4 KB            4 KB    0.00%
GOL Metadata                                                          632 KB    0.00%
Free pages                                                    118,757,448 KB   54.34%
-------------------------------------------------------------------------------------
Total                                                         218,561,976 KB  100.00%

[pbsurf]

Actually also have the output from the second to last run:

Building planet.gol from planet-250317.osm.pbf using default settings...                                                     Java heap space%
java.lang.OutOfMemoryError: Java heap space
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.allocateTable(LongLongHashMap.java:1340) 
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.rehash(LongLongHashMap.java:1228)                      
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.rehashAndGrow(LongLongHashMap.java:1221)               
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.addKeyValueAtIndex(LongLongHashMap.java:772)           
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.put(LongLongHashMap.java:365)                          
        at com.geodesk.gol.compiler.FeatureTile.setCoordinates(FeatureTile.java:262)                                                 
        at com.geodesk.gol.compiler.Compiler$Task.readNodes(Compiler.java:154)                                                       
        at com.geodesk.gol.compiler.Compiler$Task.readTile(Compiler.java:360)                                                        
        at com.geodesk.gol.compiler.Compiler$Task.run(Compiler.java:461)                                                             
        at com.geodesk.gol.Processor$Worker.process(Processor.java:70)                                                               
        at com.geodesk.gol.Processor$Worker.run(Processor.java:90)                                                           
04:46:33 Interrupted, ending...
04:46:33 Failed, run() quits.
04:46:33 Interrupted, ending...
04:46:33 Interrupted, ending...
Internal error: java.lang.OutOfMemoryError: Java heap space
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space                                                              
        at com.geodesk.gol.Processor.run(Processor.java:172)
        at com.geodesk.gol.compiler.Compiler.compileAll(Compiler.java:521)                                                          
        at com.geodesk.gol.BuildCommand.compile(BuildCommand.java:429)                                                               
        at com.geodesk.gol.BuildCommand.perform(BuildCommand.java:159)                                                               
        at com.clarisma.common.cli.Application.run(Application.java:98)                                                              
        at com.geodesk.gol.GolTool.main(GolTool.java:48)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.allocateTable(LongLongHashMap.java:1340)               
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.rehash(LongLongHashMap.java:1228)                      
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.rehashAndGrow(LongLongHashMap.java:1221)               
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.addKeyValueAtIndex(LongLongHashMap.java:772)           
        at org.eclipse.collections.impl.map.mutable.primitive.LongLongHashMap.put(LongLongHashMap.java:365)                          
        at com.geodesk.gol.compiler.FeatureTile.setCoordinates(FeatureTile.java:262)                                                 
        at com.geodesk.gol.compiler.Compiler$Task.readNodes(Compiler.java:154)                                                       
        at com.geodesk.gol.compiler.Compiler$Task.readTile(Compiler.java:360)                                                        
        at com.geodesk.gol.compiler.Compiler$Task.run(Compiler.java:461)                                                             
        at com.geodesk.gol.Processor$Worker.process(Processor.java:70)                                                               
        at com.geodesk.gol.Processor$Worker.run(Processor.java:90)                                                           
04:46:33 Interrupted, ending...
04:46:34 Interrupted, ending...        

[clarisma]

Thanks for the logs! Based on gol check, the tiles themselves are fine, but it appears the GOL was not properly cleared when gol build failed due to out of memory, so the GOL contains unreferenced junk data. To free up space, you can run these commands:

gol save planet planet-tiles
gol load new-planet planet-tiles -n

new-planet.gol should come in under 100 GB.
Afterwards, you can delete the old GOL and the planet-tiles folder.

If your virtual server comes with lots of storage, it is likely using mechanical drives, so these will dramatically slow a process with heavy paging.

(The No exports errors are benign, these are caused by relations that have no actual members and hence cannot be placed spatially)

[clarisma]

Sorry, running a bit behind on the thread-safety issue (#15). Working on it this week.