Tilemaker Reading Error #592

ArmyGuy62 · 2023-11-24T17:24:38Z

ArmyGuy62
Nov 24, 2023

I have followed the setup for Ubuntu (I am using Ubuntu 22.04)
Everything appears to have installed properly but when I run the following command I get this unexpected error
john@Tiler22:/data/dump/on_mbtiles$ tilemaker --input /data/dump/ontario-latest.osm.pbf --output /data/dump/on_mbtiles/ontario.mbtiles
Couldn't open .json config: config.json
Am I missing a configuration?
TIA

Answered by systemed

Nov 24, 2023

You need to either have config.json in your current directory, or point to a JSON config file somewhere with the --config argument. Same applies for the Lua processing script. Typically you might use something like

tilemaker --input my.osm.pbf --output my.mbtiles \
    --config ~/tilemaker/resources/config-openmaptiles.json \
    --process ~/tilemaker/resources/process-openmaptiles.lua

View full answer

systemed · 2023-11-24T17:33:34Z

systemed
Nov 24, 2023
Maintainer

You need to either have config.json in your current directory, or point to a JSON config file somewhere with the --config argument. Same applies for the Lua processing script. Typically you might use something like

tilemaker --input my.osm.pbf --output my.mbtiles \
    --config ~/tilemaker/resources/config-openmaptiles.json \
    --process ~/tilemaker/resources/process-openmaptiles.lua

5 replies

ArmyGuy62 Nov 24, 2023
Author

Thanks for that.....makes sense
I do have one more question, do the shapefiles need to be seated in a particular folder or are they included in the download?
My .shp file error:
Reading .shp ocean
Unable to open coastline/water_polygons.shp or coastline/water_polygons.SHP.
Reading .shp urban_areas
Unable to open landcover/ne_10m_urban_areas/ne_10m_urban_areas.shp or landcover/ne_10m_urban_areas/ne_10m_urban_areas.SHP.
Reading .shp ice_shelf
Unable to open landcover/ne_10m_antarctic_ice_shelves_polys/ne_10m_antarctic_ice_shelves_polys.shp or landcover/ne_10m_antarctic_ice_shelves_polys/ne_10m_antarctic_ice_shelves_polys.SHP.
Reading .shp glacier
Unable to open landcover/ne_10m_glaciated_areas/ne_10m_glaciated_areas.shp or landcover/ne_10m_glaciated_areas/ne_10m_glaciated_areas.SHP.
Sorting loaded shapes

Thanks again

ArmyGuy62 Nov 24, 2023
Author

Also, is this normal. It seemed to be going great and then stopped

Reading .pbf /data/dump/ontario-latest.osm.pbf
Block 13703/14586 ways 0 relations 0
Sorting nodes
Block 14574/14586 ways 4879 relations 0
Sorting ways
Sorting generated geometriesations 8000
Stored 109636291 nodes, 552681 ways, 0 relations
Shape points: 0, lines: 0, polygons: 0
Generated points: 2354032, lines: 3830815, polygons: 3674247
Zoom level 14, writing tile 732420 of 733420

Sorry for all the questions

cldellow Nov 24, 2023

Hello fellow (?) Ontarian --

is this normal. It seemed to be going great and then stopped

I also ran into this recently. The Ontario pbf extract has a geometry in it that is broken in some sense.

Tilemaker tries to automatically fix broken geometries, but sometimes the fix can take a very long time. I think if you keep waiting, it will eventually complete - possibly after several minutes, if memory serves. (Perhaps one day Tilemaker could be faster, or could skip geometries that take longer than some amount of time to fix.)

I do have one more question, do the shapefiles need to be seated in a particular folder or are they included in the download?

The shapefiles need to be downloaded separately. The process for getting the coastline shapefiles is described here: https://github.com/systemed/tilemaker#out-of-the-box-setup

I don't think the process for getting the landcover shapefiles is documented yet. They're provided by a third-party organization called Natural Earth, if you google each one's name (eg ne_10m_urban_areas), you should find a download page that gives you a zipfile that can be unzipped.

If you do figure it out, it might be nice to document it, either just here in the discussion, or via a PR that updates the README file.

ArmyGuy62 Nov 24, 2023
Author

Yes, in Ottawa
It's been running about an hour now so maybe I will start again with a different .pbf
I will have a look at the link and try and sort that out.
I do have a copy of all of the shapefiles from when I ran the PBF through Imposm and then rendered in MapServer
Thanks for the leads

systemed Nov 27, 2023
Maintainer

Have a try with the latest code - I've reverted a change which seemed to be causing this. Sorry for the hassle.

ArmyGuy62 · 2023-11-27T14:00:54Z

ArmyGuy62
Nov 27, 2023
Author

That’s awesome Richard, I will give it a try this morning. Also, do the shapefiles have to be in a certain place for the tilemaker command to see them. John

…

On Mon, Nov 27, 2023 at 8:25 AM Richard Fairhurst ***@***.***> wrote: Have a try with the latest code - I've reverted a change which seemed to be causing this. Sorry for the hassle. — Reply to this email directly, view it on GitHub <#592 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRHMFLATHTAMB7CRGJVE33YGSIDNAVCNFSM6AAAAAA7ZM3BLSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOBQHA3DE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

2 replies

systemed Nov 27, 2023
Maintainer

The config JSON looks in coastline/water_polygons.shp, so that's where they should be (relative path not absolute) unless you change the config.

ArmyGuy62 Nov 27, 2023
Author

Looks like everything is working....thanks for your help. I do have other questions but proof of concept is a winner
john@Tiler22:/data/Tilemaker$ tilemaker --input /data/dump/ontario-latest.osm.pbf --output /data/dump/on_mbtiles/ontario.mbtiles --config /data/Tilemaker/resources/config-openmaptiles.json --process /data/Tilemaker/resources/process-openmaptiles.lua --merge
Layer place (z0-14)
Layer boundary (z0-14)
Layer poi (z12-14)
Layer poi_detail (z14-14) -> poi
Layer housenumber (z14-14)
Layer waterway (z8-14)
Layer waterway_detail (z12-14) -> waterway
Layer transportation (z4-14)
Layer transportation_name (z8-14)
Layer transportation_name_mid (z12-14) -> transportation_name
Layer transportation_name_detail (z14-14) -> transportation_name
Layer building (z13-14)
Layer water (z6-14)
Layer ocean (z0-14) -> water
Layer water_name (z14-14)
Layer water_name_detail (z14-14) -> water_name
Layer aeroway (z11-14)
Layer aerodrome_label (z10-14)
Layer park (z11-14)
Layer landuse (z4-14)
Layer urban_areas (z4-8) -> landuse
Layer landcover (z0-14)
Layer ice_shelf (z0-9) -> landcover
Layer glacier (z2-9) -> landcover
Layer mountain_peak (z11-14)
Bounding box -95.1596, 41.6377, -74.31, 57.5083
Reading .shp ocean
Reading .shp urban_areas
Reading .shp ice_shelf
Reading .shp glacier
Sorting loaded shapes
Reading .pbf /data/dump/ontario-latest.osm.pbf
Block 13703/14586 ways 0 relations 0
Sorting nodes
Block 14573/14586 ways 8000 relations 0
Sorting ways
Sorting generated geometriesations 8000
Stored 109636291 nodes, 552681 ways, 0 relations
Shape points: 0, lines: 0, polygons: 287
Generated points: 2354032, lines: 3830815, polygons: 3674245
Zoom level 14, writing tile 830640 of 830640
Filled the tileset with good things at /data/dump/on_mbtiles/ontario.mbtiles

ArmyGuy62 · 2023-11-27T18:11:45Z

ArmyGuy62
Nov 27, 2023
Author

Hi Richard So I ran into the same problem with another .pbf file North America is the PBF and my ultimate goal is the Planet PBF Here is the error: ***@***.***:/data/Tilemaker/server$ tilemaker --input /data/dump/north-america-latest.osm.pbf --output /data/dump/on_mbtiles/na.mbtiles --config /data/Tilemaker/resources/config-openmaptiles.json --process /data/Tilemaker/resources/process-openmaptiles.lua --verbose Layer place (z0-14) Layer boundary (z0-14) Layer poi (z12-14) Layer poi_detail (z14-14) -> poi Layer housenumber (z14-14) Layer waterway (z8-14) Layer waterway_detail (z12-14) -> waterway Layer transportation (z4-14) Layer transportation_name (z8-14) Layer transportation_name_mid (z12-14) -> transportation_name Layer transportation_name_detail (z14-14) -> transportation_name Layer building (z13-14) Layer water (z6-14) Layer ocean (z0-14) -> water Layer water_name (z14-14) Layer water_name_detail (z14-14) -> water_name Layer aeroway (z11-14) Layer aerodrome_label (z10-14) Layer park (z11-14) Layer landuse (z4-14) Layer urban_areas (z4-8) -> landuse Layer landcover (z0-14) Layer ice_shelf (z0-9) -> landcover Layer glacier (z2-9) -> landcover Layer mountain_peak (z11-14) Bounding box -180, 5.57228, 180, 85.0418 Reading .shp ocean Shapefile entity #3017 type 5 is invalid. Parts:28. Reason:Geometry has invalid self-intersections. A self-intersection point was found at (-110.642, 25.7673); method: i; operations: u/i; segment IDs {source, multi, ring, segment}: {0, 0, -1, 214}/{0, 0, -1, 216}... failed to correct. Reason: Geometry has invalid self-intersections. A self-intersection point was found at (-110.642, 25.7673); method: i; operations: u/i; segment IDs {source, multi, ring, segment}: {0, 0, -1, 214}/{0, 0, -1, 216} Shapefile entity #9230 type 5 is invalid. Parts:223. Reason:Geometry has invalid self-intersections. A self-intersection point was found at (-71.4807, 45.5092); method: i; operations: i/u; segment IDs {source, multi, ring, segment}: {0, 0, -1, 30261}/{0, 0, -1, 30263}... failed to correct. Reason: Geometry has invalid self-intersections. A self-intersection point was found at (-71.4807, 45.5092); method: i; operations: i/u; segment IDs {source, multi, ring, segment}: {0, 0, -1, 30261}/{0, 0, -1, 30263} Shapefile entity #10693 type 5 is invalid. Parts:3163. Reason:Geometry has invalid self-intersections. A self-intersection point was found at (9.87713, 83.8271); method: i; operations: i/u; segment IDs {source, multi, ring, segment}: {0, 0, 2915, 3}/{0, 0, 2915, 5}... failed to correct. Reason: Geometry has invalid self-intersections. A self-intersection point was found at (9.87713, 83.8271); method: i; operations: i/u; segment IDs {source, multi, ring, segment}: {0, 0, 2915, 3}/{0, 0, 2915, 5} Reading .shp urban_areas Reading .shp ice_shelf Reading .shp glacier Shapefile entity #360 type 5 is invalid. Parts:7. Reason:Geometry has invalid self-intersections. A self-intersection point was found at (-152.169, 78.8039); method: t; operations: u/u; segment IDs {source, multi, ring, segment}: {0, 0, 0, 5}/{0, 0, 0, 87}... corrected Shapefile entity #1875 type 5 is invalid. Parts:54. Reason:Geometry has invalid self-intersections. A self-intersection point was found at (-141.942, 76.9594); method: t; operations: u/u; segment IDs {source, multi, ring, segment}: {0, 0, -1, 2543}/{0, 0, -1, 2505}... corrected Sorting loaded shapes Reading .pbf /data/dump/north-america-latest.osm.pbf Killed62133/242025 ways 0 relations 0

…

On Mon, Nov 27, 2023 at 9:26 AM Richard Fairhurst ***@***.***> wrote: The config JSON looks in coastline/water_polygons.shp, so that's where they should be (relative path not absolute) unless you change the config. — Reply to this email directly, view it on GitHub <#592 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRHMFMVYWUXYLMISVS2MDLYGSPH7AVCNFSM6AAAAAA7ZM3BLSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOBRGU2TG> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

systemed Nov 27, 2023
Maintainer

That's because you don't have enough RAM to do North America, I'm afraid. You can try using the --store option pointing to somewhere on an SSD, which will reduce RAM usage and use more on-disk - e.g. --store /path/to/my/ssd/temp/. You'll still need a lot of RAM for North America even then though.

ArmyGuy62 · 2023-11-30T20:27:53Z

ArmyGuy62
Nov 30, 2023
Author

Just wanted to say thanks for all the tips and tricks. Got North America completed in about 13 hours

…

On Mon, Nov 27, 2023, 1:55 p.m. Richard Fairhurst ***@***.***> wrote: That's because you don't have enough RAM to do North America, I'm afraid. You can try using the --store option pointing to somewhere on an SSD, which will reduce RAM usage and use more on-disk - e.g. --store /path/to/my/ssd/temp/. You'll still need a lot of RAM for North America even then though. — Reply to this email directly, view it on GitHub <#592 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRHMFPGUWVXMKQBGUVKLA3YGTOZ7AVCNFSM6AAAAAA7ZM3BLSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOBUGQ2DM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tilemaker Reading Error #592

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Tilemaker Reading Error #592

ArmyGuy62 Nov 24, 2023

Replies: 4 comments · 8 replies

systemed Nov 24, 2023 Maintainer

ArmyGuy62 Nov 24, 2023 Author

ArmyGuy62 Nov 24, 2023 Author

cldellow Nov 24, 2023

ArmyGuy62 Nov 24, 2023 Author

systemed Nov 27, 2023 Maintainer

ArmyGuy62 Nov 27, 2023 Author

systemed Nov 27, 2023 Maintainer

ArmyGuy62 Nov 27, 2023 Author

ArmyGuy62 Nov 27, 2023 Author

systemed Nov 27, 2023 Maintainer

ArmyGuy62 Nov 30, 2023 Author

ArmyGuy62
Nov 24, 2023

Replies: 4 comments 8 replies

systemed
Nov 24, 2023
Maintainer

ArmyGuy62 Nov 24, 2023
Author

ArmyGuy62 Nov 24, 2023
Author

ArmyGuy62 Nov 24, 2023
Author

systemed Nov 27, 2023
Maintainer

ArmyGuy62
Nov 27, 2023
Author

systemed Nov 27, 2023
Maintainer

ArmyGuy62 Nov 27, 2023
Author

ArmyGuy62
Nov 27, 2023
Author

systemed Nov 27, 2023
Maintainer

ArmyGuy62
Nov 30, 2023
Author