Add experimental support for lua profiles #736
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://github.com/onthegomap/planetiler/actions/runs/7141262792 ℹ️ Base Logs ed4c320ℹ️ This Branch Logs c4b18d2 |
|
I ran a performance test of a lua profile vs. equivalent 1-file java profile: Lua: 10m6s total, OSM pass 2: 6m29s (2h50m CPU time), nodes took 4m25s cpu:2h13m Lua-- java -Xmx30g -jar planetiler.jar power.lua --area=planet --bounds=planet --storage=mmap --nodemap-type=array --output=power-lua.pmtiles --osm-path=planet-latest.osm.pbf
planetiler.output.name = "Power"
planetiler.output.path = { "data", "power.pmtiles" }
local area = planetiler.args:get_string("area", "geofabrik area to download", "massachusetts")
planetiler.add_source('osm', {
type = 'osm',
url = 'geofabrik:' .. area,
-- any java method or field that takes a Path can be called with a list of path parts from lua
path = { 'data', 'sources', area .. '.osm.pbf' }
})
function planetiler.process_feature(source, features)
if source:can_be_line() and source:has_tag("power", "line") then
features
:line("power")
:set_min_zoom(7)
:inherit_attr_from_source("power")
:inherit_attr_from_source("voltage")
:inherit_attr_from_source("cables")
:inherit_attr_from_source("operator")
elseif source:isPoint() and source:has_tag("power", "pole") then
features
:point("power")
:set_min_zoom(13)
:inherit_attr_from_source("power")
:inherit_attr_from_source("ref")
:inherit_attr_from_source("height")
:inherit_attr_from_source("operator")
end
endJava// java -Xmx30g -cp planetiler.jar Power.java --area=planet --bounds=planet --storage=mmap --nodemap-type=array --output=power-java.pmtiles --osm-path=planet-latest.osm.pbf
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.Planetiler;
import com.onthegomap.planetiler.Profile;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.reader.SourceFeature;
import java.nio.file.Path;
class Power implements Profile {
public void processFeature(SourceFeature source, FeatureCollector features) {
if (source.canBeLine() && source.hasTag("power", "line")) {
features
.line("power")
.setMinZoom(7)
.inheritAttrFromSource("power")
.inheritAttrFromSource("voltage")
.inheritAttrFromSource("cables")
.inheritAttrFromSource("operator");
} else if (source.isPoint() && source.hasTag("power", "pole")) {
features
.point("power")
.setMinZoom(13)
.inheritAttrFromSource("power")
.inheritAttrFromSource("ref")
.inheritAttrFromSource("height")
.inheritAttrFromSource("operator");
}
}
public String name() {
return "power";
}
public static void main(String[] argv) throws Exception {
Arguments args = Arguments.fromArgsOrConfigFile(argv);
var area = args.getString("area", "geofabrik area to download", "massachusetts");
Planetiler.create(args)
.setProfile(new Power())
.addOsmSource("osm", Path.of("data/sources/" + area + ".osm.pbf"), "geofabrik:" + area)
.overwriteOutput(Path.of("data/buildings.pmtiles"))
.run();
}
} |
|
Here's a comparison of tradeoffs between different ways to write a profile: https://gist.github.com/msbarry/284eff36489ca736c7dab836ace39d24
|
|
Added lua type definition generator you can run with Lua with lua extension with type definitons lua-profile-h264.mp41-file Java with Java extension pack but no other build tools: java-profile-h264.mp4 |
|
I made an example vscode project you can clone if you want to try this out: https://github.com/msbarry/planetiler-profiles |
|
Kudos, SonarCloud Quality Gate passed! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add experimental support for lua profiles. To use it, run
java -jar planetiler.jar profile.luawith a script like this:See
planetiler-experimental/src/test/resourcesfor more examples, andLuaEnvironmentfor the full set of java utilities exported to lua. Lua has access to the full java API through reflection so anything should be possible, but we can write more convenient utilities when people discover useful patterns that we want to make easier to use.IDE Support
VS Code has good lua support with the lua extension. Run
java -jar planetiler.jar lua-types > types.luato generate a type definition file in the same directory as your profile to provide autocomplete and validation as you write profiles:Testing Lua Profiles
You can write a yaml file like this to test that your profile produces the expected output vector tile features for a given input source (ie. OSM) feature:
This is the same format used for yaml schema tests, so this PR promotes that code to planetiler-core and defines a lua implementation you can use as well.
Please give feedback!
To set up a local workspace to try this out with VS Code:
Lua profiles are an alternative to single-file java profiles that you can run without any build tools with
java -cp planetiler.jar Profile.java. We should only merge lua profile support if people think that lua profiles provide enough usability benefit over single-file java profiles to warrant the extra maintenance.If we end up merging this, followups listed in #738. Please let me know what you think!