Fix globbed read/import bugs #449

translatenix · 2024-04-24T19:28:42Z

This PR fixes several bugs related to globbed reads/imports and makes code improvements along the way.

Bug fixes:

Fix caching of globbed reads
- Glob pattern isn't guaranteed to be a constant
- Both glob resolution and ResolvedGlobElement.getPath() depend on the current module URI
Fix ClassCastException when reflecting on globbed import
Avoid creating an unbounded number of Truffle root nodes per globbed read/import

See commit messages for further details.

SharedMemberNode enables generating non-constant object members at run time without generating an unbounded number of Truffle root nodes.

Introduce VmObjectBuilder, a uniform way to build `VmObject`s whose `ObjectMember`s are determined at run time. Replace some manual object building code with VmObjectBuilder. Add some assertions to fix IntelliJ warnings.

- Leverage SharedMemberNode to have a single Truffle root node per globbed read/import expression instead of one root node per resolved glob element. - Remove caching in ReadGlobNode because it only works correctly if glob pattern is a string *constant*. - Remove caching in StaticReadNode (now ReadGlobElementNode/ImportGlobElementNode) because it seems unnecessary and the implementation doesn't look quite right.

Problem: The result of a globbed read depends not only on the glob pattern but also on the current module URI. However, ResourceManager does not take this into account when caching globbed reads, causing wrong results. Changes: - Cache globbed reads per read expression. This is also how globbed imports are cached, except that import URIs are constant. - Make ReadGlobNode and ImportGlobNode code as similar as possible. - Reduce code duplication by inheriting from AbstractReadNode. - Add some tests.

bioball

Overall, great work!!

I have some minor blocking comments, but overall this is a great improvement. Thanks so much for your PR.

bioball · 2024-05-06T22:38:13Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ImportGlobNode.java

+ currentModule.getOriginal(),
+ currentModule.getUri(),
+ globPattern);
+ var builder = new VmObjectBuilder();


Suggested change

var builder = new VmObjectBuilder();

var builder = new VmObjectBuilder(resolvedElements.size());

bioball · 2024-05-06T22:51:04Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ImportGlobNode.java

 private final String globPattern;
+ private final SharedMemberNode importGlobElementNode;


Should declare this @Child. Also, it's good to avoid allocating an extra node until it's needed.

Suggested change

private final SharedMemberNode importGlobElementNode;

@Child @LateInit private SharedMemberNode importGlobElementNode;

Add:

private SharedMemberNode getMemberNode() { if (importGlobElementNode == null) { CompilerDirectives.transferToInterpreterAndInvalidate(); var language = VmLanguage.get(this); importGlobElementNode = new SharedMemberNode( sourceSection, sourceSection, "", language, new FrameDescriptor(), new ImportGlobElementNode(sourceSection, language, currentModule)); } return importGlobElementNode; }

Then within executeGeneric:

-builder.addEntry(entry.getKey(), importGlobElementNode); +builder.addEntry(entry.getKey(), getMemberNode());

Same comment applies to ReadGlobNode.

bioball · 2024-05-06T22:57:43Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ReadGlobElementNode.java

+import org.pkl.core.util.GlobResolver.ResolvedGlobElement;
+
+/** Used by {@link ReadGlobNode}. */
+public class ReadGlobElementNode extends ExpressionNode {


"Element" clashes with our element/entry/property lexicon.

Maybe:

Suggested change

public class ReadGlobElementNode extends ExpressionNode {

public class ReadGlobMemberBodyNode extends ExpressionNode {

Same comment on the import side; let's rename ImportGlobElementNode to maybe ImportGlobMemberBodyNode

bioball · 2024-05-06T22:58:22Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ReadGlobElementNode.java

+ @TruffleBoundary
+ private Object readResource(VmObjectLike mapping, String path) {
+ @SuppressWarnings("unchecked")
+ var globElements = (Map<String, ResolvedGlobElement>) mapping.getExtraStorage();


Ahh, great idea! I didn't think about this, but this is much better. And this mirrors how we do this in the for generators too.

bioball · 2024-05-06T23:12:04Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ReadGlobElementNode.java

+ @TruffleBoundary
+ private Object readResource(VmObjectLike mapping, String path) {
+ @SuppressWarnings("unchecked")
+ var globElements = (Map<String, ResolvedGlobElement>) mapping.getExtraStorage();
+ var resourceUri = globElements.get(path).getUri();
+ var resource = VmContext.get(this).getResourceManager().read(resourceUri, this).orElse(null);
+ if (resource == null) {
+ throw exceptionBuilder().evalError("cannotFindResource", resourceUri).build();
+ }
+ return resource;


Let's get rid of this truffle boundary:

Suggested change

@TruffleBoundary

private Object readResource(VmObjectLike mapping, String path) {

@SuppressWarnings("unchecked")

var globElements = (Map<String, ResolvedGlobElement>) mapping.getExtraStorage();

var resourceUri = globElements.get(path).getUri();

var resource = VmContext.get(this).getResourceManager().read(resourceUri, this).orElse(null);

if (resource == null) {

throw exceptionBuilder().evalError("cannotFindResource", resourceUri).build();

}

return resource;

private Object readResource(VmObjectLike mapping, String path) {

@SuppressWarnings("unchecked")

var globElements = (Map<String, ResolvedGlobElement>) mapping.getExtraStorage();

var resourceUri = VmUtils.getMapValue(globElements, path).getUri();

var resource = VmContext.get(this).getResourceManager().read(resourceUri, this).orElse(null);

if (resource == null) {

CompilerDirectives.transferToInterpreter();

throw exceptionBuilder().evalError("cannotFindResource", resourceUri).build();

}

return resource;

This change will require VmUtils.getMapValue(), which should be as simple as:

@TruffleBoundary public static <K, V> V getMapValue(Map<K, V> map, K key) { return map.get(key); }

bioball · 2024-05-06T23:32:02Z

pkl-core/src/main/java/org/pkl/core/ast/expression/unary/ReadGlobNode.java

+ currentModule,
+ currentModule.getUri(),
+ globPattern);
+ var builder = new VmObjectBuilder();


Suggested change

var builder = new VmObjectBuilder();

var builder = new VmObjectBuilder(resolvedElements.size());

bioball · 2024-05-06T23:52:16Z

pkl-core/src/main/java/org/pkl/core/runtime/ResourceManager.java

- /**
- * Resolves the glob URI into a set of URIs.
- *
- * <p>The glob URI must be absolute. For example: {@code "file:///foo/bar/*.pkl"}.


Ah, this is an artifact of how globbing was first implemented (glob patterns expanded to their absolute path). Moving away from that yields the bug that you noticed. We should have added an assertion here.

bioball · 2024-05-07T00:06:27Z

pkl-core/src/main/java/org/pkl/core/runtime/VmMap.java

+ @TruffleBoundary
+ public VmMapping toMapping() {
+ var builder = new VmObjectBuilder(keyOrder.size());
+ for (var key : keyOrder) {
+ var value = map.get(key);
+ assert value != null;
+ builder.addEntry(key, value);
+ }
+ return builder.toMapping();
+ }
+
+ @TruffleBoundary
+ public VmDynamic toDynamic() {
+ var builder = new VmObjectBuilder(keyOrder.size());
+ for (var key : keyOrder) {
+ var value = map.get(key);
+ assert value != null;
+ if (key instanceof String) {
+ builder.addProperty(Identifier.get((String) key), value);
+ } else {
+ builder.addEntry(key, value);
+ }
+ }
+ return builder.toDynamic();
+ }
+


This is a replacement for code that was visible to Truffle inlining, so it shouldn't be then moved behind a TruffleBoundary.

Here is an implementation of these two methods in a way that can be inlined:

Suggested change

@TruffleBoundary

public VmMapping toMapping() {

var builder = new VmObjectBuilder(keyOrder.size());

for (var key : keyOrder) {

var value = map.get(key);

assert value != null;

builder.addEntry(key, value);

}

return builder.toMapping();

}

@TruffleBoundary

public VmDynamic toDynamic() {

var builder = new VmObjectBuilder(keyOrder.size());

for (var key : keyOrder) {

var value = map.get(key);

assert value != null;

if (key instanceof String) {

builder.addProperty(Identifier.get((String) key), value);

} else {

builder.addEntry(key, value);

}

}

return builder.toDynamic();

}

public VmMapping toMapping() {

var builder = new VmObjectBuilder(getLength());

for (var entry : this) {

builder.addEntry(VmUtils.getKey(entry), VmUtils.getValue(entry));

}

return builder.toMapping();

}

public VmDynamic toDynamic() {

var builder = new VmObjectBuilder(getLength());

for (var entry : this) {

var key = VmUtils.getKey(entry);

var value = VmUtils.getValue(entry);

if (key instanceof String) {

builder.addProperty(Identifier.get((String) key), value);

} else {

builder.addEntry(key, value);

}

}

return builder.toDynamic();

}

bioball · 2024-05-07T00:08:01Z

pkl-core/src/main/java/org/pkl/core/stdlib/base/MapNodes.java

- VmUtils.createEmptyMaterializedFrame(),
- BaseModule.getMappingClass().getPrototype(),
- members);
+ return self.toMapping();


Should change class toDynamic to also call out to self.toDynamic() yeah?

bioball · 2024-05-07T00:21:59Z

pkl-core/src/main/java/org/pkl/core/util/GlobResolver.java

@@ -463,15 +465,15 @@ private static Pair<String, String[]> splitGlobPatternIntoBaseAndWildcards(
 * <p>Each pair is the expanded form of the glob pattern, paired with its resolved absolute URI.
 */
 @TruffleBoundary
- public static List<ResolvedGlobElement> resolveGlob(
+ public static Map<String, ResolvedGlobElement> resolveGlob(


Why did this change from List to Map?

translatenix force-pushed the fix-glob branch from d366767 to ba883a4 Compare April 25, 2024 01:54

translatenix added 7 commits April 25, 2024 13:02

Split MemberNode into (Regular/Shared)MemberNode

80da58f

SharedMemberNode enables generating non-constant object members at run time without generating an unbounded number of Truffle root nodes.

Invert shouldRunTypeCheck to match its name

6779fbe

Introduce VmObjectBuilder

16f2c3a

Introduce VmObjectBuilder, a uniform way to build `VmObject`s whose `ObjectMember`s are determined at run time. Replace some manual object building code with VmObjectBuilder. Add some assertions to fix IntelliJ warnings.

Simplify code

88b69cc

Fix ClassCastException when reflecting on globbed import

dd04e7c

translatenix force-pushed the fix-glob branch from ba883a4 to f40ef4e Compare April 25, 2024 20:11

This was referenced May 4, 2024

Non-constant globbed reads return wrong results #474

Open

Reads with same glob pattern return same result even if located in different modules/directories #475

Open

bioball requested changes May 7, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix globbed read/import bugs #449

Fix globbed read/import bugs #449

translatenix commented Apr 24, 2024 •

edited

bioball left a comment

bioball May 6, 2024

bioball May 6, 2024 •

edited

bioball May 6, 2024

bioball May 6, 2024

bioball May 6, 2024

bioball May 6, 2024

bioball May 6, 2024

bioball May 7, 2024 •

edited

bioball May 7, 2024

bioball May 7, 2024

	var builder = new VmObjectBuilder();
	var builder = new VmObjectBuilder(resolvedElements.size());

		private final String globPattern;
		private final SharedMemberNode importGlobElementNode;

	private final SharedMemberNode importGlobElementNode;
	@Child @LateInit private SharedMemberNode importGlobElementNode;

	public class ReadGlobElementNode extends ExpressionNode {
	public class ReadGlobMemberBodyNode extends ExpressionNode {

Fix globbed read/import bugs #449

Are you sure you want to change the base?

Fix globbed read/import bugs #449

Conversation

translatenix commented Apr 24, 2024 • edited

bioball left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bioball May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bioball May 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

translatenix commented Apr 24, 2024 •

edited

bioball May 6, 2024 •

edited

bioball May 7, 2024 •

edited