Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[K/N] Expose program name in runtime #5281

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1ff362a
[K/N] Expose program name in runtime
vonox7 Mar 21, 2024
05d57ee
Merge branch 'refs/heads/master' into slawicek/program-name
vonox7 Apr 29, 2024
f73045e
Move programName testcase to new test infrastructure
vonox7 Apr 29, 2024
fb67427
Add testcase for calling Platform.getProgramName from within a library
vonox7 Apr 30, 2024
2703f15
Execute Kotlin executable via execv to pass custom programNames
vonox7 Apr 30, 2024
3b66b6b
Support argc=0
vonox7 Apr 30, 2024
285dd67
Merge branch 'JetBrains:master' into slawicek/program-name
vonox7 Apr 30, 2024
69e660f
Use proper executor
vonox7 May 15, 2024
a69b027
Merge remote-tracking branch 'vali/slawicek/program-name' into slawic…
vonox7 May 15, 2024
6d07cf2
Remove unneeded gradle task
vonox7 May 15, 2024
2fcf1f0
Ignore different windows path & line-endings
vonox7 May 15, 2024
3c933a7
Let linux behave the same as windows/macos
vonox7 May 26, 2024
802c8e7
Add empty program name validation
vonox7 May 26, 2024
64ffcfc
Fix testcase on windows
vonox7 May 26, 2024
2d44f5f
Add comment for argv[0] == "" check
vonox7 Jun 3, 2024
06b0d67
Add explicit return
vonox7 Jun 3, 2024
27d7688
Set return value to 1 in case of program fail
vonox7 Jun 3, 2024
fba8ce4
Remove unneeded include
vonox7 Jun 3, 2024
0aa4600
Copy programName so no need to reset it
vonox7 Jun 4, 2024
7fb104f
Merge branch 'JetBrains:master' into slawicek/program-name
vonox7 Jun 4, 2024
7868f85
Print errno for easier exec debugging
vonox7 Jun 4, 2024
6323590
Merge branch 'JetBrains:master' into slawicek/program-name
vonox7 Jun 4, 2024
e39351d
Add missing import for linux
vonox7 Jun 4, 2024
fd2ebbf
Try to update qemu
vonox7 Jun 6, 2024
d09c9f0
Revert "Try to update qemu"
vonox7 Jun 6, 2024
841f6e0
Remove include
vonox7 Jun 8, 2024
ee0ed60
Add include back
vonox7 Jun 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 7 additions & 1 deletion kotlin-native/runtime/src/launcher/cpp/launcher.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,12 @@ using namespace kotlin;
//--- Setup args --------------------------------------------------------------//

OBJ_GETTER(setupArgs, int argc, const char** argv) {
if (argc > 0) {
kotlin::programName = argv[0];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about doing kotlin::programName = strndup(argv[0], 4096)? No need to free the result, and we don't have to worry about lifetimes. 4096 is just some random not-too-small not-too-big number to protect us from argv[0] being too large. Technically, the OS has some limits already, but might as well protect ourselves.

Copy link
Contributor Author

@vonox7 vonox7 Jun 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I applied your idea.

}

// The count is one less, because we skip argv[0] which is the binary name.
ObjHeader* result = AllocArrayInstance(theArrayTypeInfo, argc - 1, OBJ_RESULT);
ObjHeader* result = AllocArrayInstance(theArrayTypeInfo, std::max(0, argc - 1), OBJ_RESULT);
ArrayHeader* array = result->array();
for (int index = 1; index < argc; index++) {
ObjHolder result;
Expand All @@ -58,6 +62,8 @@ extern "C" RUNTIME_USED int Init_and_run_start(int argc, const char** argv, int
Kotlin_shutdownRuntime();
}

kotlin::programName = nullptr; // argv[0] might not be valid after this point

return exitStatus;
}

Expand Down
11 changes: 11 additions & 0 deletions kotlin-native/runtime/src/main/cpp/Runtime.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,8 @@ bool kotlin::initializeGlobalRuntimeIfNeeded() noexcept {
return true;
}

const char* kotlin::programName = nullptr;

extern "C" {

RUNTIME_NOTHROW void AppendToInitializersTail(InitNode *next) {
Expand Down Expand Up @@ -335,6 +337,15 @@ KBoolean Konan_Platform_isFreezingEnabled() {
return kotlin::compiler::freezingChecksEnabled();
}

OBJ_GETTER0(Konan_Platform_getProgramName) {
if (kotlin::programName == nullptr) {
// null in case Platform.getProgramName is called from within a library and the main function of the binary is not built with Kotlin
RETURN_OBJ(nullptr)
} else {
RETURN_RESULT_OF(CreateStringFromCString, kotlin::programName)
}
}

bool Kotlin_memoryLeakCheckerEnabled() {
return g_checkLeaks;
}
Expand Down
3 changes: 3 additions & 0 deletions kotlin-native/runtime/src/main/cpp/Runtime.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

#include "Porting.h"
#include "Memory.h"
#include "KString.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems unused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.


#ifdef __cplusplus
extern "C" {
Expand Down Expand Up @@ -50,6 +51,8 @@ namespace kotlin {
// Returns `true` if initialized.
bool initializeGlobalRuntimeIfNeeded() noexcept;

extern const char* programName;

}

#endif // RUNTIME_RUNTIME_H
10 changes: 10 additions & 0 deletions kotlin-native/runtime/src/main/kotlin/kotlin/native/Platform.kt
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,13 @@ public object Platform {
public val isFreezingEnabled: Boolean
get() = Platform_isFreezingEnabled()

/**
* Representation of the name used to invoke the program executable.
* [null] if the Kotlin code was compiled to a native library and the executable is not a Kotlin program.
*/
public val programName: String?
get() = Platform_getProgramName()
SvyatoslavScherbina marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qurbonzoda could you please review the stdlib change?


/**
* If the memory leak checker is activated, by default `true` in debug mode, `false` in release.
* When memory leak checker is activated, and leak is detected during last Kotlin context
Expand Down Expand Up @@ -161,6 +168,9 @@ private external fun Platform_isDebugBinary(): Boolean
@GCUnsafeCall("Konan_Platform_isFreezingEnabled")
private external fun Platform_isFreezingEnabled(): Boolean

@GCUnsafeCall("Konan_Platform_getProgramName")
private external fun Platform_getProgramName(): String?

@GCUnsafeCall("Konan_Platform_getMemoryLeakChecker")
private external fun Platform_getMemoryLeakChecker(): Boolean

Expand Down
1 change: 1 addition & 0 deletions native/native.tests/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ val stdlibK2Test = nativeTest("stdlibK2Test", "stdlib & frontend-fir")
val kotlinTestLibraryTest = nativeTest("kotlinTestLibraryTest", "kotlin-test & !frontend-fir")
val kotlinTestLibraryK2Test = nativeTest("kotlinTestLibraryK2Test", "kotlin-test & frontend-fir")
val partialLinkageTest = nativeTest("partialLinkageTest", "partial-linkage")
val programNameTest = nativeTest("programNameTest", "program-name")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit redudant. I mean, running these tests separately shouldn't happen frequently, so adding a Gradle task is overkill.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've removed it.

val cinteropTest = nativeTest("cinteropTest", "cinterop")
val debuggerTest = nativeTest("debuggerTest", "debugger")
val cachesTest = nativeTest("cachesTest", "caches")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#include <stdio.h>
#include "programName_api.h"

int main() {
programName();
fflush(NULL);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add an explicit return 0;?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, done

}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Platform.programName is null within library
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
@file:OptIn(kotlin.experimental.ExperimentalNativeApi::class)

@CName("programName")
fun programName() {
println("Platform.programName is " + Platform.programName + " within library")
}
11 changes: 11 additions & 0 deletions native/native.tests/testData/programName/kotlinPrintEntryPoint.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
@file:OptIn(kotlin.experimental.ExperimentalNativeApi::class)

import kotlin.native.Platform

fun main(args: Array<String>) {
// Remove path and extension (.kexe or .exe)
val programFileName = Platform.programName?.substringAfterLast("/")?.substringBeforeLast(".")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test fails on Windows, because it uses \ instead of /.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you. The sanitization is now platform agnostic.


println("programName: $programFileName")
println("args: ${args.joinToString()}")
}
14 changes: 14 additions & 0 deletions native/native.tests/testData/programName/main.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
printf("calling exec...\n");
fflush(NULL);

// Kotlin executable name is in argv[1]
// Forward argv[2..n] to kotlin executable as arguments (the first one should be the programName according to posix)
execv(argv[1], &(argv[2]));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, execv requires that the list of arguments must be terminated by a NULL pointer. Is it always guaranteed here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

argv is an array of size argc + 1 and the last member is always NULL: https://en.cppreference.com/w/c/language/main_function


printf("exec failed\n");
return 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's return non-zero exit code?

Copy link
Contributor Author

@vonox7 vonox7 Jun 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I also now print the errno for easier debugging in case the exec syscall fails.

}
11 changes: 11 additions & 0 deletions native/native.tests/testData/standalone/entryPoint/programName.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
@file:OptIn(kotlin.experimental.ExperimentalNativeApi::class)

import kotlin.native.Platform
import kotlin.test.*

fun main(args: Array<String>) {
// Remove path and extension (.kexe or .exe)
val programFileName = Platform.programName!!.substringAfterLast("/").substringBeforeLast(".")

assertEquals("standalone_entryPoint_programName", programFileName)
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package org.jetbrains.kotlin.konan.test.blackbox

import org.jetbrains.kotlin.konan.test.blackbox.support.*
import org.jetbrains.kotlin.konan.test.blackbox.support.compilation.TestCompilationResult.Companion.assertSuccess
import org.jetbrains.kotlin.konan.test.blackbox.support.util.ClangDistribution
import org.jetbrains.kotlin.konan.test.blackbox.support.util.compileWithClang
import org.jetbrains.kotlin.native.executors.runProcess
import org.junit.jupiter.api.Tag
import org.junit.jupiter.api.Test
import java.io.File
import kotlin.test.assertEquals
import kotlin.time.Duration.Companion.seconds

@Tag("program-name")
class ProgramNameTest : AbstractNativeSimpleTest() {

@Test
fun programNameTest() {
// 1. Compile main.c to main.cexe

val cExecutable = buildDir.resolve("main.cexe")
compileWithClang(
clangDistribution = ClangDistribution.Llvm,
sourceFiles = listOf(sourceDir.resolve("main.c")),
outputFile = cExecutable,
additionalClangFlags = listOf("-Wall", "-Werror"),
).assertSuccess()

// 2. Compile kotlinPrintEntryPoint.kt to kotlinPrintEntryPoint.kexe

val kotlinCompilation = compileToExecutableInOneStage(
generateTestCaseWithSingleFile(
sourceFile = sourceDir.resolve("kotlinPrintEntryPoint.kt"),
testKind = TestKind.STANDALONE_NO_TR,
extras = TestCase.NoTestRunnerExtras("main")
)
).assertSuccess()

// 3. run main.cexe (with different parameters) to call kotlin executable

fun validate(expected: String, vararg args: String) {
val binaryName = kotlinCompilation.resultingArtifact.executableFile.path
val result = runProcess(cExecutable.absolutePath, binaryName, *args) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails if the test target is different from the host. You can reproduce this by running this test with -Pkotlin.internal.native.test.target=<target> Gradle property.

Please use a proper executor through

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this insight. I changed it accordingly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test fails on linuxArm64 target with:

java.lang.AssertionError: Expected <calling exec...
programName: app
args:>, actual <calling exec...
exec failed>.
  at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16)
  at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:694)
  at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11)
  at kotlin.test.Asserter$DefaultImpls.assertEquals(Assertions.kt:713)
  at kotlin.test.DefaultAsserter.assertEquals(DefaultAsserter.kt:11)
  at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:63)
  at kotlin.test.AssertionsKt.assertEquals(Unknown Source)
  at kotlin.test.AssertionsKt__AssertionsKt.assertEquals$default(Assertions.kt:62)
  at kotlin.test.AssertionsKt.assertEquals$default(Unknown Source)
  at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest$validate(ProgramNameTest.kt:49)
  at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest(ProgramNameTest.kt:54)
...

To reproduce, you can run the test with -Pkotlin.internal.native.test.target=linuxArm64 Gradle property on a Linux/x86_64 machine. I don't have access to such a machine at the moment. Could you please check the test, if possible?

Our test infrastructure runs tests on linuxArm64 through qemu, maybe the problem is somehow related to that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could reproduce the problem by running the test via ./gradlew :native:native.tests:test --tests "org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest" -Pkotlin.internal.native.test.target=linux_arm64 on an x86 linux machine. However you are right, the current test infrastructure with qemu user mode emulation seems to not handle the exec syscall in the way we need it. I also tried upgrading qemu to version 9.0.0, but this also didn't resolve the issue. If I understand the following qemu-comment correctly, qemu can't exec into a new process which runs again on qemu, but always does a syscall to the host system. This also explains why exec fails with error code 8 (Exec format error), as qemu would instruct the x86 host system to run the arm64 binary:

at the point of execve the process leaves QEMU's control
(qemu syscall.c)

I see now some ways to approach the problem:
A) Disable the testcase in case of testRunSettings.executor is EmulatorExecutable. Note that this testcase already tests the handling of the "default" program name from the exec syscall the JVM does for us. So we already know that reading a value from argv[0] works. Then this PR could probably be merged when no new points come up.
B) Create a FullHostSystemEmulatorExecutable, which uses qemu system emulation instead of qemu user space emulation. I'm not sure if this wouldn't be way out of scope for this PR.

Do you have any other ideas? If not, are you fine with approach A?

--
I also managed to run the EmulatorExecutable with the following command, which just does qemu -> main.cexe -> qemu -> app.kexe: /.../qemu-aarch64 -L /.../sysroot /.../main.cexe /.../qemu-aarch64 qemu-aarch64 -L /.../sysroot /.../app.kexe (minimal refactoring needed in EmulatorExecutable.kt). However, also in this case we can not control the exec syscall that actually starts app.kexe, as this is part of the qemu user mode emulation infrastrucutre. There qemu sets then manually argv[0]=app when launching app.kexe.

timeout = 60.seconds
}
assertEquals("calling exec...\n$expected", result.stdout)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this code should be fixed to handle \r\n on Windows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you again! The assertion is now platform agnostic due to additional sanitization.

assertEquals("", result.stderr)
}

// kotlinPrintEntryPoint removes .kexe
validate("programName: app\nargs:", "app.kexe")

// Simulate a custom program name, see e.g. https://busybox.net/downloads/BusyBox.html#usage
validate("programName: customProgramName\nargs:", "customProgramName")
validate("programName: customProgramName\nargs: firstArg, secondArg", "customProgramName", "firstArg", "secondArg")

// No program name - this would not be POSIX compliant, see https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html:
// "[...] requires a Strictly Conforming POSIX Application to pass at least one argument to the exec function"
// However, we should not crash the Kotlin runtime because of this.
validate("programName: null\nargs:")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux, this fails for me with

java.lang.AssertionError: Expected <calling exec...
programName: null
args:>, actual <calling exec...
programName: 
args:>.

Please take a look.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very interesting. Linux behaves here differently as windows & macOS. I fixed the problem, and added more explanation in the code. I used the following C standard PDF: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf.

Please tell me if you would want to let windows+macOS behave like linux. The way I wrote it now feels more natural to me, however I fully understand that all 3 options have its benefits and drawbacks:

A) The way I wrote it now: all 3 platforms behave the same; no program name leads to programName=null, and empty programName leads to programName="".
B) all 3 platforms behave the same; however both no program name and empty program Name lead to programName="". (the linux way of thinking)
C) Fully native behaviour: no programName will be programName="" on linux and programName=null on macOS/windows; and empty programName will be everywhere programName="".

}

companion object {
private val sourceDir = File("native/native.tests/testData/programName")
}
}