[Feature request] Getting rid of sharedUserId

[twaik]

Feature description

Lot of users have problems with sharedUserId.
Termux:X11 a lot of time does not have this problem.
Probably it will be good for Termux:API to implement something similar.

Connection establishing problem

Currently the most complicated problem is establishing connection.
termux-api command creates two abstract sockets and executes am broadcast with passing names of these abstract sockets as an arguments (extras).
It is problematic for two reasons.

1. We can not pass abstract sockets bypassing android application sandbox because of selinux restrictions.
2. We can not pin abstract or any other type of socket to intent because of am restrictions.

I see 2 ways to handle this.

First way

You can use some Java code (yeah, invoking app_process for this) to send broadcast with pinned binder, which will return file descriptors of sockets (probably created by socketpair, abstract sockets can not be shared this way) created by termux-api command when Termux:API's BroadcastReceiver invokes this Binder. Something like this:

some code

    /** @noinspection DataFlowIssue*/
    @SuppressLint("DiscouragedPrivateApi")
    public static Context createContext() {
        Context context = null;
        PrintStream err = System.err;
        try {
            // `android.app.ActivityThread.systemMain().getSystemContext()` is not used to avoid `java.lang.RuntimeException: Unable to instantiate Application():android.content.res.Resources$NotFoundException: Resource ID #0x600a6`
            java.lang.reflect.Field f = Class.forName("sun.misc.Unsafe").getDeclaredField("theUnsafe");
            f.setAccessible(true);
            Object unsafe = f.get(null);
            // Hiding harmless framework errors, like this:
            // java.io.FileNotFoundException: /data/system/theme_config/theme_compatibility.xml: open failed: ENOENT (No such file or directory)
            System.setErr(new PrintStream(new OutputStream() { public void write(int arg0) {} }));
            context = ((android.app.ActivityThread) Class.
                    forName("sun.misc.Unsafe").
                    getMethod("allocateInstance", Class.class).
                    invoke(unsafe, android.app.ActivityThread.class))
                    .getSystemContext();
        } catch (Exception e) {
            Log.e("Context", "Failed to instantiate context:", e);
            context = null;
        } finally {
            System.setErr(err);
        }
        return context;
    }

    @SuppressLint({"WrongConstant", "PrivateApi"})
    void sendBroadcast() {
        Bundle bundle = new Bundle();
        bundle.putBinder("", new Binder() {
            @Override
            protected boolean onTransact(int code, Parcel data, Parcel reply, int flags) {
                if (code == Binder.FIRST_CALL_TRANSACTION) {
                    /// ... Sending ParcelFileDescriptors obtained with `adoptFd` calls
                }
                return true;
            }
        });

        Intent intent = new Intent(ACTION_START);
        intent.putExtra("", bundle);
        intent.setPackage("com.termux.api");

        if (getuid() == 0 || getuid() == 2000)
            intent.setFlags(0x00400000 /* FLAG_RECEIVER_FROM_SHELL */);

        try {
            ctx.sendBroadcast(intent);
        } catch (Exception e) {
            if (e instanceof NullPointerException && ctx == null)
                Log.i("Broadcast", "Context is null, falling back to manual broadcasting");
            else
                Log.e("Broadcast", "Falling back to manual broadcasting, failed to broadcast intent through Context:", e);

            String packageName;
            try {
                packageName = android.app.ActivityThread.getPackageManager().getPackagesForUid(getuid())[0];
            } catch (RemoteException ex) {
                throw new RuntimeException(ex);
            }
            IActivityManager am;
            try {
                //noinspection JavaReflectionMemberAccess
                am = (IActivityManager) android.app.ActivityManager.class
                        .getMethod("getService")
                        .invoke(null);
            } catch (Exception e2) {
                try {
                    am = (IActivityManager) Class.forName("android.app.ActivityManagerNative")
                            .getMethod("getDefault")
                            .invoke(null);
                } catch (Exception e3) {
                    throw new RuntimeException(e3);
                }
            }

            assert am != null;
            IIntentSender sender = am.getIntentSender(1, packageName, null, null, 0, new Intent[] { intent },
                    null, PendingIntent.FLAG_CANCEL_CURRENT | PendingIntent.FLAG_ONE_SHOT, null, 0);
            try {
                //noinspection JavaReflectionMemberAccess
                IIntentSender.class
                        .getMethod("send", int.class, Intent.class, String.class, IBinder.class, IIntentReceiver.class, String.class, Bundle.class)
                        .invoke(sender, 0, intent, null, null, new IIntentReceiver.Stub() {
                            @Override public void performReceive(Intent i, int r, String d, Bundle e, boolean o, boolean s, int a) {}
                        }, null, null);
            } catch (Exception ex) {
                throw new RuntimeException(ex);
            }
        }
    }

This code is got from Termux:X11, but maybe most of it may be replaced with similar code from TermuxAm.

This code must be called on every termux-api invocation only in the case if socketpair created sockets are used. In the case if Java code will create server socket (i.e. in $TMPDIR/termux-api-socket) and stay in background (and will send sockets of newly-created connections to Termux:API through Broadcasts or already-established socketpair connection) you will need to launch this code only once. termux-api can check if Unix socket in $TMPDIR/termux-api-socket is connectable and if there is a lock file that should be created by Java code. OF COURSE this java code can be used to verify if we use Termux:API apk with right signature (hardcoded on compilation stage).

// Java
android.content.pm.PackageInfo targetInfo = (android.os.Build.VERSION.SDK_INT <= 32) ?
    android.app.ActivityThread.getPackageManager().getPackageInfo(BuildConfig.APPLICATION_ID, android.content.pm.PackageManager.GET_SIGNATURES, 0) :
    android.app.ActivityThread.getPackageManager().getPackageInfo(BuildConfig.APPLICATION_ID, (long) android.content.pm.PackageManager.GET_SIGNATURES, 0);
assert targetInfo.signatures.length == 1 && BuildConfig.SIGNATURE == targetInfo.signatures[0].hashCode() : packageSignatureMismatchErrorText;

// build.gradle
android.defaultConfig.buildConfigField "int", "SIGNATURE", String.valueOf(Arrays.hashCode(keyStore.getCertificate(signingConfig.keyAlias).getEncoded()))

Second way

You can add an ability to pass file descriptors through Binder to regular TermuxAm and termux-am-socket in similar way. In this case we will be able to go on using am broadcast command.
But in this case you can not verify if right Termux:API apk is installed. Or probably you may integrate signature verification to both TermuxAm and termux-am-socket too :).

Restrictions

That will be problematic for root users. For some reason SELinux blocks Unix sockets from being passed through Binder. Probably you will need to create two pipes (one for reading and one for writing) and pass one side of both pipes through Binder.

File access problem

In the case of disabling sharedUserId Termux:API will lose access to files in $PREFIX and $HOME.
I am proposing to not send stdin and stdout contents in both ways directly. Instead of that you can implement some simple command system to send buffers got from stdin/stdout, create/modify/delete files, pass file descriptors (i.e. for termux-usb) and do other stuff.
To avoid creating complicated IPC system you can go libxcb way and pass smth like union Event which will contain all possible events. I did this in Termux:X11:

some code

typedef enum {
    EVENT_SCREEN_SIZE,
    EVENT_TOUCH,
    EVENT_MOUSE,
    EVENT_KEY,
    EVENT_UNICODE,
    EVENT_CLIPBOARD_ENABLE,
    EVENT_CLIPBOARD_ANNOUNCE,
    EVENT_CLIPBOARD_REQUEST,
    EVENT_CLIPBOARD_SEND,
} eventType;
typedef union {
    uint8_t type;
    struct {
        uint8_t t;
        uint16_t width, height, framerate;
    } screenSize;
    struct {
        uint8_t t;
        uint16_t type, id, x, y;
    } touch;
    struct {
        uint8_t t;
        float x, y;
        uint8_t detail, down, relative;
    } mouse;
    struct {
        uint8_t t;
        uint16_t key;
        uint8_t state;
    } key;
    struct {
        uint8_t t;
        uint32_t code;
    } unicode;
    struct {
        uint8_t t;
        uint8_t enable;
    } clipboardEnable;
    struct {
        uint8_t t;
        uint32_t count;
    } clipboardSend;
} lorieEvent;

// and some code to handle that 
JNIEXPORT void JNICALL
Java_com_termux_x11_LorieView_handleXEvents(JNIEnv *env, jobject thiz) {
    checkConnection(env);
    if (conn_fd != -1) {
        lorieEvent e = {0};

        again:
        if (read(conn_fd, &e, sizeof(e)) == sizeof(e)) {
            switch(e.type) {
                case EVENT_CLIPBOARD_SEND: {
                    char clipboard[e.clipboardSend.count + 1];
                    read(conn_fd, clipboard, sizeof(clipboard));
                    clipboard[e.clipboardSend.count] = 0;
                    log(DEBUG, "Clipboard content (%zu symbols) is %s", strlen(clipboard), clipboard);
                    jmethodID id = (*env)->GetMethodID(env, (*env)->GetObjectClass(env, thiz), "setClipboardText","(Ljava/lang/String;)V");
                    jobject bb = (*env)->NewDirectByteBuffer(env, clipboard, strlen(clipboard));
                    jobject charset = (*env)->CallStaticObjectMethod(env, Charset.self, Charset.forName, (*env)->NewStringUTF(env, "UTF-8"));
                    jobject cb = (*env)->CallObjectMethod(env, charset, Charset.decode, bb);
                    (*env)->DeleteLocalRef(env, bb);

                    jstring str = (*env)->CallObjectMethod(env, cb, CharBuffer.toString);
                    (*env)->CallVoidMethod(env, thiz, id, str);
                    break;
                }
                case EVENT_CLIPBOARD_REQUEST: {
                    (*env)->CallVoidMethod(env, thiz, (*env)->GetMethodID(env, (*env)->GetObjectClass(env, thiz), "requestClipboard", "()V"));
                    break;
                }
            }
        }

        int n;
        if (ioctl(conn_fd, FIONREAD, &n) >= 0 && n > sizeof(e))
            goto again;
    }
}

This code can be combined with poll or select to not create threads for handling stdin/stdout/socket events separately.

In the case if you need to send/receive some additional data you can simply use `write`/`read` or `sendv`/`recv` (for the case when you pass file descriptors) after reading the `Event` struct/union.

@tareksander probably all of that is applicable to termux-gui too since it is based on termux-api and it is incompatible with F-Droid builds of Termux and its plugins.
Ping @agnostic-apollo @Grimler91 @tareksander.

[tareksander]

The first approach with a persistent connection sound almost like what I made for termux/termux-app#2921, just that my version is integrated into the Termux app and (as usual in contrast to your approaches lol) only uses the public Android APIs.

If you're running as root, can't you just drop back to the Termux user? I think apollo made a utility that drops the user and applies the correct SELinux stuff.

I don't know much about Android internals, is a process created with app_process and using Android APIs registered with the system somehow and a prime target for killing because it has no active components, or does app_process slip under the radar?

Regarding protocols, just sending unions is probably fine, but could leak stack data through uninitialized struct padding bytes. I don't think any plugin to date has to handle confidential/secure data or cryptographic keys or something like that, so it shouldn't be an issue.
Another issue with that is that it only supports C or native languages that can have struct layouts like C. For something like Termux:API it could be nice to write the libraries directly in the actual language you want to use instead of relying on C bindings. For Termux:GUI I use protocol buffers, which is probably overkill for most things, but flatbuffers seems to be more lightweight.
A custom Wayland protocol where you strip out the display stuff would also be an option: There's tools to generate Wayland bindings in many languages, and it comes with file descriptor passing out-of-the-box. Wayland is async, but all requests are handled in-order, so you can just block for a specific event to make it sync if you need to.

[twaik]

as usual in contrast to your approaches lol

I always had problems with making termux-app more compatible with termux-x11 so I decided to make it as more independent and self-sufficient as I can do. So now termux-x11 works even without termux-app or plugins installed in system (with chroot or non-termux proot environments).

If you're running as root, can't you just drop back to the Termux user?

Dropping to termux user is process-wide operation. It will work in the case if you want to launch some commands but it will may fail to pass you file descriptors (i.e. in the case if you use termux-usb, but I am not so sure).

I don't know much about Android internals, is a process created with app_process and using Android APIs registered with the system somehow and a prime target for killing because it has no active components, or does app_process slip under the radar?

System treats it as a regular child process, like bash or ls. It can run as long as other commands.

could leak stack data through uninitialized struct padding bytes.

I always fill the whole allocated area of event I am planning to send with zeros, right after allocation. I am pretty sure that is not a problem.

For something like Termux:API it could be nice to write the libraries directly in the actual language you want to use instead of relying on C bindings.

I experimented with writing Java binding for Wayland which should run just fine in Android. https://github.com/twaik/wayland-java-experiment . Of course currently it is not complete, but potentially we can make it a base of Termux:API or Termux:GUI but it may be an overkill.

[agnostic-apollo]

1. There are certain APIs they rely on access to termux rootfs, they would then have to be moved to termux-app if sharedUserId is ever removed. But then again, there are plans to add certain API support to termux-app as well.
2. Any design, whether twaik-crazy or stable, would need to work on latest targetSdkVersion. Does yours?
3. It should work if called with root, dropping from root to termux context is possible for the current process, I was writing a tool for it but its not complete, but it requires patching sepolicy, and is a slightly time consuming process. Check https://android.stackexchange.com/questions/217016/how-to-run-a-program-in-an-app-context-with-magisk/217104#217104
4. I mentioned this in some other discussion, we need to move to filesystem socket design instead of abstract namepsace. It will solve root issues as well, as "In the Linux implementation, pathname sockets honor the permissions of the directory they are in. Creation of a new socket fails if the process does not have write and search (execute) permission on the directory in which the socket is created". So if termux or api app creates the directory under apps directory, then both termux or root owned processes could just create sockets that api app should be able to access.
5. I also want stderr support as well as errors shouldn't be sent to stdout stream, with a separate socket. I also want continuous output support where long running APIs keep sending stdout and stderr on separate streams. There are also issues related to broadcast receiver having some few second time limit, so things need to move to a service.
6. I also want other third party apps with RUN_COMMAND permission to be able to call termux-app and termux-api app APIs. ResultReturner design needs to be overhauled where it doesn't matter whether result needs to be sent back via a PendingIntent, to stdout/stderr/exit_code files like termux-app RUN_COMMAND intent does, or to socket streams. I was doing some work related to that long ago, it requires a service to manage each execution action and its result. External apps can use PendingIntent or possibly use contentprovider to create sockets if they want streamed data instead, it "should" work.
7. There should be very little overhead for the design. Why gain time with termux-am-socket if its just wasted on some heavy protocol.
8. Security sensitive APIs do exist like KeyStore, with update in KeyStoreAPI Upgrade (Encrypt/Decrypt + more) #556.

[tareksander]

2. My approach worked last time I checked, but in the last few Android versions there may have been new SELinux restrictions. I'll download the latest emulator image and check.
3. AFAIK since my approach doesn't need UID checks, nothing should prevent it from working as root.
4. My approach uses filesystem sockets already, so all good.
5. You can always pass more file descriptors to each other once you have a unix socket connection.
6. It also includes a new way to run commands in Termux, with no restrictions on output length and streaming stdin, stdout and stderr through pipes.
7. How complicated the underlying protocol should be would depend on the plugin: For Termux:API passing structs and unions is fine, something more complicated like Termux:GUI needs something a bit more maintainable and with more language support.making a custom binary protocol that's shared between plugins is also an option if you want to control exactly how much protocol overhead there is.
8. As long as you zero out the buffer before sending, no data can be leaked. But you only need to forget once for a potential data leak.

I'll test tomorrow if it still works in new Android versions.
EDIT: Welp, I get a build error when trying to build termuy-app, and from some googling I'm apparently having a too up-to-date JDK...

[tareksander]

Found a version that doesn't break the build, but works on my machine. I fixed everything now for API 34, still works, including FD passing over unix socket connection.

[agnostic-apollo]

5/6. Ah, yes, that would work too.

Thanks for the fixes. Will your design even work with app_process started processes to bind to the service, as there is no real context. How would it work for native processes like termux-api-package ones to connect to the termux-api-app? Haven't given thought into it myself.

[tareksander]

You'd check if the filesystem socket accepts connections, if it doesn't you poke the plugin with am broadcast to let it re-connect to Termux and set up its listener for the filesystem socket again. After that each connection on the filesystem socket gets passed to the plugin.

For the plugin to not get killed there are 2 options:

1. Foreground service: I would recommend against this if possible, multiple active plugins could easily clutter the notification area.
2. Callback service: Instead of passing a callback Binder to Termux, you pass the component name of a service in your app. Termux will then bind to this service, and that will make the service inherit the foreground priority Termux gets through its foreground service.

Because the service interaction is managed through the Termux app, the only thing you need is to call am or termux-am and access a filesystem socket.

[agnostic-apollo]

hmmm, I see. Hopefully, synchronization or incoming queue delay won't be an issue.

A foreground service is already going to be used by plugin apps to manage API actions in future, and notification can be disabled on Android >= 8.

Callback service could be looked into too.