Add NSInputStream.source() and BufferedSource.inputStream() functions for Apple's NSInputStream #1123

jeffdgr8 · 2022-06-22T20:36:09Z

Modeled after JVM's InputStream.source() and BufferedSource.inputStream(), add extension functions to similarly map to and from Apple's NSInputStream.

JakeWharton · 2022-07-22T12:46:15Z

FYI #1131 moves a Darwin-specific extension function onto the actual type's companion so that it becomes visible across the Objective-C bridge. We will want to do that same here, but that can be done in a follow-up.

jeffdgr8 · 2022-07-23T16:03:11Z

@JakeWharton excellent. I'll take a look at moving to a companion object. Looks like once #1131 is merged, we'll have to duplicate actual BufferedSource in appleMain and nonAppleMain source sets, adding the extension to the appleMain companion?

I'm assuming NSInputStream.source() would be appropriate on Source's companion. There's only a single interface defined in common though. Does it need to be converted to expect/actual across all source sets for this change? And would actuals be in jvm, nonAppleMain, and appleMain?

I have these extensions working in my KMM database bindings with a local maven build and things are working well in API tests, by the way.

swankjesse

Love this. Thanks for your patience on my slow review.

swankjesse · 2022-07-25T01:04:13Z

okio/src/appleMain/kotlin/okio/Source.kt

+import platform.darwin.UInt8Var
+
+/** Returns a source that reads from `in`. */
+fun NSInputStream.source(): Source = NSInputStreamSource(this)


I think this file should be called source.kt (lowercase name) . . . we’ve been doing lowerCamelFileNames for files with free functions, UpperCamelFileNames when the file matches the primary declaration class.

Should BufferedSource.kt and ByteString.kt, also in this directory, be renamed bufferedSource.ktand byteString.kt as well? Or perhaps BufferedSource.kt should be inputStream.kt?

@swankjesse after merging the latest upstream changes, the linter doesn't like the lowercase filename source.kt, wanting PascalCase. I renamed this file NSInputStreamSource.kt. Let me know if there's a preferable name.

swankjesse · 2022-07-25T01:05:18Z

okio/src/appleMain/kotlin/okio/Source.kt

+ }
+ if (bytesRead < 0) throw IOException(input.streamError?.localizedDescription)
+ if (bytesRead == 0L) {
+ if (tail.pos == tail.limit) {


swankjesse · 2022-07-25T01:07:00Z

okio/src/appleTest/kotlin/okio/AppleBufferedSourceTest.kt

+ byteArray.fill(-5)
+ assertEquals(-1, nsis.read(cPtr, 4))
+ assertNotNull(nsis.streamError)
+ assertEquals("closed", nsis.streamError?.localizedDescription)


swankjesse · 2022-07-25T01:11:30Z

okio/src/appleMain/kotlin/okio/BufferedSource.kt

+ if (bufferedSource is RealBufferedSource) {
+ if (bufferedSource.closed) throw IOException("closed")
+
+ if (internalBuffer.size == 0L) {


Please rewrite this if clause as

if (bufferedSource.exhausted()) return 0

That will guarantee the buffer has at least one byte in it after!

Oh, nice. I see RealBufferedSource.commonExhausted() is this exact logic. I went ahead and replaced this same logic in other places as well 3571012. Let me know if you'd rather have that reverted though.

swankjesse · 2022-07-25T01:15:37Z

okio/src/appleMain/kotlin/okio/BufferedSource.kt

+ }
+ }
+
+ override fun getBuffer(


I don’t think this is a big deal for now, but eventually it might be a problem .. . .

Using usePinned to get the address of a byte within a buffer is fine, but I worry that the caller might need usePinned when they later read from the buffer.

(This is only an issue if Kotlin’s GC ever learns to relocate objects)

I wondered if this could be problematic, but couldn't think of a way to pass the unpinning to the caller. I figure typical usage would read the buffer immediately after calling the function and wouldn't hold the reference longer. Still a minute chance GC occurs right between unpinning and reading though, and even tinier chance the address becomes invalid.

Copying the buffer would nullify the usefulness of the API and violate the documented behavior of returning in 0(1). I figured having it implemented is probably preferable to not, but alternatively we could just return false.

I wonder what the behavior would be to pin() the address without unpin()ing it. Would GC still collect it once the reference is released, just disallow relocating it? Or would this create a memory leak?

val pinned = s.data.pin() buffer?.pointed?.value = pinned.addressOf(s.pos).reinterpret() length?.pointed?.value = (s.limit - s.pos).convert() return true

The memory leak question is also a good one. At the moment this API doesn't tell the GC about this reference to the ByteArray, so it could be collected (or recycled) before the caller gets to use it.

Maybe we should pin it here, and/or take other steps to avoid recycling, then unpin either on the next call to this method or on close(). That would work and it wouldn't leak.

Great idea! I'll make that change.

Unpin on next call or on close()

jeffdgr8 · 2022-07-27T15:51:12Z

@JakeWharton I took a crack at moving these extensions to the companion objects and testing in an iOS app with a KMM shared library with okio as a dependency. I could see the issue, where top-level extensions in the KMM library are exposed in Swift/ObjC, but okio's extensions are not, as a dependency.

The issue I ran into is that BufferedSource and Source are both interfaces. So their companion objects are not accessible on their @protocols in ObjC/Swift, as ByteString's is, as a class.

I'm not sure if there's an intuitive class these extensions could live in instead. It might make sense to create a utility object with the functions available there, but you still have the issue of needing to access the object in the Kotlin library in order for it to be exposed in ObjC/Swift (but at least this is possible, where the Kt class for top-level extensions isn't possible to access in Kotlin).

Thinking about usability in both Kotlin and ObjC/Swift, I think it could make sense to still have the top-level extension as part of the API, rather than deprecating as you did when moving to the companion. This makes using in Kotlin nicer without needing to wrap in with(Foo) { ... }. But then additionally create a utility object that can expose the extensions in ObjC/Swift as another way to access the API.

There are likely a good number of extensions that suffer from this limitation (e.g. Source.buffer()). It'd be good to have a way to expose all related extensions through a single mechanism, whatever it is. I liked the way that the companion object comes along with a class being used in Kotlin. I'm not sure what might be able to work similarly for interfaces.

Ultimately, seems like it would make sense for Kotlin to respect the gradle dependency configuration when generating the ObjC headers for transitive dependencies. If a dependency is declared as implementation, then its full API shouldn't be exposed. But if it's declared as api, then it would make sense to expose the full API, same as the parent library's. But maybe this should be reserved only for transitiveExport.

Also, what is the @CName annotation you're using do? It sounds interesting, based on its documentation. Playing around with it, I don't see a function with the externName provided though. Does this only affect C/C++ and not ObjC?

serandel · 2023-06-02T14:31:54Z

Naive question, is this PR still in mind? I see an approval from last year, but I don't know if it needs more, or it has been discarded. Talking to Jeff, this is the only blocker to have a Kotlin Multiplatform Couchbase Lite library, which would be really awesome.

Until square/okio#1123 is merged and released.

swankjesse · 2023-07-18T12:11:33Z

I’d like to backport this once it lands:
Kotlin/kotlinx-io#174

jeffdgr8 · 2023-07-19T03:31:49Z

I’d like to backport this once it lands: Kotlin/kotlinx-io#174

Great, I'm planning to implement the additions there on this Okio branch as well.

Support NSRunLoop scheduling Other changes from kotlinx-io PR

jeffdgr8 · 2023-08-10T06:35:22Z

I backported the changes from Kotlin/kotlinx-io#174 including support for NSOutputStream sink extensions and NSRunLoop scheduling.

swankjesse · 2023-08-11T00:15:56Z

Oh that’s awesome. Thanks!

okio/build.gradle.kts

okio/src/appleMain/kotlin/okio/BuffersApple.kt

jeffdgr8 added 10 commits June 21, 2022 01:00

NSInputStream.source() and BufferedSource.inputStream(): NSInputStream

e0548f2

Set streamError and return -1 on read failure

5578fd8

Implement NSInputStream.getBuffer()

b95de57

convert() native types

ddb66c3

Test NSInputStream.close()

a6c3e16

Additional test assertions

7a8c292

appleTest TestUtil

b8d1ba9

Add doc comments

c319152

Resolve checks

506fa4f

Avoid data copy & read source to buffer

aa0b6d3

swankjesse force-pushed the master branch from 2a696bb to ef6bf7f Compare June 27, 2022 00:31

jeffdgr8 and others added 2 commits June 27, 2022 11:49

Resolve checks

9ac4e4d

Merge branch 'master' into nsinputstream

7e9cbfa

jeffdgr8 added 3 commits July 22, 2022 15:34

Override open() as no-op

72efc44

Rename variable to avoid ambiguity

0f74ac7

Move private functions in class

3191c97

swankjesse approved these changes Jul 25, 2022

View reviewed changes

swankjesse reviewed Jul 25, 2022

View reviewed changes

jeffdgr8 added 6 commits July 24, 2022 22:45

Code review feedback

031a3a1

Replace additional RealBufferedSource.commonExhausted() logic

3571012

Remove variable

41c38a5

Keep buffer pinned for getBuffer() caller

c59ea79

Unpin on next call or on close()

null pinnedBuffer after unpin()

fb883f9

Merge branch 'master' into nsinputstream

e554a72

jeffdgr8 added 2 commits February 28, 2023 21:39

Merge remote-tracking branch 'upstream/master' into nsinputstream

f917c2e

Fix lint errors

a40e1c7

jeffdgr8 added a commit to jeffdgr8/kotbase that referenced this pull request Jun 3, 2023

Okio NSInputeStream Source extensions

f586b0b

Until square/okio#1123 is merged and released.

jeffdgr8 added a commit to jeffdgr8/kotbase that referenced this pull request Jun 3, 2023

Okio NSInputeStream Source extensions

de7408f

Until square/okio#1123 is merged and released.

jeffdgr8 mentioned this pull request Jul 12, 2023

Add Source and Sink extensions for Apple's NSInputStream and NSOutputStream Kotlin/kotlinx-io#174

Merged

jeffdgr8 added 2 commits August 9, 2023 15:14

Merge remote-tracking branch 'upstream/master' into nsinputstream

97192bb

Add NSOutputStream extensions

bbddc69

Support NSRunLoop scheduling Other changes from kotlinx-io PR

PromanSEW reviewed Aug 11, 2023

View reviewed changes

okio/build.gradle.kts Show resolved Hide resolved

fzhinkin reviewed Aug 24, 2023

View reviewed changes

okio/src/appleMain/kotlin/okio/BuffersApple.kt Outdated Show resolved Hide resolved

jeffdgr8 added 2 commits August 24, 2023 15:29

Fix from Kotlin/kotlinx-io#215

8b3fe7e

Remove comment

f5d63c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add NSInputStream.source() and BufferedSource.inputStream() functions for Apple's NSInputStream #1123

Add NSInputStream.source() and BufferedSource.inputStream() functions for Apple's NSInputStream #1123

jeffdgr8 commented Jun 22, 2022 •

edited

JakeWharton commented Jul 22, 2022

jeffdgr8 commented Jul 23, 2022

swankjesse left a comment

swankjesse Jul 25, 2022

jeffdgr8 Jul 25, 2022

jeffdgr8 Mar 1, 2023

swankjesse Jul 25, 2022

swankjesse Jul 25, 2022

swankjesse Jul 25, 2022

jeffdgr8 Jul 25, 2022

swankjesse Jul 25, 2022

swankjesse Jul 25, 2022 •

edited

jeffdgr8 Jul 25, 2022

swankjesse Jul 25, 2022

jeffdgr8 Jul 25, 2022

jeffdgr8 commented Jul 27, 2022 •

edited

serandel commented Jun 2, 2023

swankjesse commented Jul 18, 2023

jeffdgr8 commented Jul 19, 2023

jeffdgr8 commented Aug 10, 2023

swankjesse commented Aug 11, 2023

Add NSInputStream.source() and BufferedSource.inputStream() functions for Apple's NSInputStream #1123

Are you sure you want to change the base?

Add NSInputStream.source() and BufferedSource.inputStream() functions for Apple's NSInputStream #1123

Conversation

jeffdgr8 commented Jun 22, 2022 • edited

JakeWharton commented Jul 22, 2022

jeffdgr8 commented Jul 23, 2022

swankjesse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swankjesse Jul 25, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffdgr8 commented Jul 27, 2022 • edited

serandel commented Jun 2, 2023

swankjesse commented Jul 18, 2023

jeffdgr8 commented Jul 19, 2023

jeffdgr8 commented Aug 10, 2023

swankjesse commented Aug 11, 2023

jeffdgr8 commented Jun 22, 2022 •

edited

swankjesse Jul 25, 2022 •

edited

jeffdgr8 commented Jul 27, 2022 •

edited