Server-Side Swift Support

[spprichard]

Feature Description

The project currently allows Swift Developers to import & use Llama.cpp on Apple platforms (iOS, MacOS, iPadOS, etc). However, there are other use cases for Swift outside of Apple's platforms. Such as Server-Side Swift with frameworks like Vapor or Humminbird where a Swift Developer may want to use Llama.cpp. Generally speaking, server-side Swift applications are deployed via docker, where the image is base on Linux.

I've found that when writing server-side Swift applications with depend on Llama.cpp where Llama.cpp is brought into the application via Swift Package Manager, I run into issues complaining Foundation. I might be missing something here, but it appears to me that Llama.cpp relies on the Apple platform's Foundation implementation. Which limits the platforms Swift developers could use Llama.cpp. There is ongoing work by Apple to create a "Linux compatible" version of Foundation.

Additionally, as far as I can tell, Llama.cpp relies on some aspects of Objective-C. As discussed in this post, there appears to be desire to remove this Objective-C dependency. In favor of taking advantage of the Swift/C++ Interop.

To me, between Foundation on Linux & Swift/C++ Interop, it appears technically possible to run Swift applications which leverage Llama.cpp on non-Apple platforms. As an aside, I am still trying to get my head wrapped around this, so forgive me if I don't have all the details correct.

Motivation

I bring this all up to ask the question, is there desire to configure/refactor the Swift specific aspects of Llama.cpp in such a way that would allow for Swift developers to use Llama.cpp on non-Apple platforms (Linux) such as in Server-Side Swift applications?

Questions

• I am not sure how Metal support would affect this. I assume we would be able to compile metal for linux, but I am speaking out of my element here, so looking for someone to potentially correct me here.

[ggerganov]

The Foundation framework is only needed by the Metal code in ggml-metal.m. The Metal backend is completely optional. For example, building like this will produce llama.cpp that does not depend on Metal and Foundation even on Apple platforms:

make clean
LLAMA_NO_METAL=1 make -j main

Objective-C is also used only by the Metal backend. Running on Linux, the Obj-C code never gets build because there is no Metal.

Unless I'm missing something, you should already be able to build and run Swift apps on non-Apple platforms.

[0xTim]

@ggerganov I think the issue here is trying to include the library as a Swift package using the Package.swift manifest. That specifically includes the Objective-C file and metal resources etc, meaning you can't pull the library in via SwiftPM on Linux. Ideally those files would be wrapped in something like #if canImport(Darwin) to only include those files when building for Apply platforms, and not for Linux

Ideally the C++ code would be exposed directly to Swift using the new C++/Swift interop too, which requires an explicit opt-in

[ggerganov]

I think something along those lines was proposed here: ggerganov/whisper.cpp#1990

Could you please take a look and see if this is what you are looking for? If yes, we can update llama.cpp respectively

[spprichard]

Yes I think this is close, with the addition of opting into C++/Swift interop support with

swiftSettings: [.interoperabilityMode(.Cxx)],

Should point out, C++ interop requires swift-tools-version to increase to 5.9

@0xTim your thoughts?

[spprichard]

@ggerganov I've created this example project to verify the changes discussed here

[spprichard]

@ggerganov I've also went ahead and opened this PR, as requested. Similar to the Whisper.cpp PR mentioned above

A question however, should the C++ interop change be in this PR or would you want that in a separate PR? Additionally, I am not sure the current state of the Swift/C++ interop, if it would require being put behind some sort of experimental flag such that we conditionally add it in the Package.swift? @0xTim do you have any insight or suggestions here?

[ggerganov]

Hm, I'm not sure what you mean by "Swift/C++ interop". llama.cpp has a C-style interface in the llama.h file - it should be usable in Swift

[spprichard]

I'm still working on understanding all of it, but as of Swift 5.9, Swift supports interoperability with C++. As I understand it, it allows for Swift to call C++ directly (without compiling?). So for folks that want to write Swift libraries on top of Llama.cpp, they can call the C++ API directly without things like Unsafe pointers & what not (I think). I imagine there is probably some advantage to this from the Swift runtime perspective as well.

Thats the extent of my knowledge on this, so maybe someone with deeper technical knowledge could elaborate further.

[spprichard]

Wanted to follow up on this, I spoke with some folks from the C++ Interop Working group. They looked specifically at llama.cpp & concluded that based on the current structure of the project C/C++ Interop should work. In my personal testing it does. Also, no changes would be required on llama.cpp side. Enabling C++ Interop would be done on the consuming package side. Meaning, a Swift package built on top of Llama.pp would enabled Interop. Also note, C Interop is on by default. So all good news on that front.

Some investigation might need to be done on how the 1 objective-c file ggml-metal.m affects this. So far in my testing by simply adding

swiftSettings: [
    .interoperabilityMode(.Cxx),
]

I've been able to achieve what I need 👍

[reneleonhardt]

@spprichard Thank you for this amazing contribution!
iOS is about to be released, is there something left to do in llama.cpp for Web Distribution from an AI / Swift 5.10+ point of view for the new iPads?
https://www.swift.org/swift-evolution/
https://www.forbes.com/sites/davidphelan/2024/04/17/ios-175-release-date-when-the-new-iphone-sweeping-change-will-debut-apple-iphone-15-pro-max/
Unfortunately it is restricted to 1 Million installations, not really useful to open up the marketplace lol 😆