Consider adding malli

[borkdude]

Malli is a data-driven schema library for Clojure. It can be quite convenient to have this in scripts to validate incoming arguments, http responses, etc.

$ ./bb -e "(require (quote [malli.core :as m])) (m/validate [:and [:fn '(fn [x] (int? x))] [:> 6]] 7)"
true

Please use a thumbs up on the original post or a thumbs down if don't like this idea. You can elaborate in comments.

Also see #558 for an issue on adding spec (which is still in alpha, waiting for v2) which serves more or less the same purpose.
Do we need both? Are we going to wait for some more years?

Alternatives:

• https://github.com/green-coder/minimallist
• https://github.com/borkdude/spartan.spec

[zoedsoupe]

I really liked the idea of having these validations scripts

[borkdude]

Please use a thumbs up on the original post or a thumbs down if don't like this idea. You can elaborate in comments.

[bbqbaron]

I haven't used Malli enough to judge it fully, but if there's critical mass in the community or experts think Malli is a strong default option to push, I guess all that's left is "what additional complexity is composed on bb by its presence?" Obviously no one has to use Malli, so I assume that would just be the usual concerns with building another dependency into it. Is there any risk that Babashka suffers from bloat or fragility for each extra library it packs in (I'm not familiar with Graal)? If not, seems like an auto yes? 👍

[borkdude]

It seems the binary would grow with another 2-5 mb. For every lib we add this might happen, so eventually we could end up with a bigger and bigger bb. But there is a LEAN flag which we can use to provide leaner builds, if this is a problem. Still, we should be careful to select which libraries go in, because once they are in, bb is not going to disable them anymore (unless maybe in the very short term if we discover some problems).

[bbqbaron]

It seems the binary would grow with another 2-5 mb. For every lib we add this might happen, so eventually we could end up with a bigger and bigger bb. But there is a LEAN flag which we can use to provide leaner builds, if this is a problem. Still, we should be careful to select which libraries go in, because once they are in, bb is not going to disable them anymore (unless maybe in the very short term if we discover some problems).

To devil's-advocate against the bloat, I suppose one could say "look, if your script is big enough to need schema, it should be a lein project; there's a reason there are no typed scripting languages (that I know of)". I don't strongly support that argument but I think it's a reasonable, probably common perspective.

[wilkerlucio]

I think Malli is quite opinionated, I would rather keep babashka cleaner. If your code is big enough to need schemas, IMO you can add an additional library dep for it. It's quite possible that in 6 months or a year there could be a new favorite one around. But then to remove Malli would mean to break all the apps depending on it natively. Given babaska can never turn back on those, I would prefer to have fewer built-in dependencies.

[borkdude]

Thank you @wilkerlucio, that is a good point.

Note that we already have some alternatives as of today:

• https://github.com/green-coder/minimallist
• https://github.com/borkdude/spartan.spec

These libraries can be used from source.

[n2o]

I'd also prefer to support official libraries and wait for spec2 as discussed in the referenced issue above. I really love the work done by the people from metosin, but it does not seem wise to add support for all the community libs.

[maxp]

Good idea!

[borkdude]

We cannot add support for "all the community libs" but we can choose one that dominates in a certain area. Currently babashka doesn't have anything for data validation yet. This will change with spec2 of course.

[j0ni]

Malli is IMO the best of the current bunch, but I'm not sure I understand why it would be built-in rather than a pod.

[borkdude]

@j0ni It can definitely be a pod if you're just validating some EDN that can be sent over the wire, but for integration with defn, etc that would not work. So you might run into some edge cases with pods. That said, worth a try.

[lukaszkorecki]

I agree with @wilkerlucio - Prismatic's Schema used to be dominant choice, but it doesn't look like it's used in new projects anymore, partially because of Spec, which never really materialized. There's a big chance that Malli will get "pushed out" once Spec2 arrives.

Not being negative or anything, but just trying to have a long term view on things.

Just to balance things out: if it's a matter of adding a couple of MB to the final binary, I don't mind 😉

[borkdude]

Really appreciate the feedback! Agree we should have a long term view.

[stevebuik]

My use case for this is validating tools.cli parsed args. For now I am custom coding this but I would use Malli if it was available.

I can see the sense in the idea of not bundling it but instead just ensuring that Malli can be loaded/required as a dep.

[borkdude]

Meanwhile I have created this experimental project:

https://github.com/babashka/pod-babashka-malli

Not sure yet if this is a valid approach, since it has some differences with malli. Maybe these can be ironed out in collaboration with @ikitommi.

(prn (m/validate [:cat 'int? 'int?] [1 1]))
;; => true

(prn (me/humanize [:cat 'int? 'int?] [1 :foo]))
;; => [nil ["should be an int"]]

$ time bb test.clj
true
[nil ["should be an int"]]
bb test.clj   0.01s  user 0.01s system 51% cpu 0.049 total

[logseq-cldwalker]

Thanks for making the experimental pod. For all the interest in this issue, I'm a bit surprised more people aren't using it. I'm considering using malli vs spec1 at work and what just works with the bb ecosystem is a consideration in what library we'll go with

[borkdude]

clojure.spec works well if you use this fork: https://github.com/babashka/spec.alpha

More info in blog: https://blog.michielborkent.nl/using-clojure-spec-alpha-with-babashka.html

[stevebuik]

Thanks. For my tools.cli use-case, a custom fn was an easy/simple replacement for Malli.

I'll use this on the next one but if it becomes non-experimental since it's for prod CI.

I'm curious why it needs to be a pod instead of a plain dep/require?

[borkdude]

@stevebuik Because babashka/sci cannot currently run malli from source.

[devn]

Given malli says alpha on the tin, and spec2 isn't in the wild just yet, it may be prudent to keep pushing on the pod approach and hold off on a final decision until some additional dust has settled.

That said, malli is great and I think I'd prefer it to spec2 given my experience with the ergonomics of using spec1. If spec2's developer story winds up like spec1 with people building tools on top of it to make it nice to use, then might we wind up in a situation where we'll be discussing which community-supported spec2 helper libs are worth including alongside it?

On the topic of Schema and long-term thinking, I would be careful about judging the fitness of a lib based on its popularity. I work on a work project that uses Schema, and I am happy we didn't jump into spec1 simply because it was blessed or fashionable.

As others have pointed out, perhaps the average babashka program does not warrant the addition of a full-fledged validation library. Perhaps spec2 would be enough. Perhaps something even simpler than both of these would suffice (people in this thread point out arg validation as a use case). I won't pretend to know, because I haven't sat down and done a thorough review of the tradeoffs involved in all of the above.

In short, I don't have a concrete answer: Deferring a final decision doesn't seem like it'll hurt all that much. I prefer malli if a choice is to be made soon on inclusion, but spec2 is a fit predator and its guaranteed long-term support shouldn't be discounted in the tradeoff matrix, whether I find it enjoyable to use or not. I support the desire for a small binary. The philosophy of the kinds of programs bb is designed to support matters a great deal in situations like this, and so I would probably just punt and leave it to the primary maintainer on what they feel meets the philosophical goals of bb the best.

[piotr-yuxuan]

If I may, here is an attempt to bring malli schemas to CLI args parsing and validation. It's still early-stage, but any technical feedback / suggestions / improvements will be appreciated!

https://github.com/piotr-yuxuan/malli-cli

[piotr-yuxuan]

Now it has env-var support: piotr-yuxuan/malli-cli@3ce0e34

[dgb23]

I'm under the impression that malli could be used as a frontend to spec2 judging from the way spec2 is going. Is that intuition completely mad or is there some merit to it?

[borkdude]

I now converted this issue into a discussion, so we can elaborate on different topics in threads, if desirable.

[lins05]

Looks like the main concert is the binary size. If sci could run malli from source then everyone would be happy.

[borkdude]

The main concerns:

• Increase in binary size (2-5mb) should be justified
• Malli is opinionated (not necessarily bad), not yet a common "infrastructure" project, but it could be over time.
• We usually include libraries for direct usefulness but also if they are used a lot in libraries which will then suddenly become compatible with babashka. Some people find that for small scripts you usually don't need a lot of validation. But if libraries that are useful for scripting start to rely on malli, that would still be a good reason to include it.
• Still waiting on spec2. It seems to make sense to first include that and then see whether we still need malli in bb. One could also argue that malli is a better fit for bb than spec since it's good ad ad-hoc validation.

I love malli! But we have to be careful about what libraries we take on for the above reasons. It might be better to wait a bit and see what happens in the ecosystem than to jump ahead.

And now your argument: yes, I'd love to see a port of malli that is source-compatible with bb, even if it supported only a subset (like spartan.spec does for spec). Unfortunately malli isn't compatible from source right now.

[dgb23]

The more I think and read about spec, the less I'm convinced if these kind of use cases are a core consideration of the library. I think it is more about formal specifications of data structures and functions, and how they change over time, or at least that seems more in focus.

The question is: is spec also about parsing and modeling like in this use case? I know it can be used this way, but I think one of the reasons for malli to exist, is that spec didn't seem suitable for these tasks as much as the authors would have liked. If they have orthogonal concerns and operate nicely together after spec2, then it might be useful to include both anyways.

I say these things as a irregular clojure user though. My perspective comes from using the REPL a lot to probe API's and do ad-hoc/one-time data transformations and migrations, also do a bit of automation with babashka. For my use cases babashka is about ergonomics and malli seems to be a library that is very ergonomic. But I also don't need to make as broad considerations as some of you do.

[borkdude]

A PR to add compatibility to bb for malli: metosin/malli#718

[borkdude]

Malli 0.8.9 can now be used with babashka 0.8.157!

[stevebuik]

great. the right tool for the job