clojure.core test suite

[jeaye]

This may already exist somewhere, but jank could really use a fully clojure test suite for every clojure.core function. This would be beneficial for all clojure dialects, so perhaps we keep it in a separate repo. Work on this can start prior to jank actually supporting all of those functions. Usage of clojure.test makes sense, as long as there's no interop.

If this doesn't exist, it'd be quite the undertaking, due to the size of clojure.core. However, it'd also be an excellent sweat bed for jank to run continuously. Worth the effort.

[jeaye]

I discussed this in Slack here: https://clojurians.slack.com/archives/C03S1KBA2/p1690926466763509

Alex said he's interested in something like this, especially if we:

1. Keep it in a separate repo
2. Require contributers to sign Clojure's CA
3. License it EPL

[mateodif]

I would suggest looking into Lokke's testing suite as a basis, I think it could be pretty useful.

https://github.com/lokke-org/lokke/tree/main/test

[jeaye]

I would suggest looking into Lokke's testing suite as a basis, I think it could be pretty useful.

https://github.com/lokke-org/lokke/tree/main/test

Excellent suggestion! Thanks for sharing that so we have it here for reference.

[lafka]

This may have been discussed in the mentioned slack thread, i don't have access so i wouldn't know.

Is there any reason why the existing tests/code from clojure.core can't be reused? If there are any incompatibilities between EPL and MPL, creating a separate repository would allow reuse of existing code.

[jeaye]

This may have been discussed in the mentioned slack thread, i don't have access so i wouldn't know.

Is there any reason why the existing tests/code from clojure.core can't be reused? If there are any incompatibilities between EPL and MPL, creating a separate repository would allow reuse of existing code.

The slack thread is me asking Alex Miller (Clojure team lead) if Clojure has a test suite for clojure.core at all, since I haven't found it. He confirmed that there isn't one. I asked if he'd be interested in taking one on, if I got it going. He said yes.

[lafka]

I'm not a clojure native so apologize if i'm missing something obvious. Skimming through the clojure repo it seems atleast parts of it have tests, Like take, drop, partition, and more are covered in test/clojure/test_clojure/sequences.clj.

Would picking out relevant cases from existing tests in clojure and copying it to a different repo be a good a start? Is that aligned with what you're looking for?

[jeaye]

I'm not a clojure native so apologize if i'm missing something obvious. Skimming through the clojure repo it seems atleast parts of it have tests, Like take, drop, partition, and more are covered in test/clojure/test_clojure/sequences.clj.

Would picking out relevant cases from existing tests in clojure and copying it to a different repo be a good a start? Is that aligned with what you're looking for?

Clojure has some coverage of some functions (not many), yeah. However, what I have in mind is a systematic and consistent approach to cover every function in the same way, as much as possible. I intend to come up with a set of rules for types of inputs and outputs, common failures, etc, and then apply those to each function.

[lafka]

Ok, i have made https://github.com/lafka/jank.core.tests. If you'll find it useful i'll start adding test cases one by one over there. For now i'm running the tests using clojure, getting it to with jank itself should be the next logical step.

If you have any preference on how to structure the files let me know, otherwise i'll keep it all in a single test file.

[jeaye]

Ok, i have made https://github.com/lafka/jank.core.tests. If you'll find it useful i'll start adding test cases one by one over there. For now i'm running the tests using clojure, getting it to with jank itself should be the next logical step.

If you have any preference on how to structure the files let me know, otherwise i'll keep it all in a single test file.

Oh, cool! Thanks for starting on this. What I had planned for this, which may be some guidance for you if you'd like to take it on, is to use ChatGPT (or similar) to generate these tests. The reason is, clojure.core has 551 defs and defns. If each one has a dozen unit tests, we're talking about several thousand tests. Writing each one by hand would be more than cumbersome, which is exactly why I think this test suite doesn't exist yet.

Now, whether or not you want to use an LLM to help generate these tests, the first thing we need to start with is a set of common things to check for. Here's a starting list:

Testing questions

Common cases

• What happens if it's given all valid inputs? (this will require some manual work to identify edge cases)
• Are there any special cases for inputs?
• What happens when the transducer arity is called?

Edge cases

• What happens when the input is nil? (apply to all inputs)
• What happens when the input is an incorrect shape (i.e. a number instead of a sequence)? (apply to all inputs)
• What happens when the input is an infinite sequence (note that not some fns are explicitly eager and can't do this)?
• If the input accepts a map, what happens with both array maps and hash maps?
• If the input accepts a set, what happens with both sorted sets and hash sets?

Things we don't need to test

• Invalid arity (too many/too few args, the runtime does this for us, not the fn itself)

Perhaps we should iterate on the list before writing any further code and then be sure we structure the code for each function to follow the list. Since this will be thousands of test cases, I imagine one file per core fn would be sane.

For now i'm running the tests using clojure, getting it to with jank itself should be the next logical step.

Please don't focus on jank at all (and let's call this the clojure.core test suite). jank implements clojure.core, just as Clojure does. We can get the entire test suite running for Clojure JVM first, before worrying about jank, since jank's clojure.core progress is about at 20% right now.

[lafka]

... use ChatGPT (or similar) to generate these tests. The reason is, clojure.core has 551 defs and defns. If each one has a dozen unit tests, we're talking about several thousand tests. Writing each one by hand would be more than cumbersome, which is exactly why I think this test suite doesn't exist yet.

It's going to take quite some time and i won't be able to do everything by myself but I can at least start with the basic things. I was thinking to look for existing test cases in clojure and if none exists look for examples on clojuredocs.org, in clojurescript source, and in clojerl source. That should give a decent coverage.

What happens when the transducer arity is called?

Is this the case for (filter odd?) or (filter odd? [1 2 3])?

• What happens when the input is nil? (apply to all inputs)
• What happens when the input is an incorrect shape (i.e. a number instead of a sequence)? (apply to all inputs)

Sounds reasonable, I will include nil and at least one different type/shape to the inputs. Further edge cases can be added later on.

Perhaps we should iterate on the list before writing any further code and then be sure we structure the code for each function to follow the list. Since this will be thousands of test cases, I imagine one file per core fn would be sane.

I can restructure the tests added so far into separate files, grouping by prefix where possible (ie aset-* functions in aset.clj). Once we have a "minimal" repo to show we can follow up with Clojure team and see if it's useful to them.

I also need someone to review the tests. It would be good to get more people onboard before we have hundreds of files and tests cases. Do you know someone who could be up for the task?

[jeaye]

It's going to take quite some time and i won't be able to do everything by myself but I can at least start with the basic things. I was thinking to look for existing test cases in clojure and if none exists look for examples on clojuredocs.org, in clojurescript source, and in clojerl source. That should give a decent coverage.

Of course! You're not going to be alone in this. I'm thinking we can get a project going and then send a shout out to the community to have people jump in and tackle the tests for a particular fn.

What happens when the transducer arity is called?

Is this the case for (filter odd?) or (filter odd? [1 2 3])?

Yep, the first one. That arity returns a fn and can be used as a transducer.

I can restructure the tests added so far into separate files, grouping by prefix where possible (ie aset-* functions in aset.clj). Once we have a "minimal" repo to show we can follow up with Clojure team and see if it's useful to them.

I also need someone to review the tests. It would be good to get more people onboard before we have hundreds of files and tests cases. Do you know someone who could be up for the task?

First and foremost, me. You may ping me for this at any time and count on me to be adding tests as well. However, I'd request that before we get hundreds of files and test cases, we consider how we're going to approach them consistently. For example, copy/pasting examples from Clojure docs can be useful, but isn't going to get us consistent coverage. Having a check list of questions, like I started above, will ensure we're covering our bases for every fn, followed by the edge cases and oddities for that particular fn.

@Samy-33, would you be interested in adding tests and reviewing? This test suite will be instrumental in building confidence for jank's clojure.core.

Once we have some momentum here, I'll drop a note to the jank community and we'll see if we can get more coverage and reviews.

[Samy-33]

@jeaye, sure. Let's do this.

I'm only sparsely available this month. So, in the next couple of weeks I'll be participating in the review process. From the next month onward, I will start adding tests.

[lafka]

Going by the checklist I've added tests for aset-int here. I'll proceed with rest of the aset tests since they're more or less the same.

Going forward i'll add new tests as PRs to that repository and tag you both for review.

[jeaye]

Nice! I'll follow up with a new repo in the jank-lang org so we can all work there, once I've finished up a couple of things. Thanks for keeping this going, lafka. ❤️