MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty #20694

mdhaber · 2024-05-10T23:44:43Z

Reference issue

Address stats part of gh-19805

What does this implement/fix?

Eliminates inconsistent errors and excessive warnings emitted by most stats reducing functions when 1D input is too small. Documents required sample sizes. Emits a single warning when NaN outputs are generated without NaNs in the input or when nan_policy='omit'.

Return values (usually NaNs) are already correct for the most part; this just deals with warnings and errors.

Examples below are for skew, but the behavior of most reducing functions is adjusted.

Before:

import numpy as np
from scipy import stats

stats.skew([])  # too noisy
# /usr/local/lib/python3.10/dist-packages/scipy/stats/_stats_py.py:1193: RuntimeWarning: Mean of empty slice.
#   mean = a.mean(axis, keepdims=True)
# /usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:121: RuntimeWarning: invalid value encountered in divide
#   ret = um.true_divide(
# /usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
#   return _methods._mean(a, axis=axis, dtype=dtype,
# /usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
#   ret = ret.dtype.type(ret / rcount)
# nan

stats.skew([np.nan], nan_policy='omit')  # too noisy
# /usr/local/lib/python3.10/dist-packages/scipy/stats/_stats_py.py:1193: RuntimeWarning: Mean of empty slice.
#   mean = a.mean(axis, keepdims=True)
# /usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:121: RuntimeWarning: invalid value encountered in divide
#   ret = um.true_divide(
# /usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
#   return _methods._mean(a, axis=axis, dtype=dtype,
# /usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
#   ret = ret.dtype.type(ret / rcount)
# nan

stats.skew([[]], axis=1)  # silent
# array([nan])

stats.skew([[np.nan]], nan_policy='omit', axis=1)  # silent
# array([nan])

After: (maybe I should change "is too small" → "contains too few observations")

import numpy as np
from scipy import stats

stats.skew([])
# UserWarning: One or more sample arguments is too small; all returned values will be NaN. See documentation for sample size requirements.
#   stats.skew([])
# nan

stats.skew([np.nan], nan_policy='omit')
# After omitting NaNs, one or more sample arguments is too small; all returned values will be NaN. See documentation for sample size requirements.
#  stats.skew([np.nan], nan_policy='omit')
# nan

stats.skew([[]], axis=1)
# UserWarning: All axis-slices of one or more sample arguments are too small; all elements of returned arrays will be NaN. See documentation for sample size requirements.
#  stats.skew([[]], axis=1)
# array([nan])

stats.skew([[np.nan]], nan_policy='omit', axis=1)
# UserWarning: After omitting NaNs, one or more axis-slices of one or more sample arguments is too small; corresponding elements of returned arrays will be NaN. See documentation for sample size requirements.
#  stats.skew([[np.nan]], nan_policy='omit', axis=1)
# array([nan])

Additional information

Consider ignoring whitepace changes and reviewing the commits separately. Each has a distinct purpose summarized by the commit message.

@rgommers @h-vetinari all I'd ask for now (if you're still interested in gh-19805) is to 1) do some before/after testing to see whether you prefer the new behavior (almost all reducing functions are fair game) and 2) help me answer these two questions.

Questions before I continue:

Initially, I've changed all the tests that looked for a function-specific error message or warning; now they look for the standard behavior. It took a lot of time, but I think it was important to investigate each case. That said, should I just eliminate these function-specific tests? I would want to add a generic test anyway, and that would make these scattered tests redundant.
Should the generic test (which looks for the correct warning and return value) check every function, or should I just check, say, a few representative functions (e.g. wilcoxon with one argument, mannwhitneyu with two arguments, and kruskal with three arguments)?

To do:

fix special cases (e.g. mode, f_oneway)
fix _axis_nan_policy test - it's very broken because it actually looked for the warnings/errors produced by each function
add test that correct warning is emitted in each case
add function-specific warning information

For a follow-up - fix for array-API.

We still need to adjust a lot of things about the documentation. Some documentation assumes the samples are 1D, whereas in others it is careful to refer to axis-slices of ND arrays. I would prefer that we write the API documentation using 1D language but consistently link to a tutorial that describes how axis/nan_policy/keepdims work with N-D arrays. That's for a separate PR.

Also, most hypothesis tests can easily get a method argument that computes a more accurate p-values instead of returning NaN for small samples (e.g. like pearsonr and anderson_ksamp now have). That can be a series of follow-up PRs.

… samples

[skip ci]

mdhaber · 2024-05-20T19:24:56Z

@h-vetinari @rgommers @ilayn As participants in gh-19805, if you could suggest answers to these two questions and compare the before/after behavior, I'd appreciate it.

Many stats functions had tests that looked for a function-specific error message or warning when there were too few observations. I changed each of these tests to look for the new, standard behavior instead. It took a lot of time, but I think it was important to investigate each case. That said, should I just eliminate these function-specific tests? I need to add a generic test that looks for the new behavior anyway, and that would make these scattered tests redundant.
The generic test will looks for the correct warning and return value. Should it check every function, or should I just check, say, a few representative functions (e.g. wilcoxon with one argument, mannwhitneyu with two arguments, and kruskal with three arguments)?

This PR is going to accumulate merge conflicts, and it would be nice to get it into the release candidate to get wider feedback on the elimination of the ad hoc warnings/errors and new standardized warnings.

h-vetinari

Generally this looks really nice! :)

I changed each of these tests to look for the new, standard behavior instead.

Love this!

That said, should I just eliminate these function-specific tests? I need to add a generic test that looks for the new behavior anyway, and that would make these scattered tests redundant.

Fine by me to just do a generic test (modulo caveats below)

Should it check every function, or should I just check, say, a few representative functions

Parametrization is easy and if the tests run fast, I don't see a problem to run this for (almost) every function.

h-vetinari · 2024-05-20T23:58:57Z

scipy/stats/_hypotests.py

- if (nx < 5) or (ny < 5):
- raise ValueError('x and y should have at least 5 elements, but len(x) '
- f'= {nx} and len(y) = {ny}.')


I don't think we should remove more specific error messages that are only valid for certain functions. It's certainly beneficial to add that constraint to the docstring, but only adding it there and not enforcing it anymore is IMO a step back.

I understand that this makes testing harder or less consistent, but I'd still prefer to skip/special-case these rather than throw away that extra information.

We can probably keep them for for 1D input. What I'll do is try-except the function and append the error message to the warning or something. For ND I would prefer not to do this for now: no warning is emitted at all now, so emitting some sort of warning is only a step in the right direction.

but only adding it there and not enforcing it anymore is IMO a step back.

Wait - this makes me think that there might be a misunderstanding. These limitations are still enforced, but rather than raising a ValueError or return NaN and emit multiple warnings (which may or may not be informative), the function returns nan and emits a standard "small sample" warning that refers to the documentation for specific sample size limitations.

My first response #20694 (comment) still holds - for 1D input, we could append this function-specific information to the warning message without too much trouble. LMK if that's what you'd like to see.

Update: never mind. I think we're on the same page. #20694 (comment)

h-vetinari · 2024-05-20T23:59:18Z

scipy/stats/_morestats.py

- if N < 3:
- raise ValueError("Data must be at least length 3.")


In this case, we could probably append "Data must be at least length 3." to the more generic message, which would also make the testing still work (because extra characters afterwards don't break the matching of the error messages).

OK, and this makes me think that there is no misunderstanding : ) Yes, exactly - I wouldn't have to revamp the testing.

h-vetinari · 2024-05-21T00:01:19Z

scipy/stats/_morestats.py

- if N < 3:
- raise ValueError("Not enough observations.")


h-vetinari · 2024-05-21T00:01:32Z

scipy/stats/_stats_py.py

- if n < 8:
- raise ValueError(
- f"skewtest is not valid with less than 8 samples; {n} samples were given.")


h-vetinari · 2024-05-21T00:01:43Z

scipy/stats/_stats_py.py

- if n < 5:
- raise ValueError(
- "kurtosistest requires at least 5 observations; %i observations"
- " were given." % int(n))


scipy/stats/tests/test_axis_nan_policy.py

rgommers · 2024-05-21T08:07:24Z

Agree with @h-vetinari's answers to the two questions - choices LGTM.

…arnings

[skip cirrus] [skip circle]

[skip circle] [skip cirrus]

…ption

…keepdims

mdhaber

Some comments to aid the reviewer.

mdhaber · 2024-05-22T18:01:46Z

scipy/stats/_axis_nan_policy.py

 if any(contains_nan) and nan_policy == 'omit':
 # consider passing in contains_nan
 samples = _remove_nans(samples, paired)
+ too_small_msg = too_small_1d_omit


Provide specific warning message depending on whether the 1D sample is only too small after omitting NaNs or if there were not enough observations to begin with.

mdhaber · 2024-05-22T18:04:26Z

scipy/stats/_axis_nan_policy.py

- # if is_too_small(samples):
- # return tuple_to_result(NaN, NaN)
- # but some existing functions raise exceptions, and changing
- # behavior of those would break backward compatibility.


We decided this was OK.

mdhaber · 2024-05-22T18:06:03Z

scipy/stats/_axis_nan_policy.py

- # behavior of those would break backward compatibility.
+ if override['empty'] and is_too_small(samples, kwds):
+ warnings.warn(too_small_msg, SmallSampleWarning, stacklevel=2)
+ return tuple_to_result(*np.full(n_out, NaN))


The comment was very old, and it was from a time when the decorator only applied to hypothesis tests that returned a statistic and p-value. Now, the decorator is applied to more general reducing functions, so this is updated accordingly.

mdhaber · 2024-05-22T19:17:55Z

scipy/stats/_morestats.py

- # raise ValueError on empty input
- if N == 0:
- raise ValueError("Data input must not be empty")
-


I'm removing function-specific errors about empty input since they add no function-specific information.

mdhaber · 2024-05-22T19:18:37Z

scipy/stats/_morestats.py

@@ -376,10 +372,10 @@ def kstatvar(data, n=2, *, axis=None):
 N = data.shape[axis]

 if n == 1:
- return kstat(data, n=2, axis=axis) * 1.0/N
+ return kstat(data, n=2, axis=axis, _no_deco=True) * 1.0/N


_no_deco calls the function itself, without the decorator. I don't remember exactly why I was prompted to make this change here, but it's useful to reduce overhead anyway. There's no reason to use the decorated version here because any nans have already been omitted if necessary, etc.

mdhaber · 2024-05-22T20:49:02Z

scipy/stats/tests/test_morestats.py

+ sup.filter(RuntimeWarning, "invalid value encountered")
+ sup.filter(UserWarning, r"var\(\): degrees of freedom is <= 0.")
+ sup.filter(RuntimeWarning, "Degrees of freedom <= 0 for slice")


Again, we'll deal with other backends later.

mdhaber · 2024-05-22T20:52:51Z

scipy/stats/tests/test_variation.py

@@ -117,14 +123,11 @@ def test_mean_zero(self, xp):
 y2 = variation(x2, axis=1)
 xp_assert_equal(y2, xp.asarray([xp.inf, xp.inf]))

- @pytest.mark.parametrize('x', [[0.]*5, [], [1, 2, np.inf, 9]])
+ @pytest.mark.parametrize('x', [[0.]*5, [1, 2, np.inf, 9]])


The empty input case has a conceptually different reason for returning NaN, and it's tested by generic tests, so I've omitted it.

mdhaber · 2024-05-22T20:53:26Z

scipy/stats/tests/test_variation.py

@@ -173,7 +187,7 @@ def test_more_nan_policy_omit_tests(self, ddof, expected, xp):
 # The slightly strange formatting in the follow array is my attempt to
 # maintain a clean tabular arrangement of the data while satisfying
 # the demands of pycodestyle. Currently, E201 and E241 are not
- # disabled by the `# noqa` annotation.
+ # disabled by the `noqa` annotation.


Started getting a lint complaint.

mdhaber · 2024-05-22T21:03:18Z

scipy/stats/tests/test_stats.py

@@ -4959,7 +4993,10 @@ def test_ttest_rel_axis_size_zero(b, expected_shape):
 # The results should be arrays containing nan with shape
 # given by the broadcast nonaxis dimensions.
 a = np.empty((3, 1, 0))
- result = stats.ttest_rel(a, b, axis=-1)
+ with np.testing.suppress_warnings() as sup:
+ # first case should warn, second shouldn't?


Here's what this comment is about:

import numpy as np from scipy.stats import skew skew(np.ones((0, 1))) # array([nan]) and warning # SmallSampleWarning: All axis-slices of one or more sample arguments are too small; all elements of returned arrays will be NaN. See documentation for sample size requirements. skew(np.ones((0, 0))) # array([], dtype=float64)

Both of these calls take the statistic along a slice with length 0. However, the result of the second one is an empty array, so no NaNs actually get produced, and the warning seems unnecessary, so I didn't think the output was surprising enough to deserve a warning. LMK if you disagree.

mdhaber · 2024-05-22T21:10:18Z

scipy/stats/tests/test_stats.py

 result = stats.f_oneway([10], [11], [12], [13])
 assert_equal(result, (np.nan, np.nan))

- @pytest.mark.parametrize('args', [(), ([1, 2, 3],)])
+ @pytest.mark.parametrize('args', [([1, 2, 3],)])


This is technically undesirable:

from scipy import stats stats.f_oneway([]) # F_onewayResult(statistic=nan, pvalue=nan) and "too small" warning stats.f_oneway([1, 2, 3]) # at least two inputs are required; got 1.

This probably affects all decorated functions that accepts *args. It happens because the decorator produces the warning and result before checking the number of arguments.

I guess we leave this for a follow-up? If so, could you open an issue about it?

Fixed for f_oneway and added the test back.

ilayn · 2024-05-22T21:31:06Z

This one went a bit over my head to be honest. So I must leave this to you folks unfortunately. Apologies about that.

mdhaber · 2024-05-22T21:54:00Z

@ilayn If it helps, don't worry about the diff. If you're interested enough, just take a look at the example behavior in the top post. I don't think we need everyone to review line by line, but if you're willing to just play with a few stats functions (among those listed here) with empty arguments before and after this PR to confirm that we like the new behavior better, that would be appreciated.

h-vetinari

I didn't go over it with a fine-toothed comb, but basically this LGTM. Just some minor comments.

h-vetinari · 2024-05-23T01:33:53Z

scipy/stats/tests/test_stats.py

 result = stats.f_oneway([10], [11], [12], [13])
 assert_equal(result, (np.nan, np.nan))

- @pytest.mark.parametrize('args', [(), ([1, 2, 3],)])
+ @pytest.mark.parametrize('args', [([1, 2, 3],)])


I guess we leave this for a follow-up? If so, could you open an issue about it?

h-vetinari · 2024-05-23T01:35:09Z

scipy/stats/tests/test_axis_nan_policy.py

+ elif (nan_policy == 'omit' and data_generator in {"all_nans", "mixed"}
+ and hypotest not in too_small_special_case_funcs):
+ with pytest.warns(SmallSampleWarning, match=too_small_nd_omit):
+ hypotest(*data, axis=axis, nan_policy=nan_policy, *args, **kwds)


This is still an open TODO, or am I misunderstanding your comment?

h-vetinari · 2024-05-23T01:43:49Z

scipy/stats/_axis_nan_policy.py

@@ -579,13 +607,15 @@ def hypotest_fun_out(*samples, **kwds):
 return tuple_to_result(*res)

 # Addresses nan_policy == "omit"
+ too_small_msg = too_small_nd_omit


I think this assignment is redundant? We only have one user in the rest of the function, which could just use too_small_nd_omit directly

mdhaber · 2024-05-23T02:25:04Z

Yes, those things are still open items. I should rework the test before we merge, and I might be able to fix the issue with too few arguments not emitting an error easily. I also said I would try to include the more detailed information in the error messages, but I'd like to save that for a follow-up to prevent this from continuing to accumulate merge conflicts. I'll try to finish by tomorrow.

[skip cirrus] [skip circle]

[skip circle] [skip cirrus]

mdhaber · 2024-05-24T14:16:56Z

OK, I think this is now doing what I intended to do here. A few limitations:

It only emits the generic warning for now rather than appending a function-specific part of the message. I can try to append the function-specific part as described above in a follow-up.
The issue with calling f_oneway with too few arguments is resolved, but the problem existed before this PR., and it applies to other functions that accepts *args, so I think there is a more general fix to be applied. I can fix that in the same follow-up.
It doesn't improve the behavior for other array API backends; those still emit warnings or raise errors as before. We'll want to rework the decorator significantly for array API support (and maybe actually modify most functions to handle nan_policy with n-D arrays/axis natively using something like WIP: stats.masked_array: array API compatible masked arrays #20363), and we'll fix this then.

One remaining question: by design, this is going to produce some warnings where there were none before. Should we make the SmallSampleWarning public to make them easier to filter?

scipy/stats/_stats_py.py

mdhaber added 4 commits May 10, 2024 09:29

MAINT: stats: silence warnings of reducing functions

9c9e9ff

ENH: stats: emit consistent warning from reducing functions for small…

2e3416c

… samples

TST: stats: adjust most old tests about too small samples

1ed5e37

MAINT: stats: remove dead code

bf8f004

[skip ci]

mdhaber added scipy.stats maintenance Items related to regular maintenance tasks labels May 10, 2024

mdhaber requested review from rgommers and h-vetinari May 10, 2024 23:44

mdhaber changed the title ~~WIP/MAINT: stats: enforce consistent behavior when sample is too small or empty~~ WIP/MAINT: stats: enforce consistent reducing function warning when sample is too small or empty May 10, 2024

mdhaber changed the title ~~WIP/MAINT: stats: enforce consistent reducing function warning when sample is too small or empty~~ WIP/MAINT: stats: enforce consistent warning from reducing functions when sample is too small or empty May 11, 2024

mdhaber changed the title ~~WIP/MAINT: stats: enforce consistent warning from reducing functions when sample is too small or empty~~ WIP/MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty May 11, 2024

h-vetinari reviewed May 21, 2024

View reviewed changes

mdhaber added 2 commits May 20, 2024 18:40

Merge remote-tracking branch 'upstream/main' into stats_warnings

fd8bf94

MAINT: stats: undo removal of some function-specific code

0826633

mdhaber mentioned this pull request May 21, 2024

TST: stats: refactor tests of normality tests #20756

Merged

mdhaber added 13 commits May 21, 2024 07:51

STY: stats: fix PEP8

0485c41

TST: stats.bartlett: skip test with jax backend

28c0d32

TST: stats._axis_nan_policy: adjust test

4179dcf

Merge remote-tracking branch 'upstream/main' into stats_warnings

77b5852

TST: stats._axis_nan_policy: generic tests for standard 'too small' w…

2288837

…arnings

TST: stats: fixup and raise ValueError for debugging CI

072dd0a

[skip cirrus] [skip circle]

MAINT: stats.axis_nan_policy: try to catch warnings as errors

78763a8

[skip circle] [skip cirrus]

MAINT: stats: introduce SmallSampleWarning

bf686fc

Merge remote-tracking branch 'upstream/main' into stats_warnings

a0b8269

TST: _lib.tests.test_warnings: add exception for axis/nan-policy tests

271ea2d

MAINT: stats._axis_nan_policy: remove unnecessary override['empty'] o…

2fe96e0

…ption

MAINT: stats._axis_nan_policy: fix interaction between too_small and …

e346907

…keepdims

MAINT: stats._axis_nan_policy: rename warning message variable

6924a8f

mdhaber added 2 commits May 22, 2024 12:52

MAINT: stats: tidying up PR

e6b4fee

MAINT: stats: more tidying up

61fdb49

mdhaber commented May 22, 2024

View reviewed changes

mdhaber mentioned this pull request May 22, 2024

WIP:ENH: stats.ttest_rel: add array API support #20762

Closed

h-vetinari reviewed May 23, 2024

View reviewed changes

mdhaber mentioned this pull request May 23, 2024

ENH: stats.ttest_ind: add array API support #20771

Open

mdhaber added 5 commits May 23, 2024 10:34

Merge remote-tracking branch 'upstream/main' into stats_warnings

31d763f

MAINT: stats.f_oneway: fix lack of error with <2 samples

619d460

TST: stats._axis_nan_policy: strengthen main test

82baf63

MAINT: stats._axis_nan_policy: refactor funny indexing

8bcb4dd

[skip cirrus] [skip circle]

MAINT: stats.ttest_rel/ttest_1samp: hopefully fix failures

68818e9

[skip circle] [skip cirrus]

mdhaber marked this pull request as ready for review May 24, 2024 00:52

mdhaber requested a review from tupui as a code owner May 24, 2024 00:52

mdhaber requested a review from h-vetinari May 24, 2024 00:52

mdhaber changed the title ~~WIP/MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty~~ MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty May 24, 2024

Merge branch 'main' into stats_warnings

82dff3a

mdhaber added this to the 1.14.0 milestone May 25, 2024

mdhaber commented May 25, 2024

View reviewed changes

scipy/stats/_stats_py.py Show resolved Hide resolved

mdhaber added 2 commits May 25, 2024 06:05

Apply suggestions from code review

fc1c827

TST: stats: use skip_xp_backends fixture properly

7cc797c

mdhaber mentioned this pull request May 25, 2024

DOC: stats: added Raises section to a few functions #20496

Draft

Merge remote-tracking branch 'upstream/main' into stats_warnings

3de8917

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty #20694

MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty #20694

mdhaber commented May 10, 2024 •

edited

mdhaber commented May 20, 2024 •

edited

h-vetinari left a comment

h-vetinari May 20, 2024

mdhaber May 21, 2024

mdhaber May 21, 2024 •

edited

h-vetinari May 20, 2024

mdhaber May 21, 2024

h-vetinari May 21, 2024

h-vetinari May 21, 2024

h-vetinari May 21, 2024

rgommers commented May 21, 2024

mdhaber left a comment

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

mdhaber May 22, 2024

h-vetinari May 23, 2024

mdhaber May 23, 2024 •

edited

ilayn commented May 22, 2024

mdhaber commented May 22, 2024 •

edited

h-vetinari left a comment

h-vetinari May 23, 2024

h-vetinari May 23, 2024

h-vetinari May 23, 2024

mdhaber commented May 23, 2024

mdhaber commented May 24, 2024 •

edited

		if N < 3:
		raise ValueError("Data must be at least length 3.")

MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty #20694

Are you sure you want to change the base?

MAINT: stats: make reducing functions emit consistent warning when sample is too small or empty #20694

Conversation

mdhaber commented May 10, 2024 • edited

Reference issue

What does this implement/fix?

Additional information

mdhaber commented May 20, 2024 • edited

h-vetinari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgommers commented May 21, 2024

mdhaber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber May 23, 2024 • edited

Choose a reason for hiding this comment

ilayn commented May 22, 2024

mdhaber commented May 22, 2024 • edited

h-vetinari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber commented May 23, 2024

mdhaber commented May 24, 2024 • edited

mdhaber commented May 10, 2024 •

edited

mdhaber commented May 20, 2024 •

edited

mdhaber May 21, 2024 •

edited

mdhaber May 23, 2024 •

edited

mdhaber commented May 22, 2024 •

edited

mdhaber commented May 24, 2024 •

edited