Integrate X-LoRA #1491

EricLBuehler · 2024-02-20T12:51:53Z

Paper link: https://arxiv.org/abs/2402.07148

This PR integrates X-LoRA by creating a new tuner model type on the level of LoraModel. Please see #1472.

Changes

Although the new model type is a subclass of LoraModel, this is only an implementation detail to remove the need for nested PeftModels. In a similar vein of thought, I have updated the signatures of the tuner __init__ functions to allow the method swapping and ensure the xLoRAModel is a tuner and not on the "level" of PeftModel.

Update signatures of tuner __init__ functions to take a back reference (not used for all but xLoRAModel).
Implement and export xLoRAModel and xLoRAConfig.
The special API for X-LoRA is located in xLoRAModel.

Status

Instantiate a LoraModel somewhere before layer swapping in xLoRAModel.__init__
Fully integrate.
Test implementation.
Add documentation for methods.

BenjaminBossan · 2024-02-22T08:46:04Z

Let me know once this is ready for review.

EricLBuehler · 2024-02-22T12:55:00Z

Hi @BenjaminBossan, I think this is ready for review.

BenjaminBossan

Thanks so much for working on this PR. I did not do an in-depth review yet, but from what I saw it already looks very solid.

Before working on the details, I think it's best to get the overall design polished. For this, I still have a few questions, please check out my comments. Together, I'm sure we can figure out a way to simplify some of the code.

Also, it would really help to have examples and/or tests to see X-LoRA in action. Do you have something that you could add? Later, we should also work on documentation, but this can wait until for now.

Edit: For other readers, here is the paper link: https://arxiv.org/abs/2402.07148

src/peft/tuners/xlora/insertion.py

src/peft/tuners/xlora/model.py

BenjaminBossan · 2024-02-22T15:46:27Z

src/peft/tuners/xlora/classifier.py

+ npy = result.numpy()
+ numpy.save(file, npy)
+
+ def flush_log_scalings(self, path: str):


Could you explain why we need this? Is this just for debugging/monitoring purposes?

This API enables the user to get a log of the scalings. It is useful for generating visualizations such as this.

I see. In that case, there should be some kind of example provided to illustrate how to use this, I think otherwise users will not discover this feature.

Also, I'd suggest for this method to only return the indices_map and allow the caller to decide themselves if they want to save it on disk as json or do something else with it.

Wouldn't returning the indices_map (or maybe the seqlens_map would be better, as it contains the tensors) make flush_log_scalings redundant to get_scalings_log? Of course, a dedicated method to calculate the indices_map may be helpful, and given the addition of such a method I think removing flush_log_scalings would be OK.

Additionally, there is actually a wrapper method on XLoraModel with the rest of the API which contains a docstring. This way, it should be easy to find. This method in the classifier is actually internal, I have prefixed it with _ now.

BenjaminBossan · 2024-02-22T15:52:36Z

src/peft/peft_model.py

@@ -149,7 +155,7 @@ def active_adapters(self) -> list[str]:
 adapters = self.active_adapter
 if isinstance(adapters, str):
 adapters = [adapters]
- return adapters
+ return list(filter(lambda x: len(x) > 0, adapters))


Not sure why this is needed. Is this because of self.active_adapter = "". Could you please explain the reason behind that?

During some of the testing I did, self.active_adapters would return a function instead of executing the @property. Through debugging, I discovered that it was somehow connected to self.active_adapter not being set, as no adapter is initially loaded.

However, suppose that I could allow the default adapter to be loaded, and then delete it later. That should make those obsolete. I have pushed a commit to hopefully resolve this.

BenjaminBossan · 2024-02-22T15:53:30Z

src/peft/peft_model.py

@@ -123,7 +129,7 @@ def __init__(self, model: PreTrainedModel, peft_config: PeftConfig, adapter_name
 else:
 self._peft_config = None
 cls = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type]
- self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
+ self.base_model = cls(model, {adapter_name: peft_config}, adapter_name, self)


I really want to avoid this, as this looks like a mixing of concerns. Surely, we can figure out a better way. Could you explain why this was needed?

This was needed because in XLoraModel.__init__ we load adapters into the PeftModel via load_adapter and for use by the XLoraClassifier. This way, automatic loading is achieved. However, all that matters is that we are able to get some reference to PeftModel into the constructor.

Perhaps we could add some code after we call cls(model...), and check if self.base_model is a XLoraModel. Then, we could call a __post_init__ which would take the PeftModel self, and do the adapter loading. Would this be more elegant?

I just pushed a commit (c5cdfc3) which is the latest one currently for easy reversion. Does this resolve the concern?

src/peft/peft_model.py

BenjaminBossan · 2024-02-23T16:04:02Z

src/peft/tuners/xlora/classifier.py

+ npy = result.numpy()
+ numpy.save(file, npy)
+
+ def flush_log_scalings(self, path: str):


I see. In that case, there should be some kind of example provided to illustrate how to use this, I think otherwise users will not discover this feature.

Also, I'd suggest for this method to only return the indices_map and allow the caller to decide themselves if they want to save it on disk as json or do something else with it.

src/peft/tuners/xlora/classifier.py

src/peft/tuners/xlora/model.py

BenjaminBossan · 2024-02-23T16:19:01Z

src/peft/tuners/xlora/model.py

+ model_peft.get_nb_trainable_parameters,
+ model_peft.generate,
+ )
+ model_peft.save_pretrained = peft_model_wrapper.save_pretrained # type: ignore


I don't think I fully understand this yet. Above, we call:

self.base_model.__xlora_post_init__(model, peft_config, adapter_name)

but here it looks like this method requires 4 arguments. What is the model_peft, is this the PeftModel?

So what I suspect is happening here is that you want to modify the behavior of generate and save_pretrained of the PeftModel without actually modifying PeftModel, is this the goal?

When it comes to generate, the PeftModel calls generate on the base_model anyway, could we not add the modification at that point?

When it comes to save_pretrained, I think we could check if we can somehow make the two approaches work together, I'd need to understand what the custom save_pretrained method does differently.

One easy way that we could allow custom save_pretrained methods without a lot of changes in PeftModel would be something like this:

class PeftModel: def save_pretrained(...): if hasattr(self.base_model, "_custom_save_pretrained"): return self.base_model._custom_save_pretrained(...) # same code as right now

This way, we only added 2 extra lines in PeftModel but would allow the underlying model to implement their own save_pretrained method.

BenjaminBossan · 2024-02-23T16:19:52Z

src/peft/tuners/xlora/insertion.py

+ # TODO(EricLBuehler): Evaluate effectiveness and performance degradation
+ self.peft_model.base_model.eval()
+ if not self.config.use_trainable_adapters:
+ for name, param in self.peft_model.base_model.named_parameters():


This looks strange to me, generate should normally not have anything to do with parameter updates. Could you explain why this is required?

Certainly! We discovered during training that the adapters were being set to trainable after each generate call. If you could provide any insight to why that may be the case, it would be great!

src/peft/tuners/xlora/model.py

BenjaminBossan · 2024-02-27T14:17:29Z

We discovered during training that the adapters were being set to trainable after each generate call.

This should certainly not happen. Could you please open an issue and provide an example so that we can fix it?

Regarding the state of the PR, it's actually a bit hard for me to tell which comments have been addressed and which haven't (as GH removes them when the line has been changed, even if it may not have been addressed). Regardless of some designs which I think could be improved, I think the most efficient way forward would be if you could provide an example/tests so that I can see X-LoRA in action. This may help answer some questions I still have.

EricLBuehler · 2024-02-27T19:40:22Z

Thank you, I work on some example code to reproduce the behavior and will raise an issue.

Regarding examples of X-LoRA, we have put together some examples using the xlora package API here. Although this is slightly different (ex., add_xlora_to_model and from_pretrained which are parts of get_peft_model and PeftModel respectively in this PR) from the XLoraModel proposed here, it documents the same methods.

I hope this will work as a demonstration of X-LoRA in action, would it be better for me to provide some examples using this PR?

BenjaminBossan · 2024-02-28T10:53:58Z

Regarding examples of X-LoRA, we have put together some examples using the xlora package API here. Although this is slightly different (ex., add_xlora_to_model and from_pretrained which are parts of get_peft_model and PeftModel respectively in this PR) from the XLoraModel proposed here, it documents the same methods.

I hope this will work as a demonstration of X-LoRA in action, would it be better for me to provide some examples using this PR?

I took a look at these examples, thanks. But let's get something to work based on this PR, even if very basic. This helps me understand what code paths are taken and how the different parts interact. Otherwise, the review is much harder for me. Also, we will need something like this eventually because we want to add tests for X-LoRA. As I said, it can be very basic for a start.

EricLBuehler · 2024-02-28T16:42:55Z

I was able to put together a small example which shows how one would use the API as documented here, which I have attached in a plain text file as GH does not allow .py to be attached.
example.txt

This is very basic and just shows creation, generation and simple API usage. I hope this helps!

BenjaminBossan · 2024-02-29T14:57:49Z

I tried to run your example but encountered some problems:

There is still a merge conflict in the forward method of lora.Linear. This is because we merged DoRA recently. My suggestion would be to apply X-LoRA only when not using DoRA. A quick and dirty solution should be enough for now, I think we'll rework that part in the future anyway.
As I don't have the adapter checkpoints referenced in the script, I tried to create some checkpoints with randomly initialized LoRA weights. However, this led to a bizarre error when trying to load them. It turns out that when I tried to create a normal LoRA adapter, it was not applied to any layer, despite setting target_modules correctly. Could it be possible that something has been messed up?

EricLBuehler · 2024-02-29T15:35:06Z

Thank you for trying it out. I ran the following code on a local machine with the latest installation from this branch to test the loading of a normal LoRA adapter, and it seemed to work, as after printing the model I can see lora.Linear in the specified layers.

from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "HuggingFaceH4/zephyr-7b-beta"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=[
        "q_proj",
        "gate_proj",
        "o_proj",
        "v_proj",
        "k_proj"
    ],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")
print(model)

Strangely, when I printed out the model in the X-LoRA test script, it showed proper injection of adapters as well as the classifier. To begin fixing this, could you please provide a minimum-reproducible example for the LoRA adapter loading so that I can find the error?

BenjaminBossan · 2024-02-29T15:55:39Z

could you please provide a minimum-reproducible example for the LoRA adapter loading so that I can find the error?

Using your exact branch with the merge conflict removed, when I run your script with this slight modification, I get the issue of no LoRA weights being applied:

- model.add_adapter(lora_config, adapter_name="adapter_1")
+ from peft import get_peft_model
+ model = get_peft_model(model, lora_config)

EricLBuehler · 2024-02-29T16:23:14Z

I found the bug: it was because of a mistake in a hasattr check. I added this because an XLoraModel should not have a default adapter injected. Perhaps you could try it again?

BenjaminBossan · 2024-03-01T10:45:06Z

Thanks for fixing this, it should now be working. There is still an issue with a merge conflict being unresolved as mentioned earlier:

There is still a merge conflict in the forward method of lora.Linear. This is because we merged DoRA recently. My suggestion would be to apply X-LoRA only when not using DoRA. A quick and dirty solution should be enough for now, I think we'll rework that part in the future anyway.

EricLBuehler · 2024-03-01T16:42:12Z

I have now fixed the merge conflict.

EricLBuehler · 2024-03-05T13:37:43Z

Hi @BenjaminBossan, I'm not sure if you have had a chance to look at the updated PR, as merge conflict has been resolved, and I think it is ready for another review. Would you prefer a new PR to be opened as a cleaner slate for further reviews?

BenjaminBossan

Thanks a lot for working on my previous comments. I was busy with some other stuff and only today could do another in detailed review of your PR. As I have to go now, it is unfortunately not 100% complete, but I still wanted to give you the feedback so you don't have to wait.

In addition to the individual comments I made, I have a few general questions:

How is the X-LoRA adapter trained? Could you please provide an example? Eventually, we'll want to move this to unit tests.
Could you please add the copyright notice to all new modules?
X-LoRA only really works with transformers language models, right? Can we document this more clearly? Also, do you think it would be possible to make this work with other types of models?
I'm not a fan of the type annotations of the style self.inner: nn.ModuleList = nn.ModuleList([]) or model: nn.Module = self.model, especially when followed by a # type: ignore. Same with the use of typing.cast. Is that because your IDE flags the code otherwise? Maybe you could deactivate the type checker for this project, as PEFT isn't really well annotated.

src/peft/tuners/tuners_utils.py

BenjaminBossan · 2024-03-05T16:20:48Z

src/peft/peft_model.py

+ if not isinstance(config, XLoraConfig):
+ raise TypeError(f"Expected 'XLoraConfig', got '{type(config)}' instead.")
+
+ device = infer_device() # As in PeftModel.load_adapter, torch_device = infer_device(


comment is cut off?

src/peft/tuners/xlora/classifier.py

src/peft/tuners/xlora/config.py

src/peft/tuners/xlora/model.py

BenjaminBossan · 2024-03-05T16:51:11Z

src/peft/tuners/xlora/model.py

+ ) -> None:
+ super().__init__(model, config, adapter_name)
+
+ def _xlora_post_init(


I wonder if it wouldn't be better to convert this to a standalone function, something like def post_init_lora(peft_model). Not sure if we need all the other arguments, can they not be derived from the PeftModel instance?

I'm not sure. Most of those are deeply nested by the time post_init_lora is called, so I thought it would increase readability to pass it this way. Would you prefer it if they are accessed through the PeftModel?

src/peft/tuners/xlora/util.py

BenjaminBossan · 2024-03-05T16:56:04Z

src/peft/peft_model.py

+ config.device = torch.device(device)
+
+ # If we are passed adapters in the kwargs, it is already in the config.
+ # If no adapters are passed, config.adapters is None


Do I understand correctly that this is for the case where we call save_pretrained on the X-LoRA model and then load this pretrained model again with from_pretrained? The only "new" thing added in that case would be the X-LoRA classifier, right?

Yes, that is correct. I simply load the weights for the classifier here.

BenjaminBossan · 2024-03-05T17:14:05Z

src/peft/tuners/lora/layer.py

@@ -434,7 +446,14 @@ def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
 x = x.to(lora_A.weight.dtype)

 if not self.use_dora[active_adapter]:
- result = result + lora_B(lora_A(dropout(x))) * scaling
+ if _xlora_layer is not None:


I'm not so happy with this addition to the LoraLayers. It makes reading and understanding them more complex and requires all LoRA layers to be updated (e.g. what about the bnb lora layers?).

I couldn't come up with an alternative yet, but I wonder if we could achieve something with wrapping and/or forward hooks. I'll continue to think about this tomorrow but wanted to let you know already in case you have some ideas.

I agree, it is not very easy to read. Perhaps we could implement some sort of hook in the LoraLayer so that techniques such as DoRA and X-LoRA could use that instead of modifying the layer source?

EricLBuehler · 2024-03-06T13:29:37Z

Thank you for your review! I have updated the code with your suggestions.

How is the X-LoRA adapter trained? Could you please provide an example? Eventually, we'll want to move this to unit tests.
Each LoRA adapter for an X-LoRA model is trained by training a LoRA adapter separately. The weights are then loaded, and the X-LoRA model is trained by only training the classifier. Would you recommend that I add a few sentences to the docstring detailing what is above, or some other code example? If a code example would be better, what part of the X-LoRA training process should I show?
Could you please add the copyright notice to all new modules?
I have added it to each new module.
X-LoRA only really works with transformers language models, right? Can we document this more clearly? Also, do you think it would be possible to make this work with other types of models?
With the current implementation of the classifier, it will not work with other types of models as it requires a sequence length. However, it would be possible to make it work with other types of models by changing the way the resulting scalings are reshape/expanded. I have added a note in the XLoraModel docstring.
I'm not a fan of the type annotations of the style self.inner: nn.ModuleList = nn.ModuleList([]) or model: nn.Module = self.model, especially when followed by a # type: ignore. Same with the use of typing.cast. Is that because your IDE flags the code otherwise? Maybe you could deactivate the type checker for this project, as PEFT isn't really well annotated.
Yes, my IDE would flag the code otherwise. I have removed these.

BenjaminBossan

Thanks for all the updates. I left a few comments, but let's focus more on the bigger picture and not each detail.

When I tried to run your code, I encountered an error:

AttributeError: 'XLoraConfig' object has no attribute 'target_modules'

I also think some other parts of the code don't quite work. To avoid this in the future, let's start adding unit tests. Let's start simple and add a new file tests/test_xlora.py with a functional test based on the example you posted earlier. It should also contain a test of training the XLoRA classifier.

Regarding the issue of only working with transformers language models, I think it's fine for the start. We can think of generalizing this in a follow up PR.

Again, thanks a lot for this contribution and your patience.

src/peft/tuners/xlora/classifier.py

BenjaminBossan · 2024-03-07T16:26:46Z

src/peft/tuners/xlora/classifier.py

+
+ logits = self.last.forward(hidden_state)
+
+ ### Repeat to make layerwise scalings if the classifier layer does not


Could you please add this explanation as a comment? Thanks.

src/peft/tuners/xlora/util.py

src/peft/tuners/xlora/model.py

src/peft/tuners/lora/layer.py

BenjaminBossan · 2024-03-26T13:04:39Z

How is the state of the PR? Let me know if you need any help with it.

BenjaminBossan · 2024-05-28T10:38:16Z

Ah, ok. Just to confirm, we should define a model hook inside _enable_peft_forward_hooks, which has access to the forward-kwargs and implements the dual forward pass mechanism? I think that would work.

Yes, so when you do module.register_forward_pre_hook(pre_forward, with_kwargs=True), the *args, **kwargs passed to pre_forward will be the one from the forward call. I don't think they should be shadowed by the generate-kwargs (name-wise), but even if they are, we could just give them a different name.

EricLBuehler · 2024-05-29T12:16:26Z

I just updated it to use the _enable_peft_forward_hooks and fully use hooks instead of overriding the forward method. Is this what you had in mind?

EricLBuehler · 2024-05-29T13:00:08Z

I think this should be ready for another review: there have been a significant amount of changes made, and the hacking is essentially gone.

BenjaminBossan

Thanks for the updates. I didn't do an in-depth review yet, as the CI is failing. I'll do a proper review once we get it to be green :)

As to why it's failing, part of this seems to be that you use some newer Python type annotation features that are not supported with older Python versions. Your safest bet is to add this to all modules that you added:

from __future__ import annotations

I'm not sure if this fully explains what's going, but let's start with that and see if the CI still complains.

src/peft/tuners/xlora/layer.py

BenjaminBossan · 2024-05-29T15:13:23Z

src/peft/tuners/lora/model.py

- peft_config.target_modules = set(
- TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING[model_config["model_type"]]
- )
+ if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING:


Is this change also needed for X-LoRA?

No, I don't think so. The X-LoRA model doesn't handle target modules at all, as it delegates that to the LoRA layers.

EricLBuehler · 2024-05-29T15:55:03Z

I added from __future__ import annotations to each new file, I think it should work now?

BenjaminBossan · 2024-05-29T16:13:44Z

I think the changes in _prepare_adapter_config are breaking a bunch of tests. Could you please check?

EricLBuehler · 2024-05-29T16:45:10Z

Ah ok, I think that should be fixed now.

BenjaminBossan · 2024-05-30T10:05:31Z

Thanks for the latest fix, but a bunch of tests are still failing. It seems that this is because of this line:

https://github.com/huggingface/peft/pull/1491/files#diff-95787fcc5356568929e383449ade2c1bac5da43deccab17318652b62ed171ae7R446

Probably, replacing it with if isinstance(getattr(model, "base_model", None), XLoraModel): should do the trick.

Generally, you could locally run pytest tests/test_custom_models.py, which is reasonably fast and covers a lot of PEFT code, before pushing your changes. That way, you have some quick feedback if something is broken and don't have to wait for me to trigger the CI.

EricLBuehler · 2024-05-30T12:09:58Z

I'm now getting this result, do you think it could be a hardware issue:

FAILED tests/test_custom_models.py::PeftCustomModelTester::test_merge_layers_fp16_33_Vanilla_MLP_1_LOHA - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

However, I don't think any of the changes in this PR would affect that?

BenjaminBossan · 2024-05-30T13:40:16Z

I'm now getting this result, do you think it could be a hardware issue:

My guess is that you're using an older PyTorch version. This should not be related to this PR.

EricLBuehler · 2024-05-30T15:27:50Z

Ok, thanks! After updating, pytest tests/test_custom_models.py passes all tests.

EricLBuehler · 2024-05-30T16:39:01Z

I think the tests are failing because we are trying to test on the CPU while the tests cast to CUDA. It uses the infer_device function now.

BenjaminBossan · 2024-05-31T09:26:44Z

Tests are still failing. One fix is very easy, in the TestXlora class, you set torch_device = infer_device() but later you use self.device instead of self.torch_device. Then there is a failure in test_from_peft_type, which is missing the hidden_size argument. Generally, this occurs because the XLoraConfig has a few required arguments to be initialized. So either you'd have to partially initialize it for this test or provide some default values.

EricLBuehler · 2024-05-31T15:07:35Z

I fixed those tests and added some fake defaults for hidden_size and adapters accompanied by a warning. I'm not sure if that is the best way to do it, though, but I didn't want to change the test for 1 adaption method.

BenjaminBossan · 2024-05-31T15:32:58Z

Nice @EricLBuehler now we're down to a single failing test: test_functional_embedding. Could you please take a look?

EricLBuehler · 2024-05-31T15:37:29Z

Ok, I think that's fixed: it was a small bug. It's strange why the tests I ran locally didn't catch it!

BenjaminBossan · 2024-05-31T15:58:13Z

Again an error:

          device = module.lora_embedding_A[next(iter(module.lora_embedding_A))].weight.device
E AttributeError: 'Parameter' object has no attribute 'weight'

This should be fixed by just removing the .weight from the call.

It's strange why the tests I ran locally didn't catch it!

How do you run the tests locally?

BenjaminBossan reviewed Feb 22, 2024

View reviewed changes

BenjaminBossan reviewed Feb 23, 2024

View reviewed changes

BenjaminBossan reviewed Mar 5, 2024

View reviewed changes

BenjaminBossan reviewed Mar 7, 2024

View reviewed changes

EricLBuehler added 11 commits March 29, 2024 13:48

Initial commit of integration

9aeb69c

Pass a reference back to the peftmodel when creating

b9d3878

Check base model in from_pretrained

bbd3ad4

Fix assert, attr

ea53917

Fix inheritance from peft model

79c4df9

Update inheritance again

2ee88af

Update inheritance again and properly instantiate

19063dc

Update comment

d364e5b

Export config, model

c9ab310

Remove use of default attr

9ed665e

Remove use of default attr

ae32006

EricLBuehler added 2 commits May 29, 2024 08:02

Use peft forward hook

1735deb

Fix disable adapter test

d661266

Remove unnecessary var

8601832

EricLBuehler requested a review from BenjaminBossan May 29, 2024 12:58

BenjaminBossan reviewed May 29, 2024

View reviewed changes

Add futures annotations

a3b83c1

EricLBuehler force-pushed the add_xlora branch from 4ddef32 to a3b83c1 Compare May 29, 2024 15:53

EricLBuehler added 2 commits May 29, 2024 12:29

Merge remote-tracking branch 'upstream/main' into add_xlora

e3db8df

Fix changes to lora prepare adapter config

b0f3062

Fix tests

81e337d

EricLBuehler added 2 commits May 30, 2024 11:49

Fix hacks

7ddd14b

Use infer_device

7ea1a12

Fix the tests and provide fake default values

7058c15

Fix a mistake

587c78a


		logits = self.last.forward(hidden_state)

		### Repeat to make layerwise scalings if the classifier layer does not

Integrate X-LoRA #1491

Are you sure you want to change the base?

Integrate X-LoRA #1491

Conversation

EricLBuehler commented Feb 20, 2024 • edited

Changes

Status

BenjaminBossan commented Feb 22, 2024

EricLBuehler commented Feb 22, 2024

BenjaminBossan left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminBossan commented Feb 27, 2024

EricLBuehler commented Feb 27, 2024

BenjaminBossan commented Feb 28, 2024

EricLBuehler commented Feb 28, 2024

BenjaminBossan commented Feb 29, 2024

EricLBuehler commented Feb 29, 2024 • edited

BenjaminBossan commented Feb 29, 2024

EricLBuehler commented Feb 29, 2024 • edited

BenjaminBossan commented Mar 1, 2024

EricLBuehler commented Mar 1, 2024

EricLBuehler commented Mar 5, 2024

BenjaminBossan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricLBuehler commented Mar 6, 2024

BenjaminBossan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminBossan commented Mar 26, 2024

BenjaminBossan commented May 28, 2024

EricLBuehler commented May 29, 2024

EricLBuehler commented May 29, 2024

BenjaminBossan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricLBuehler commented May 29, 2024

BenjaminBossan commented May 29, 2024

EricLBuehler commented May 29, 2024

BenjaminBossan commented May 30, 2024

EricLBuehler commented May 30, 2024

BenjaminBossan commented May 30, 2024

EricLBuehler commented May 30, 2024

EricLBuehler commented May 30, 2024

BenjaminBossan commented May 31, 2024

EricLBuehler commented May 31, 2024 • edited

BenjaminBossan commented May 31, 2024

EricLBuehler commented May 31, 2024 • edited

BenjaminBossan commented May 31, 2024 • edited

EricLBuehler commented Feb 20, 2024 •

edited

BenjaminBossan left a comment •

edited

EricLBuehler commented Feb 29, 2024 •

edited

EricLBuehler commented Feb 29, 2024 •

edited

EricLBuehler commented May 31, 2024 •

edited

EricLBuehler commented May 31, 2024 •

edited

BenjaminBossan commented May 31, 2024 •

edited