Generative vignette #24

Accio · 2023-08-22T15:28:38Z

Dear colleagues,

I have added a vignette using a simple generative model to show the benefit of randomization.

If you think the approach makes sense, I plan to add another 1-2 case studies, using more functionalities in our package, to make the case. Before I do that, please check out the simple example, and let me know what you think (example, coding style, wording, etc.). Criticism and suggestions are highly welcome.

Best regards, David

vignettes/generative_necessity.Rmd

julianesiebourg · 2023-09-04T16:13:13Z

Hi @Accio,
thank you for the nicely written example. Clearly, so far a simulation example was missing.
I like how you plot the different effects on the readouts separately and then on-top of each other.

One thing I was wondering is whether we, in addition to the benefit of randomization, also show the benefit of blocking?
We can use the optimize_design function to ensure that most treatments cover most rows and columns (e.g. the randomization is really efficient), and thus get even less bias from the plate effect.
Maybe in your example it does not make much of a difference, but if the randomization is 'unlucky' you could still get many samples of Compound1 in the same column and due to the plate gradient still draw wrong conclusions.
What do you think?
I'm happy to add that part!

Accio · 2023-09-04T16:22:42Z

Thank you for the kind words! Yes, your proposal makes perfect sense. Please do it.

I planned indeed to expand on the simple example. If you take over the blocking part, then I may contribute the next level: namely adding an operator/plate-batch effect on the top. Does that make sense? I can build then on what you have built.

Best regards and thanks again for the encouraging words.

julianesiebourg · 2023-09-06T19:13:54Z

Thanks David, yes, I'll add that part!

Merge branch 'main' into generative-vignette # Conflicts: # vignettes/basic_examples.Rmd

julianesiebourg

Hi David sorry, for the delay, I didn't notice I didn't submit my suggestions!!

julianesiebourg · 2024-03-12T10:21:25Z

vignettes/generative_necessity.Rmd

+
+What is the difference between the two variants? Option 2 apparently involves more planning and labor than option 1. If manual instead of robotic pipetting is involved, option 2 is likely error-prone. So why bothering considering the later option?
+
+Randomization pays off when unwanted variance is large enough so that it may distort our estimate of the quantity in which we are interested in. In our example, the unwanted variance may come from the *plate effect*: due to variances in temperature, humidity, and evaporation between wells in the plate, cells may respond differently to *even the same treatment*. Such *plate effects* are difficult to judge practically because they are not known prior to the experiment, unless a calibration study is performed where the cells in a microtiter plate are indeed treated with the same condition and measurements are performed in order to quantify the plate effect. However, it is simple to *simulate* such plate effects *in silico* with *a generative model*, and test the effect of randomization.


Suggested change

Randomization pays off when unwanted variance is large enough so that it may distort our estimate of the quantity in which we are interested in. In our example, the unwanted variance may come from the *plate effect*: due to variances in temperature, humidity, and evaporation between wells in the plate, cells may respond differently to *even the same treatment*. Such *plate effects* are difficult to judge practically because they are not known prior to the experiment, unless a calibration study is performed where the cells in a microtiter plate are indeed treated with the same condition and measurements are performed in order to quantify the plate effect. However, it is simple to *simulate* such plate effects *in silico* with *a generative model*, and test the effect of randomization.

Randomization pays off when unwanted variance is large enough so that it may distort our estimate of the quantity in which we are interested in. In our example, the unwanted variance may come from a *plate effect*: due to variances in temperature, humidity, and evaporation between wells in the plate, cells may respond differently to *even the same treatment*. Such *plate effects* are difficult to judge practically because they are not known prior to the experiment, unless a calibration study is performed where the cells in a microtiter plate are indeed treated with the same condition and measurements are performed in order to quantify the plate effect. However, it is simple to *simulate* such plate effects *in silico* with *a generative model*, and test the effect of randomization.

julianesiebourg · 2024-03-12T10:33:02Z

vignettes/generative_necessity.Rmd

@@ -0,0 +1,207 @@
+---
+title: "On the benefits of experiment design: a generative modelling approach"


Suggested change

title: "On the benefits of experiment design: a generative modelling approach"

title: "On the benefits of experiment design: a simulation approach"

julianesiebourg · 2024-03-12T10:34:44Z

vignettes/generative_necessity.Rmd

+---
+
+In this document, we demonstrate the necessity of a proper experiment design 
+with a generative model. We show that a proper experiment design helps


Suggested change

with a generative model. We show that a proper experiment design helps

with a generative model which we use to simulate data with "batch" effects. We show that a proper experiment design helps

julianesiebourg · 2024-03-12T11:21:30Z

vignettes/generative_necessity.Rmd

+conds <- c("DMSO", sprintf("Compound%02d", 1:11))
+dat <- data.frame(SampleIndex=1:96,
+ Compound=factor(rep(conds, 8), levels=conds),
+ rawRow=rep(1:8, each=12),
+ rawCol=rep(1:12, 8)) %>%
+ mutate(trueEffect=rnorm(96, mean=10, sd=1),
+ plateEffect=0.5 * sqrt((rawRow-4.5)^2 + (rawCol-6.5)^2),
+ measurement=trueEffect + plateEffect)
+bc <- BatchContainer$new(
+ dimensions = list("plate" = 1, 
+ row = list(values=LETTERS[1:8]), 
+ col = list(values=sprintf("%02d", 1:12)))
+)
+
+bc <- assign_in_order(bc, dat)
+
+head(bc$get_samples()) %>% gt::gt()


Suggested change

conds <- c("DMSO", sprintf("Compound%02d", 1:11))

dat <- data.frame(SampleIndex=1:96,

Compound=factor(rep(conds, 8), levels=conds),

rawRow=rep(1:8, each=12),

rawCol=rep(1:12, 8)) %>%

mutate(trueEffect=rnorm(96, mean=10, sd=1),

plateEffect=0.5 * sqrt((rawRow-4.5)^2 + (rawCol-6.5)^2),

measurement=trueEffect + plateEffect)

bc <- BatchContainer$new(

dimensions = list("plate" = 1,

row = list(values=LETTERS[1:8]),

col = list(values=sprintf("%02d", 1:12)))

)

bc <- assign_in_order(bc, dat)

head(bc$get_samples()) %>% gt::gt()

set.seed(2307111)

conditions <- c("DMSO", sprintf("Compound%02d", 1:11))

# set up samples with conditions and true effects

dat <- data.frame(SampleIndex = 1:96,

Compound = factor(rep(conditions, 8), levels = conditions),

trueEffect = rnorm(96, mean = 10, sd = 1))

# add the layout plus plate effect

dat <- dat %>%

mutate(

row=rep(1:8, each=12), col=rep(1:12, 8),

plateEffect=0.5 * sqrt((row-4.5)^2 + (col-6.5)^2),

measurement=trueEffect + plateEffect)

head(dat) %>% gt::gt()

Hi @Accio, how about writing it like this?
I know that you're setting up an example here that does not have treatment effect, but somehow for me it's more intuitive to first generate the samples with the compounds and the true effects and then generate the layout and add the plate effect.

Creating the batch container does not add much here as it is actually not made use of, and it confused me because I thought some optimization or randomization would be done with it.
We can plot the platelayout directly from dat.

All fine for me, thanks for going through my codes and making improving them!

julianesiebourg · 2024-03-12T11:26:35Z

vignettes/generative_necessity.Rmd

+cowplot::plot_grid(
+ plotlist = list(plot_plate(bc,
+ plate=plate,
+ row=row, column=col, .color=Compound,
+ title="Layout by treatment"),
+ plot_plate(bc,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(bc,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(bc,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow=2
+)


Suggested change

cowplot::plot_grid(

plotlist = list(plot_plate(bc,

plate=plate,

row=row, column=col, .color=Compound,

title="Layout by treatment"),

plot_plate(bc,

plate = plate, row = row, column = col, .color = trueEffect,

title = "True effect"

),

plot_plate(bc,

plate = plate, row = row, column = col, .color = plateEffect,

title = "Plate effect"

),

plot_plate(bc,

plate = plate, row = row, column = col, .color = measurement,

title = "Measurement"

)

), ncol = 2, nrow=2

)

cowplot::plot_grid(

plotlist = list(plot_plate(dat,

plate=plate,

row=row, column=col, .color=Compound,

title="Layout by treatment"),

plot_plate(dat,

plate = plate, row = row, column = col, .color = trueEffect,

title = "True effect"

),

plot_plate(dat,

plate = plate, row = row, column = col, .color = plateEffect,

title = "Plate effect"

),

plot_plate(dat,

plate = plate, row = row, column = col, .color = measurement,

title = "Measurement"

)

), ncol = 2, nrow=2

)

julianesiebourg · 2024-03-12T13:22:49Z

vignettes/generative_necessity.Rmd

+rand_dat <- data.frame(SampleIndex=1:96,
+ Compound=sample(factor(rep(conds, 8), levels=conds)),
+ rawRow=rep(1:8, each=12),
+ rawCol=rep(1:12, 8)) %>%
+ mutate(trueEffect=rnorm(96, mean=10, sd=1),
+ plateEffect=0.5 * sqrt((rawRow-4.5)^2 + (rawCol-6.5)^2),
+ measurement=trueEffect + plateEffect)
+
+bc2 <- BatchContainer$new(
+ dimensions = list("plate" = 1, 
+ row = list(values=LETTERS[1:8]), 
+ col = list(values=sprintf("%02d", 1:12)))
+)
+
+bc2 <- assign_in_order(bc2, rand_dat)
+head(bc2$get_samples()) %>% gt::gt()


Suggested change

rand_dat <- data.frame(SampleIndex=1:96,

Compound=sample(factor(rep(conds, 8), levels=conds)),

rawRow=rep(1:8, each=12),

rawCol=rep(1:12, 8)) %>%

mutate(trueEffect=rnorm(96, mean=10, sd=1),

plateEffect=0.5 * sqrt((rawRow-4.5)^2 + (rawCol-6.5)^2),

measurement=trueEffect + plateEffect)

bc2 <- BatchContainer$new(

dimensions = list("plate" = 1,

row = list(values=LETTERS[1:8]),

col = list(values=sprintf("%02d", 1:12)))

)

bc2 <- assign_in_order(bc2, rand_dat)

head(bc2$get_samples()) %>% gt::gt()

# add the layout plus plate effect

randomized_dat <- dat %>%

slice(sample(1:n())) %>% # shuffle the order of samples in the dataset

mutate(

row=rep(1:8, each=12), col=rep(1:12, 8),

plateEffect=0.5 * sqrt((row-4.5)^2 + (col-6.5)^2),

measurement=trueEffect + plateEffect)

head(randomized_dat) %>% gt::gt()

julianesiebourg · 2024-03-12T13:23:43Z

vignettes/generative_necessity.Rmd

+cowplot::plot_grid(
+ plotlist = list(plot_plate(bc2,
+ plate=plate,
+ row=row, column=col, .color=Compound,
+ title="Layout by treatment"),
+ plot_plate(bc2,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(bc2,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(bc2,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow=2
+)


Suggested change

cowplot::plot_grid(

plotlist = list(plot_plate(bc2,

plate=plate,

row=row, column=col, .color=Compound,

title="Layout by treatment"),

plot_plate(bc2,

plate = plate, row = row, column = col, .color = trueEffect,

title = "True effect"

),

plot_plate(bc2,

plate = plate, row = row, column = col, .color = plateEffect,

title = "Plate effect"

),

plot_plate(bc2,

plate = plate, row = row, column = col, .color = measurement,

title = "Measurement"

)

), ncol = 2, nrow=2

)

cowplot::plot_grid(

plotlist = list(plot_plate(randomized_dat,

plate=plate,

row=row, column=col, .color=Compound,

title="Layout by treatment"),

plot_plate(randomized_dat,

plate = plate, row = row, column = col, .color = trueEffect,

title = "True effect"

),

plot_plate(randomized_dat,

plate = plate, row = row, column = col, .color = plateEffect,

title = "Plate effect"

),

plot_plate(randomized_dat,

plate = plate, row = row, column = col, .color = measurement,

title = "Measurement"

)

), ncol = 2, nrow=2

)

julianesiebourg · 2024-03-12T13:24:20Z

vignettes/generative_necessity.Rmd

+
+```{r}
+randMeasureDiff <- TukeyHSD(aov(measurement ~ Compound, 
+ data=bc2$get_samples()))$Compound


Suggested change

data=bc2$get_samples()))$Compound

data=randomized_dat))$Compound

julianesiebourg · 2024-03-12T13:24:37Z

vignettes/generative_necessity.Rmd

+We can also use the boxplot as a visual help to inspect the difference between the treatments, to confirm that randomization prevents plate effect from affecting the statistical inference.
+
+```{r randBoxplot, fig.height=5, fig.width=5}
+ggplot(bc2$get_samples(), 


Suggested change

ggplot(bc2$get_samples(),

ggplot(randomized_dat,

julianesiebourg · 2024-03-12T13:27:36Z

vignettes/generative_necessity.Rmd

+
+## Discussions and conclusions
+
+The simple case study discussed in this vignette is an application of generative models, which means that assuming that we know the mechanism by which the data is generated, we can simulate the data generation process and use it for various purposes. In our cases, we simulated a linear additive model of true effects of compounds and control on cell viability and the plate effect induced by positions in a microtitre plate. Using the model, we demonstrate that (1) plate effect can impact statistical inference by introducing false positive (and in other case, false negative) findings, and (2) a full randomization can guard statistical inference by reducing the effect of plate effect.


Suggested change

The simple case study discussed in this vignette is an application of generative models, which means that assuming that we know the mechanism by which the data is generated, we can simulate the data generation process and use it for various purposes. In our cases, we simulated a linear additive model of true effects of compounds and control on cell viability and the plate effect induced by positions in a microtitre plate. Using the model, we demonstrate that (1) plate effect can impact statistical inference by introducing false positive (and in other case, false negative) findings, and (2) a full randomization can guard statistical inference by reducing the effect of plate effect.

The simple case study discussed in this vignette is an application of generative models, which means that assuming that we know the mechanism by which the data is generated, we can simulate the data generation process and use it for various purposes. In our cases, we simulated a linear additive model of true effects of compounds and control on cell viability and the plate effect induced by positions in a microtitre plate. Using the model, we demonstrate that (1) plate effect can impact statistical inference by introducing false positive (and in other case, false negative) findings, and (2) a full randomization can guard statistical inference by reducing the bias of the plate effect.

Accio

thanks!

Accio added 7 commits May 17, 2023 16:00

update the basic examples, to prepare for adding the generative model

be11f95

updated conclusion

7a17082

add JDZ as contributor

3df07be

add new line after title

fb2e9e6

add vignette to show the necessity

652f3dd

a generative model showing the necessity of designIt

47028c4

update the wording

ae05373

idavydov self-requested a review August 22, 2023 15:33

julianesiebourg reviewed Sep 4, 2023

View reviewed changes

vignettes/generative_necessity.Rmd Outdated Show resolved Hide resolved

vignettes/generative_necessity.Rmd Outdated Show resolved Hide resolved

julianesiebourg added the documentation Improvements or additions to documentation label Sep 4, 2023

julianesiebourg linked an issue Sep 6, 2023 that may be closed by this pull request

Expand generative model vignette with simple optimize design call #25

Open

julianesiebourg and others added 3 commits September 19, 2023 09:44

merge main resolve conflicts

ad54b52

Merge branch 'main' into generative-vignette # Conflicts: # vignettes/basic_examples.Rmd

Update vignettes/generative_necessity.Rmd

ae59048

Update vignettes/generative_necessity.Rmd

ec762f5

julianesiebourg requested changes Apr 15, 2024

View reviewed changes

Accio commented Apr 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generative vignette #24

Generative vignette #24

Accio commented Aug 22, 2023

julianesiebourg commented Sep 4, 2023

Accio commented Sep 4, 2023

julianesiebourg commented Sep 6, 2023

julianesiebourg left a comment

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

Accio Apr 15, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

julianesiebourg Mar 12, 2024

Accio left a comment


		What is the difference between the two variants? Option 2 apparently involves more planning and labor than option 1. If manual instead of robotic pipetting is involved, option 2 is likely error-prone. So why bothering considering the later option?

		Randomization pays off when unwanted variance is large enough so that it may distort our estimate of the quantity in which we are interested in. In our example, the unwanted variance may come from the plate effect: due to variances in temperature, humidity, and evaporation between wells in the plate, cells may respond differently to even the same treatment. Such plate effects are difficult to judge practically because they are not known prior to the experiment, unless a calibration study is performed where the cells in a microtiter plate are indeed treated with the same condition and measurements are performed in order to quantify the plate effect. However, it is simple to simulate such plate effects in silico with a generative model, and test the effect of randomization.

		@@ -0,0 +1,207 @@
		---
		title: "On the benefits of experiment design: a generative modelling approach"

	with a generative model. We show that a proper experiment design helps
	with a generative model which we use to simulate data with "batch" effects. We show that a proper experiment design helps

	data=bc2$get_samples()))$Compound
	data=randomized_dat))$Compound


		## Discussions and conclusions

		The simple case study discussed in this vignette is an application of generative models, which means that assuming that we know the mechanism by which the data is generated, we can simulate the data generation process and use it for various purposes. In our cases, we simulated a linear additive model of true effects of compounds and control on cell viability and the plate effect induced by positions in a microtitre plate. Using the model, we demonstrate that (1) plate effect can impact statistical inference by introducing false positive (and in other case, false negative) findings, and (2) a full randomization can guard statistical inference by reducing the effect of plate effect.

Generative vignette #24

Are you sure you want to change the base?

Generative vignette #24

Conversation

Accio commented Aug 22, 2023

julianesiebourg commented Sep 4, 2023

Accio commented Sep 4, 2023

julianesiebourg commented Sep 6, 2023

julianesiebourg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Accio left a comment

Choose a reason for hiding this comment