Bayesian power analysis #2507

lukebrawleysmith · 2024-05-10T19:53:44Z

This pull request extends power analysis to our Bayesian engine.

This PR permits Bayesian power analysis by updating the front end Power Calculation code.
A customer can now specify a prior just as they would during an experiment.
Power analysis defaults to default stats engine for an org, and frequentist vs Bayesian power analysis can now be toggled under Analysis Settings:

If the power is switched from frequentist to Bayesian, power is recalculated using the the org defaults for a metric. If desired, the customer may select Edit and update Prior Mean and Prior Standard Deviation under the Metrics modal:

github-actions · 2024-05-10T20:08:51Z

Your preview environment pr-2507-bttf has been deployed.

Preview environment endpoints are available at:

lukesonnet

Some smaller comments and one bigger comment.

Broad question: is the unit test here set up to mirror the python notebook and we should take the python notebook as the source of power validity, right?

packages/front-end/components/PowerCalculation/types.ts

packages/back-end/types/stats.d.ts

packages/front-end/components/PowerCalculation/stats.ts

lukesonnet · 2024-05-16T18:58:16Z

packages/front-end/components/PowerCalculation/stats.ts

+ true
+ );
+ if (maxPower < power) {
+ console.log(`failing at iteration j: %d`, j);


these will need to be logged later.

how frequently do we get in this case? Is this the case where the UI will show that N/A or - or something?

so far, this case has never happened. i had this for testing purposes, happy to remove it if you want.

i haven't rigorously resolved the upper bound issue yet, so i kept this in there for now.

If it could possibly happen, then we should have proper error handling for it, if it is impossible, we should remove it.

lukesonnet · 2024-05-16T19:33:36Z

packages/front-end/components/PowerCalculation/stats.ts

+ return mdeResults;
+}
+
+export function powerMetricWeeksBayesian(


I see why you asked about copying. There's a lot of complex logic here that it might benefit us to only write once. Take a look at this slight rewrite https://github.com/growthbook/growthbook/compare/ls/power-structure?expand=1

I've done two key things:

Make some of the code a bit more typescript friendly (e.g. don't unpack a bunch of stuff, use .forEach)

Ask you to change powerEst and powerEstBayesian to instead take an object with your parameters specified. Then build those parameters and specify which function to use in one single if-else block.

My code is psuedo-code, since I didn't create those other types, and can just serve as a reference.

I incorporated your pseudo-code. However, I didn't define new types as inputs for powerEst and powerEstBayesian, as the linter was fussing at me for having different types as inputs for a powerFunction function. Also, the way I have it is simple. However, if you have more elegant approach, I'm open to incorporating it. Also, I have statsEngineSettings as attribute of powerSettings, rather than standalone object, because we always want the freq statsEngineSettings to be part of PowerCalculationParams and bayesian statsEngineSettings to be associated with PowerCalculationParamsBayesian. Again, if you have better suggestion, I'm open.

github-actions · 2024-05-16T22:17:39Z

Deploy preview for docs ready!

✅ Preview
https://docs-eiplg9yes-growthbook.vercel.app

Built with commit af9d684.
This pull request is being automatically deployed with vercel-action

lukebrawleysmith · 2024-05-17T16:20:36Z

Some smaller comments and one bigger comment.

Broad question: is the unit test here set up to mirror the python notebook and we should take the python notebook as the source of power validity, right?

lukesonnet · 2024-05-22T15:14:53Z

packages/front-end/components/PowerCalculation/stats.ts

 n: number,
 nVariations: number,
 alpha: number = 0.05,
 twoTailed: boolean = true,
- sequentialTuningParameter = 0
+ sequentialTesting: false | number


In general I think having something typed as either false | number isn't the best approach if we can avoid it. It's a bit confusing, but no need to change it right now.

I agree, I can talk to Romain about this, he suggested this input when only freq power analysis was available. Now that we are implementing Bayesian, I can see if we can simplify.

lukesonnet · 2024-05-22T15:16:46Z

packages/front-end/components/PowerCalculation/stats.ts

+ mde: -999,
+ };
+ if (powerSettings.statsEngineSettings.type === "frequentist") {
+ thisPower = powerEst(


Should we call these methods powerEstFrequentist and findMdeFrequentist for consistency?

not sure why there is not an option to reply to the comment below ("If it could possibly happen, then we should have proper error handling for it, if it is impossible, we should remove it."), but I have revamped the bayesian mde code, and this does not exist anymore.

lukesonnet · 2024-05-22T15:18:39Z

packages/front-end/components/PowerCalculation/stats.ts

+ true
+ );
+ if (maxPower < power) {
+ console.log(`failing at iteration j: %d`, j);


If it could possibly happen, then we should have proper error handling for it, if it is impossible, we should remove it.

docs/docs/statistics/power.mdx

lukesonnet · 2024-05-22T15:28:42Z

docs/docs/statistics/power.mdx

@@ -107,6 +107,14 @@ In clinical trials, the standard is 80%. This means that if you were to run your

 That being said, running an experiment with less than 80% power can still help your business. The purpose of an experiment is to learn about your business, not simply to roll out features that achieve statistically significant improvement. The biggest cost to running low-powered experiments is that your results will be noisy. This usually leads to ambiguity in the rollout decision.

+## How do I run Bayesian power analysis?
+
+For Bayesian power analysis, you specify the prior distribution of the treatment effect (see here [to be written] for guidance regarding prior selection). We then estimate Bayesian power, which is the probability that the $(1 - \alpha)$ credible interval after the test does not contain 0.


This isn't the flow most people will use, since they will largely accept default priors for that metric. Maybe we need to hold off on this section until we have more details on what users will actually do, but I think for V1 of bayesian power users can't modify priors, so this section is misleading.

* Almost done wiring. * Almost there. * Default to org setting for engine. * Keep values in sync. * Split metric params.

lukebrawleysmith · 2024-05-30T17:24:57Z

Adding some context:

This PR permits Bayesian power analysis by updating the front end Power Calculation code.
A customer can now specify a prior just as they would during an experiment.
Power analysis defaults to default stats engine for an org, and frequentist vs Bayesian power analysis can now be toggled under Analysis Settings:

If the power is switched from frequentist to Bayesian, power is recalculated using the the org defaults for a metric. If desired, the customer may select Edit and update Prior Mean and Prior Standard Deviation under the Metrics modal:

I updated the testing files to match output from python notebook, where simulations confirmed that Bayesian power is accurate.

We still need to

resolve case where effect size does precede metric arguments on metric modal (currently challenging to reproduce)

lukesonnet

Left some main doc comments for @lukebrawleysmith and a few important FE comments for @romain-growthbook re: priors.

Specifically:

We need a much slimmer way for users to input priors. It's so overwhelming right now. IMO they should not change this every time they do a power calculation so we need those prior fields to either be in the StatsEngine modal or we need to greatly de-emphasize them (for example with a small text link "Override metric priors" to hide all the prior input fields until you click it to open the prior input fields).
We need to inherit metric specific priors that override the org defaults. I've left comments about this in the fiels.

lukesonnet · 2024-05-31T16:41:53Z

packages/front-end/components/PowerCalculation/PowerCalculationSettingsModal.tsx


 const form = useForm<PartialPowerCalculationParams>({
 defaultValues: params,
 });

+ const metrics = form.watch("metrics");
+ const defaultValues = Object.keys(config).reduce(


@romain-growthbook: Metrics can override organization defaults. We should apply those metric level settings instead of the org settings if they exist.

lukesonnet · 2024-05-31T16:41:57Z

packages/front-end/components/PowerCalculation/PowerCalculationStatsEngineSettingsModal.tsx

 ),
- disabled: true,


@romain-growthbook :

can we disable showing the Enable Sequential Testing div (the one on L89 below styled as className="row mt-2" unless "Frequentist" is the selected stats engine? You can see similar logic on the AnalysisForm

Can you use a resolver to pick the right stats engine default? You can see how we use scoped settings in AnalysisForm and StatsEngineSelect to pick the right stats engine (organization -> potential project level override).

design comment...

Furthermore, I think we should remove this entire span:

span> Enable Sequential Testing to look at your experiment results as many times as you like while preserving the false positive rate.{" "} <DocLink docSection="statisticsSequential"> Learn More <FaExternalLinkAlt /> </DocLink> </span>

And just put a learn more link to the docs next to the sequential testing header

lukesonnet · 2024-05-31T16:43:00Z

packages/stats/gbstats/devtools/simulation.py

@@ -56,6 +56,9 @@ def create_storage_arrays(self):
 self.lower_limit = np.empty(array_shape)
 self.upper_limit = np.empty(array_shape)
 self.results = np.empty(array_shape, dtype=object)
+ self.stat_a = np.empty(array_shape, dtype=object)
+ self.stat_b = np.empty(array_shape, dtype=object)
+ self.estimand = np.empty(array_shape)


@lukebrawleysmith , we need to store these in run_iteration()

I meant to delete this, sorry. It should be part of its own PR (I think it would be helpful for debugging to store these).

lukesonnet · 2024-05-31T16:47:16Z

packages/front-end/components/PowerCalculation/types.ts

+ metricType: "all",
+ type: "percent",
+ tooltip: "Prior mean for the relative effect size.",
+ defaultSettingsValue: (s) => s.metricDefaults?.priorSettings?.mean,


@romain-growthbook Related to my other comment about metric overrides, maybe here or somewhere else is where you need to go from organization to metric override (if override == true).

You can see this logic here: https://github.com/growthbook/growthbook/blob/main/packages/shared/src/experiments.ts#L187-L243

lukesonnet · 2024-05-31T16:47:45Z

packages/front-end/components/PowerCalculation/types.ts

+ title: "Use proper prior",
+ metricType: "all",
+ type: "boolean",
+ defaultSettingsValue: (s) => s.metricDefaults?.priorSettings?.override,


In addition to using override instead to see if the metric overrides should override the org settings, this needs to use the proper field, not override as the default.

lukesonnet · 2024-05-31T20:12:33Z

docs/static/Power.pdf

Delete this file?

lukesonnet · 2024-05-31T20:13:16Z

docs/docs/statistics/powerTechnical.mdx

+title: Power Analysis Technical Details
+description: Technical details of power analysis
+sidebar_label: Power Technical
+slug: powerTechnical


Follow the naming convention of all of the other docs. This should be power-technical.mdx and power-technical should be the slug.

lukesonnet · 2024-05-31T20:13:58Z

docs/docs/statistics/power.mdx

-where
-
-$\tilde{\sigma} = \hat{\sigma}*\sqrt{N}\sqrt{\frac{2(N\rho^2 + 1)}{N^2\rho^2}\log\left(\frac{\sqrt{N\rho^2 + 1}}{\alpha}\right)}\frac{1}{Z_{1-\alpha/2}}$.
+Bayesian MDE


Can we format this either bold or something? Right now it just looks like random text in the middle of the documentation.

thanks for catching typo, fixed

lukesonnet · 2024-05-31T20:14:41Z

docs/docs/statistics/power.mdx


 We make the following assumptions:

-1. equal sample sizes across control and treatment variations;
+1. equal sample sizes across control and treatment variations. If unequal sample sizes are used in the experiment, use the smaller of the two sample sizes. This will produce conservative power estimates.


Nice clarification, thanks.

lukesonnet · 2024-05-31T20:18:16Z

docs/docs/statistics/power.mdx

-## GrowthBook implementation
+## GrowthBook implementation (frequentist)
+
+Below we describe high-level technical details of our implementation. All technical details can be found [here](/statistics/powerTechnical).


Thanks for the updates here. I still think we need a bit more signposting and clear language.

Can we say something like

## GrowthBook implementation For both Bayesian and frequentist engines, we produce two key statistics: 1. Power - After running the experiment for a given number of weeks and for a hypothesized effect size, what is the probability of a statistically significant result? 2. Minimum Detectable Effect (MDE) - After running an experiment for a given number of weeks, what is the smallest effect that we can detect as statistically significant 80% of the time. Each engine arrives at these values differently...tech details link... etc... ### Frequentist engine ...clear signposting for **Power** and **MDE**... ### Bayesian engine ...clear signposting for **Power** and **MDE**...

thanks, I agree it's helpful for the key equations to really pop. Changes implemented.

Luke Smith and others added 25 commits April 3, 2024 19:23

first pass at simulation class

fe6b831

simulation class v1

012e9dc

Update sim study approach

53f4248

Fix mean

fc19111

Expose full result

a1201e7

Replace complex bayesian machinery with simpler model

55bcb98

Merge remote-tracking branch 'origin/main' into ls/bayesian-revamp

10dac5e

Fix tests; improper prior flag

a2cc3b2

Use ntests

a2523d7

Merge branch 'sonnet-smith-sim-study' into ls/bayesian-revamp

ed0476d

Update different prior types and rescaling

7d0ef0f

Merge remote-tracking branch 'origin/main' into ls/bayesian-revamp

4b0c7ed

Fix prior config

0559f3c

Fix tests

d7dd829

Retool how risk is calculated

a8be22a

Add risk type

9e6b850

Fix tests

2d889cb

Merge remote-tracking branch 'origin/main' into ls/bayesian-revamp

7144741

Ensure risk is flipped

e79c64d

Add risk type

d0bca1f

Fix test

fafa79f

Luke's comments; renaming

a8fd730

Merge remote-tracking branch 'origin/main' into ls/bayesian-revamp

c5321f2

first coding of bayesian power

2164d65

adding documentation

9edbd5d

lukebrawleysmith requested a review from lukesonnet May 10, 2024 19:53

Luke Smith added 3 commits May 13, 2024 10:41

linting changes

8d80674

linting changes

d93500c

adding documentation

fb27e5f

lukesonnet changed the base branch from main to node-sdk May 16, 2024 18:24

lukesonnet changed the base branch from node-sdk to main May 16, 2024 18:24

lukesonnet reviewed May 16, 2024

View reviewed changes

merging in changes from main

6a3f611

lukebrawleysmith closed this May 17, 2024

responding to sonnet comments

8c24b09

lukebrawleysmith reopened this May 20, 2024

lukesonnet reviewed May 22, 2024

View reviewed changes

romain-growthbook and others added 3 commits May 23, 2024 14:38

Romain/bayesian power (#2563)

998d5de

* Almost done wiring. * Almost there. * Default to org setting for engine. * Keep values in sync. * Split metric params.

Fix rounding error, wire engine type.

b73568b

updating documentation, responding to sonnet comments

f273721

updating technical docs

af9d684

lukebrawleysmith changed the title ~~Lukesmith bayesian power~~ Bayesian power analysis May 31, 2024

lukesonnet reviewed May 31, 2024

View reviewed changes

Bayesian power analysis #2507

Are you sure you want to change the base?

Bayesian power analysis #2507

Conversation

lukebrawleysmith commented May 10, 2024 • edited

github-actions bot commented May 10, 2024 • edited

lukesonnet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 16, 2024 • edited

lukebrawleysmith commented May 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukebrawleysmith commented May 30, 2024 • edited

lukesonnet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukebrawleysmith commented May 10, 2024 •

edited

github-actions bot commented May 10, 2024 •

edited

github-actions bot commented May 16, 2024 •

edited

lukebrawleysmith commented May 30, 2024 •

edited