Most interactions between tests are really minor and indistinguishable from random noise. Those ones aren't worth worrying about.
There are two ways to deal with the bigger interaction effects. First is to predict which experiments will meaningfully interact ahead of time and split the samples between them (making the tests take longer). Second is to run tests in parallel on all users and look at the data to determine interaction effects after the fact (and potentially need to invalidate some results).
In our experience, a mix of these approaches works best. It's really hard to predict meaningful interaction effects ahead of time, so save that for the really obvious cases (e.g. black text on a black background). For everything else, the benefit of running more experiments usually outweighs the cost of occasionally needing to throw out results because of interaction effects. It's much better to run 10 tests and need to throw out 1 than it is to run 5 tests.
We are planning on an open core model similar to PostHog. So we would eventually monetize from cloud hosting and license keys for enterprise features. We wanted to use MIT to make it as easy to adopt as possible for all companies.
Thanks for the suggestion. We added MySQL support so we can work with Matomo now, but we're definitely lacking in documentation and tutorials. There's a similar issue with people using GA4, Snowplow, Segment, and anything else with a pre-defined schema. It should be way easier to use these systems with GrowthBook. Like there's no reason to even write SQL most of the time if we know the Matomo table structure in MySQL. We're hoping to solve that issue in the next few weeks.
Yeah, Statsig is building a really cool product. There's definitely a gap in GrowthBook right now, like you mentioned, between pure feature flags and experiments. We're working on some light-weight event tracking features to try and bridge that gap. So you'll be able to see how many people are using the features, which values they are getting, and easily integrate with tools like DataDog and NewRelic for performance monitoring.
Tools like Flagr typically function as a microservice that handles all of the feature evaluation. That means you need a network request every time you want to get a feature's value. With GrowthBook, there's a single cacheable JSON definition of all of the features and then evaluation happens locally within the SDKs. We found this approach to be much better for performance and scalability.
The other missing part in most feature flagging tools is the analysis side. Most tools let you run an A/B test, but it's up to you to track the data, process it, and run it through a stats engine. We built that part directly into GrowthBook.
Yeah, that's our exact experience as well. I was super surprised to find out how much of a graveyard open source A/B testing was. Intuit (Wasabi), Facebook (PlanOut), and SeatGuru (Sixpack) open sourced their internal platforms years ago, but they are all abandoned now.
Thanks! If you can recreate the installation steps in the Dockerfile (https://github.com/growthbook/growthbook/blob/main/Dockerfil...), it should be possible to run it yourself on a shared host. Basically, install NodeJs + Python + dependencies and run the build scripts.
A lab tutorial for uberspace sounds awesome! We're happy to help you get things set up if you run into any issues.
Ax is for automated optimization using machine learning. You define the parameters and optimization function and it decides the variations, traffic splitting, and everything else for you.
Growth Book is for hypothesis testing. It let's you define and run a specific controlled experiment and then you can analyze the results and make a decision.
Do you mind explaining that a little more? As it's currently designed, a company could use DBT to model their raw data into dedicated metric tables and then Growth Book sits on top of those with a really simple SQL query and some settings (e.g. is the goal to increase or decrease the metric)
I'd love to see an open source standard way to define metrics, but haven't found anything yet.
Thanks for the comment and for your work on Alephbet! Open source A/B testing is a graveyard of abandoned projects, so it's always great to see more people actively working in this space.
Georgi at Analytics Toolkit definitely knows his stuff. We're taking a Bayesian approach instead, which I know he isn't the biggest fan of, but I think it is much easier to understand. Itamar Faran, the author of our stats engine, has a great article that goes into a lot more detail if you're interested: https://towardsdatascience.com/why-you-should-switch-to-baye...
Yes. As long as the variation assignment data and success metrics are in a supported data source (SQL, GA, or Mixpanel currently), it can be queried and analyzed in Growth Book.
We plan to add MySQL/MariaDB support soon which should let you use Matomo data as long as you have raw SQL access. For cloud-hosted Matomo, we would have to use the reporting API, which is doable but not as good since there's no way to get standard deviations out of it as far as I can tell.
We don't have native mobile SDKs yet, but it's something we want to support in the future. Mobile is a little tricky since you either need to do a new release every time you want to start/stop a test or use remote config and deal with offline, slow networks, etc.
1. GA is very limited as a data source because of sampling and the fact that they don't expose variance. So if using GA, we only support simple binomial metrics, count data (assuming Poisson distribution), and duration data (assuming exponential distribution). For SQL data sources and non-parametric data, we currently rely on the CLT and treat the sampling distribution as Normal. There's a good article that goes over the stats in more detail (Itamar, the author, wrote our stats engine) - https://towardsdatascience.com/how-to-do-bayesian-a-b-testin...
2. We have a minimum sample size threshold before we run any statistics on the data. To your point, we don't want to say something is "significant" if it's 5 conversions vs 1. This is one area we're looking to improve with better heuristics. We can't completely take the human out of the loop, but we can help give them all the info they need to make the best decision. On that front, we do show Bayesian expected loss (risk) and credible intervals in addition to just the "chance to beat control".
1. We don't store any raw user data. We pull things like mean and standard deviation from data sources, run the statistics, and store the result.
2. We use a Bayesian statistics engine which is much more immune to peeking problems and Type I errors than frequentist approaches.
3. Tests can be run either client or server side. For client side, we recommend bundling the SDK with your app (webpack, etc). We really care about performance so never want to add additional http requests or script tags of any kind if at all possible.
There are two ways to deal with the bigger interaction effects. First is to predict which experiments will meaningfully interact ahead of time and split the samples between them (making the tests take longer). Second is to run tests in parallel on all users and look at the data to determine interaction effects after the fact (and potentially need to invalidate some results).
In our experience, a mix of these approaches works best. It's really hard to predict meaningful interaction effects ahead of time, so save that for the really obvious cases (e.g. black text on a black background). For everything else, the benefit of running more experiments usually outweighs the cost of occasionally needing to throw out results because of interaction effects. It's much better to run 10 tests and need to throw out 1 than it is to run 5 tests.