Dimensions

Dimensions let you drill down into experiment results. For example, if you define a browser dimension, you can see how Safari users behaved vs Chrome.

Be careful, the more dimensions and metrics you look at for an experiment, the more likely you are to see false positives - something that looks significant when it really isn't. For example, if you break out results by country, it's pretty likely that at least one of the 100+ countries in your dataset will be signficantly different just by random chance.

It's best to treat dimensions as an exploratory tool and not something to directly draw conclusions from. The two best use cases are identfying bugs (the browser example) and getting ideas for dedicated follow-up experiments.

There are two types of dimensions: User Dimensions and Experiment Dimensions. Currently, these are only supported for SQL data sources.

User Dimensions

These are attributes of your users that are relatively stable over time. For example, subscription plan, age group, cohort.

A user dimension is defined by a simple SQL query that returns two columns: user_id and value. For example:

SELECT
  user_id,
  plan_type as value
FROM
  subscriptions

It's best to keep the number of unique values for a dimension small if possible to avoid the False Positive issues discussed above. For example, instead of a country dimension with 200 possible values, maybe you could make the values US, UK, and Other. Depending on your SQL engine, you can use something like a CASE WHEN statement:

SELECT
  user_id,
  (
    CASE WHEN country = 'us' THEN 'US'
    WHEN country = 'uk' THEN 'UK'
    ELSE 'Other' END
  ) as value
FROM
  users

Experiment Dimensions

These are attributes that are specific to the point-in-time that a user was put into an experiment. For example, browser or referrer.

Instead of a standalone SQL query, experiment dimensions are simply extra columns you return from the Experiment query defined for your data source.

As an example, if you set the following as your experiment query:

SELECT
  user_id,
  anonymous_id,
  received_at as timestamp,
  experiment_id,
  variation_id,
  browser
FROM
  experiment_viewed

The first 5 columns are standard, but browser is a custom one that can be used as an Experiment Dimension.