v2.0.0

[ruslandoga]

The highlights of this release are:

• more visitor metrics like total visits and views per visit
• period comparisons [docs] [example]
• ability to change domains names Allow domain change #2803
• automatic geolocation database downloads replace geolix with locus #2362
• more email adapters support (Mailgun, Mandrill, SendGrid) Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649

Upgrading Plausible Analytics to v2.0

⚠️ ⚠️ ⚠️

Upgrading to v2.0 requires performing a data migration.
Please read these notes until the end before deploying v2.0.0

Ensure you are using a new ClickHouse version

The steps below have been tested with clickhouse/clickhouse-server:22.6-alpine please make sure to upgrade ClickHouse to at least this version.

Here's the excerpt from v1.5 release discussion regarding the ClickHouse upgrade

---

In your docker-compose.yml update the image used for plausible_events_db to a newer ClickHouse version:

Upgrading ClickHouse to 22.6

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/community-edition#45

---

Update image tag

In your docker-compose.yml update the image used for plausible to v2.0.0

plausible:
- image: plausible/analytics:v1.5.1
+ image: plausible/analytics:v2.0.0

and restart the container

$ cd hosting
$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

This will boot up the new version of the app.

If you open the dashboards now, you wouldn't see any past metrics. This is expected as v2.0 uses the new events_v2 and sessions_v2 tables to store analytics data. We need to perform data migration to copy the data into the new tables.

Run data migration

⚠️ ⚠️ ⚠️

You need enough free disk space available for x2 of the current plausible_events_db's event-data volume size. You can use something like docker system df -v | grep hosting_event-data to check how much space the current volume is occupying.

Connect to the running plausible container and start the migration flow

$ cd hosting
$ docker compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

You can attempt this migration multiple times unless you drop v1 tables.

Drop v1 tables (optional)

Once you verify the migration went well, the old tables can be dropped. It's easiest to use clickhouse-client for this

$ ch hosting
$ docker compose exec plausible_events_db clickhouse-client

:) \c plausible_events_db
:) show tables;
:) drop table events;
:) drop table sessions;

See https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#max-table-size-to-drop for how to drop tables with more than 50GB of data.

Enable automatic MaxMind GeoLite2 updates (optional)

In your plausible-config.env set MAXMIND_LICENSE_KEY environment variable and get an automatically updated GeoLite2 City geolocation database. The database edition is configurable with MAXMIND_EDITION environment variable and defaults to GeoLite2-City.

- IP_GEOLOCATION_DB=...
+ MAXMIND_LICENSE_KEY=LNpsJCCKPis6XvBP
+ MAXMIND_EDITION=GeoLite2-City

Note that for the changes in plausible-config.env to propagate to the plausible, the container needs to be recreated:

$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

Also note that using GeoLite2-City edition requires more RAM than using GeoLite2-Country.

Now you can remove any other volumes and services used to download, store, and update geolocation databases.

Changelog

Following changes have been made since v1.5:

Added

• Add support for with_imported=true in Stats API aggregate endpoint
• Ability to use '--' instead of '=' sign in the tagged-events classnames
• 'Last updated X seconds ago' info to 'current visitors' tooltips
• Add support for more Bamboo adapters, i.e. Bamboo.MailgunAdapter, Bamboo.MandrillAdapter, Bamboo.SendGridAdapter Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649
• Ability to change domain for existing site (requires numeric IDs data migration, instructions will be provided separately) UI + API (PUT /api/v1/sites)
• Add LOG_FAILED_LOGIN_ATTEMPTS environment variable to enable failed login attempts logs add LOG_FAILED_LOGIN_ATTEMPTS #2936
• Add MAILER_NAME environment variable support add MAILER_NAME #2937
• Add MAILGUN_BASE_URI support for Bamboo.MailgunAdapter add MAILGUN_BASE_URI #2935
• Add a landing page for self-hosters add landing page #2989
• Allow optional IPv6 for clickhouse repo Allow optional IPv6 for clickhouse repo #2970

Fixed

• Fix tracker bug - call callback function even when event is ignored
• Make goal-filtered CSV export return only unique_conversions timeseries in the 'visitors.csv' file
• Stop treating page filter as an entry page filter
• City report showing N/A instead of city names with imported data Reject unknown imported cities from queries #2675
• Empty values for Screen Size, OS and Browser are uniformly replaced with "(not set)"
• Fix more pageviews with session prop filter than with no filters
• Cascade delete sent_renewal_notifications table when user is deleted Cascade delete sent_renewal_notifications table when user is deleted #2549
• Show appropriate top-stat metric labels on the realtime dashboard when filtering by a goal
• Fix breakdown API pagination when using event metrics Fix breakdown API pagination when using event metrics #2562
• Automatically update all visible dashboard reports in the realtime view
• Connect via TLS when using HTTPS scheme in ClickHouse URL update ecto to include db url scheme #2570
• Add error message in case a transfer to an invited (but not joined) user is requested improve transfer ownership error message #2651
• Fix bug with showing property breakdown with a prop filter
• Fix bug when combining goal and prop filters Fix unknown bind meta #2654
• Fix broken favicons when domain includes a slash
• Fix bug when using multiple wildcard goal filters
• Fix a bug where realtime would fail with imported data
• Fix a bug where the country name was not shown when filtering through the map

Changed

• Treat page filter as entry page filter for bounce_rate
• Reject events with long URIs and data URIs Reject events with long URIs and data URIs #2536
• Always show direct traffic in sources reports Always show direct traffic in sources reports #2531
• Stop recording XX and T1 country codes Ignore unknown countries #2556
• Device type is now determined from the User-Agent instead of window.innerWidth Use user-agent instead of screen_width to get device type #2711
• Add padding by default to embedded dashboards so that shadows are not cut off Embedded dashboard fixes #2744
• Update the User Agents database (https://github.com/matomo-org/device-detector/releases/tag/6.1.1)
• Disable registration in self-hosted setups by default disable registration by default in self-hosted setups #3014

Removed

• Remove Firewall plug and IP_BLOCKLIST environment variable
• Remove the ability to collapse the main graph Remove the ability to collapse the main graph + transition bug fix #2627
• Remove custom_dimension_filter feature flag remove custom_dimension_filter feature and views_per_visit_metric flags #2996

---

This discussion was created from the release v2.0.0.

[stijneikelboom]

After the seemingly successful migration to v2.0.0, I noticed the tmp_events_v2 and tmp_sessions_v2 tables in the plausible_events_db database.

Are these tables expected to exist after the migration? Or are they as temporary as their name suggests and can they be dropped along with the v1 tables?

My full list of tables is currently as follows:

┌─name───────────────────────┐
│ domains_lookup             │
│ events                     │
│ events_v2                  │
│ imported_browsers          │
│ imported_devices           │
│ imported_entry_pages       │
│ imported_exit_pages        │
│ imported_locations         │
│ imported_operating_systems │
│ imported_pages             │
│ imported_sources           │
│ imported_visitors          │
│ ingest_counters            │
│ schema_migrations          │
│ sessions                   │
│ sessions_v2                │
│ tmp_events_v2              │
│ tmp_sessions_v2            │
└────────────────────────────┘

[ruslandoga]

👋 @stijneikelboom

I think these tables are indeed temporary (since the partitions are moved, and these tables are truncated in the end of the script) and can be dropped.

analytics/lib/plausible/data_migration/numeric_ids.ex

Lines 143 to 150 in c550a87

	confirm_fn.("Run partition: #{part}?", fn ->
	{:ok, _} = run_sql("insert-into-tmp-events-v2", partition: part)
	{:ok, _} = run_sql("attach-tmp-events-v2", partition: part)
	{:ok, _} = run_sql("truncate-tmp-events-v2", [])
	{:ok, _} = run_sql("insert-into-tmp-sessions-v2", partition: part)
	{:ok, _} = run_sql("attach-tmp-sessions-v2", partition: part)
	{:ok, _} = run_sql("truncate-tmp-sessions-v2", [])
	end)

Seems like they were used to atomically copy the rows partition by partition into the new tables.

[aerosol]

Yeah, safe to drop them

[stijneikelboom]

Thanks both! Totally clear why these tables are there. I now see that they are empty indeed.

Dropping these tables might be a useful addition to the ‘Drop v1 tables’ section, just for tidiness.

[Lastofthefirst]

I'm having difficulties enabling city geolocation. When I try to add I get a 502 error.

Here are my docker-compose logs, can you point me in the right direction?

https://gist.github.com/Lastofthefirst/08ba4ba464eccb00826936bfa774a874

[Lastofthefirst]

(I was running some pre-v1.5.1 version and upgraded to v2.0.0. It was successful until I added city with maxmind.)

[ruslandoga]

👋 @Lastofthefirst

The error says /etc/app/geonames.csv is a directory, and it's expected to be a file. Check your mounts in the docker-compose.yml config.

[strefethen]

I attempted upgrading from v1.5.1 > v2.0.0 and ran into some significant issues due to what appears to have been some latent issues with the Clickhouse DB that I was unaware of. Since it took me several hours to get a backup working I thought I'd post what I did to get my v1.5.1 back up and running and avoid losing 6 mos. of data. For starters, I followed the v2.0.0 upgrade instructions but upon bringing Plausible backup noticed tons of errors streaming by in docker.

$ docker logs -f <plausible_containerID>

The first was (truncated):

"[error] [locus] [geolocation] database failed to load (:filesystem): {:unpack_database_from, :mmdb_blob"

...followed by

"Kernel pid terminated",application_controller,"{application_start_failure,plausible,{bad_return,{{'Elixir.Plausible.Application'"

I decided to restore my backup and ran into the Clickhouse issues restarting docker makes me think there were problems prior to the upgrade. Btw, I'd previously performed the steps to get city level data via Maxmind. I searched for multiple Clickhouse errors including "Suspiciously many (12 parts, 0.00 B in total) broken parts...". I made two changes that allowed me to get my backup working which is what I think is worth sharing.

This article (among others) was helpful.

1. Add the empty force_restore_data file via a volume reference in ./geoip/docker-compose.geoip.yaml under geoip.
2. Added the following XML to ./clickhouse/clickhouse-config.xml

    <merge_tree>
        <max_suspicious_broken_parts>12</max_suspicious_broken_parts>
    </merge_tree>

After these changes my v1.5.1 backup started working again. I haven't repeated the v2.0.0 upgrade but will give it another go.

[sebdanielsson]

UPDATE: I solved my error. See reply down below.

I'm trying to use the new Bamboo Mailgun adapter. I've added my API key and Mailgun domain but I don't receive mail anymore. Is there a way to check for error logs and also send test mails when troubleshooting? I can't see any obvious errors in my compose file and running docker logs doesn't show any mail related errors.

My compose file:

services:
  plausible_db:
    image: postgres:14-alpine
    networks:
      - plausible_network
    volumes:
      - db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
    secrets:
      - source: plausible_postgres_password
        target: /run/secrets/postgres_password
    ports:
      - "5432:5432"
    deploy:
      mode: replicated
      replicas: 1
    restart: always
  plausible_events_db:
    image: clickhouse/clickhouse-server:22.6-alpine
    networks:
      - plausible_network
    volumes:
      - event_db:/var/lib/clickhouse
      - event_db_logs:/var/log/clickhouse-server
      - event_db_config:/etc/clickhouse-server
    deploy:
      mode: replicated
      replicas: 1
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    restart: always
  app:
    image: plausible/analytics:v2.0.0
    command: sh -c "sleep 10 && /entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"
    networks:
      - plausible_network
    depends_on:
      - plausible_db
      - plausible_events_db
    ports:
      - "8888:8000"
    environment:
      - CONFIG_DIR=/run/secrets
      - DISABLE_REGISTRATION=false
      - BASE_URL=/run/secrets/plausible_base_url
      - SECRET_KEY_BASE=/run/secrets/plausible_secret_key_base
      - GOOGLE_CLIENT_ID=/run/secrets/plausible_google_client_id
      - GOOGLE_CLIENT_SECRET=/run/secrets/plausible_google_client_secret
      - MAILGUN_API_KEY=/run/secrets/plausible_mailgun_api_key
      - MAILGUN_DOMAIN=/run/secrets/plausible_mailgun_domain
    secrets:
      - source: plausible_base_url
        target: /run/secrets/BASE_URL
      - source: plausible_secret_key_base
        target: /run/secrets/SECRET_KEY_BASE
      - source: plausible_google_client_id
        target: /run/secrets/GOOGLE_CLIENT_ID
      - source: plausible_google_client_secret
        target: /run/secrets/GOOGLE_CLIENT_SECRET
      - source: plausible_mailgun_api_key
        target: /run/secrets/MAILGUN_API_KEY
      - source: plausible_mailgun_domain
        target: /run/secrets/MAILGUN_DOMAIN
    deploy:
      mode: replicated
      replicas: 1
    restart: always

networks:
  plausible_network:
    external: true

volumes:
  db:
    name: plausible_db
  event_db:
    name: plausible_event_db
  event_db_logs:
    name: plausible_event_db_logs
  event_db_config:
    name: plausible_event_db_config

secrets:
  plausible_postgres_password:
    external: true
  plausible_base_url:
    external: true
  plausible_secret_key_base:
    external: true
  plausible_google_client_id:
    external: true
  plausible_google_client_secret:
    external: true
  plausible_mailgun_api_key:
    external: true
  plausible_mailgun_domain:
    external: true

[ruslandoga]

👋 @sebdanielsson

To see the logs you can try docker compose logs

$ cd hosting
$ docker compose logs plausible

To send an email manually, you can try doing it from within the app

$ cd hosting
$ docker compose exec plausible bin/plausible remote

iex> # let's first check the config, the adapter should have Mailgun in the name and there should be api keys, etc.
iex> Application.get_env(:plausible, Plausible.Mailer)
iex> # note you'd need to replace from and to with actual addresses
iex> email = Bamboo.Email.new_email(from: "...", to: "...", subject: "test", text_body: "test")
iex> Plausible.Mailer.deliver_now(email)

[sebdanielsson]

Hi!

Thanks for the quick reply :)

Looks like it's configured to use the SMTP adapter:

[
  adapter: Bamboo.SMTPAdapter,
  server: "mail",
  hostname: "plausible.hogwarts.zone",
  port: "25",
  username: nil,
  password: nil,
  tls: :if_available,
  allowed_tls_versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"],
  ssl: false,
  retries: 2,
  no_mx_lookups: true
]

Trying to send an email results in:

{:error, {:network_failure, 'mail', {:error, :nxdomain}}}

Do I need to specify the adapter with an environment variable?

[sebdanielsson]

Can't believe I missed this:
https://github.com/plausible/docs/pull/351/files#diff-3e91011cafdf1beedd4db4e0a4b51164464ae19d0ed206b19091aafb86b0ad09R67

Will try this now!

[sebdanielsson]

Okay so basically I forgot to add two environment variables.

MAILER_ADAPTER=Bamboo.MailgunAdapter # Use the new adapter
MAILGUN_BASE_URI=https://api.eu.mailgun.net/v3 # Use EU servers. Only needed for EU Mailgun users.

[Tristan971]

The migration will take a bit of time here and I confirmed that the first few months are ok. But we have nearly 3 years to migrate, and I see that the rpc supports a non-interactive mode here

analytics/lib/plausible/data_migration/numeric_ids.ex

Line 25 in ae0019a

interactive? = Keyword.get(opts, :interactive?, true)

How can I enable it? Tried --interactive=false and interactive=false without success

[Tristan971]

Well, for anyone wondering, after a bit of looking around, the answer is:

bin/plausible rpc 'Plausible.DataMigration.NumericIDs.run([interactive?: false])'

TIL about elixir remote in-process execution, cool stuff

NB: To avoid issues like #3216 surprising you, you definitely want to do a few months manually first and to check clickhouse logs and table counts though. But once you've confirmed that most things work, you can then make your drives suffer with this :)

[Tristan971]

Couple of extra caveats I ran into to be mindful of:

• the migration is very CPU intensive (in our case, Clickhouse consuming about 20x more CPU than usual throughout it all, and that's while being IO-bound in our case; faster disks would have allowed it run even higher probably)
• you need to carefully watch for Clickhouse warnings about number of parts, ie things like Delaying inserting block by 566 ms. because there are 234 parts and their average size is 34.72 MiB mean you're going too fast and should ideally be pausing for a bit between partitions until the warning goes away
• if you don't, the warning seems to trigger at 200 parts to merge, and at 300 parts to merge Clickhouse just stops processing inserts until it's happy again (with default 23.5 configuration)

But frankly, props to the Plausible team. Other than these few bits to have in mind, it was very smooth and the replayability really helpful, especially given the amount of data involved.

[tvd0x2a]

I upgraded from v1.5.1 to v2.0.0 on Fly.io. I ran into a couple issues and hopefully these notes will make it easier for others that do this upgrade on Fly:

I needed to set ECTO_CH_IPV6=true as an env var, otherwise Clickhouse connections failed with nxdomain. Added to the docs here: plausible/docs#430 . Might be helpful to include a note in the upgrade notes in the v2.0.0 release description, if possible.

The data migration failed for me with pool error, which were really nxdomain errors. This was because v2.0.0 doesn't pass in transport options to the migration repo, see issue #3173. Fortunately already fixed in #3179, but that is not yet released as a docker image afaict. womp womp.

I built my own docker image off the commit with the fix, and then temporarily deployed that to run the migration. Then went back to the v2.0.0 release once that was done. The docker image I built and temporarily deployed to run the migration is: tvd0x2a/plausible-analytics:1c17e89b6.

The steps I used to build/publish the container, in case you rightfully don't trust rando docker images:

git clone https://github.com/plausible/analytics
cd analytics
git checkout 1c17e89b6
docker build --platform linux/amd64 -t tvd0x2a/plausible-analytics:1c17e89b6 . # probably won't work on macos arm chips, because longstanding qemu wontfix segfault bug
docker push tvd0x2a/plausible-analytics:1c17e89b6

Hopefully we get a point release with the transport options fix for the data migration tool soon. 🤞

[lightson]

Seamless upgrade as ever, though I needed to 'docker-compose' rather than 'docker compose'.

Some great new features, thanks team P!

[Daniel15]

You should upgrade your Docker Compose version. docker-compose is the legacy one which is no longer supported and stopped receiving updates in July.

[ruslandoga]

v2.1.0 release candidate is now available!

[ruslandoga]

v2.1.0-rc.1 is now available! Compared to v2.1.0-rc.0 above it includes a new SMTP client, GA4 imports, and CSV exports and imports.