Migrating from collectd to OpenTelemetry

This guide describes the migration of a GitHub Enterprise Server instance from collectd to the OpenTelemetry metrics stack.

In this article

GitHub Enterprise Server is transitioning from the collectd metrics stack to OpenTelemetry. Collectd is retired in version 3.23, and upgrading to 3.23 disables it.

Migration timeline

VersionDefault for new installationsDefault for instances upgraded from earlier releasesNotes
3.18CollectdCollectdOpenTelemetry is opt-in.
3.19 to 3.22OpenTelemetryCollectdMigration to OpenTelemetry is possible before upgrading to 3.23.
3.23 and laterOpenTelemetryOpenTelemetryCollectd is retired and cannot be used. The upgrade disables it.

Preparing to migrate

Before you migrate, complete the following preparations:

  • Test the migration on a staging instance before applying it in production.
  • Identify the monitoring integrations that depend on collectd metrics, such as external dashboards and alerts.
  • If you plan to run both metrics stacks during a transition period, review the caveats in Running both metrics stacks.

Migrating to OpenTelemetry

You can migrate to OpenTelemetry on instances running version 3.19 to 3.22. On version 3.18, OpenTelemetry is opt-in, so you can also enable it there. We recommend migrating before you upgrade to 3.23, so that you can verify your monitoring setup while you can still use collectd.

If you don't migrate, upgrading to version 3.23 or later does it for you: collectd is disabled and cannot be re-enabled, and OpenTelemetry metrics are enabled and become the only metrics stack.

Migration requires enabling OpenTelemetry metrics and disabling collectd. Both settings are available through the Management Console or the command line.

Using the Management Console

  1. From an administrative account on GitHub Enterprise Server, in the upper-right corner of any page, click .

  2. If you're not already on the "Site admin" page, in the upper-left corner, click Site admin.

  3. In the " Site admin" sidebar, click Management Console.

  4. In the monitoring settings, select Enable OpenTelemetry.

  5. Under Metrics - Collectd, select Disable collectd metrics.

  6. Under the "Settings" sidebar, click Save settings.

    Note

    Saving settings in the Management Console restarts system services, which could result in user-visible downtime.

  7. Wait for the configuration run to complete.

Using the command line

  1. SSH into your GitHub Enterprise Server instance. If your instance comprises multiple nodes, for example if high availability or geo-replication are configured, SSH into the primary node. If you use a cluster, you can SSH into any node. Replace HOSTNAME with the hostname for your instance, or the hostname or IP address of a node. For more information, see Accessing the administrative shell (SSH).

    Shell
    ssh -p 122 admin@HOSTNAME

  2. To enable OpenTelemetry metrics and disable collectd, run the following commands.

    ghe-config observability.metrics.next-enabled true
ghe-config observability.metrics.legacy-disabled true

  3. To apply the configuration, run the following command.

    Note

    During a configuration run, services on your GitHub Enterprise Server instance may restart, which can cause brief downtime for users.

    Shell
    ghe-config-apply

  4. Wait for the configuration run to complete.

Running both metrics stacks

On version 3.18 to 3.22, running the collectd and OpenTelemetry stacks at the same time is possible but not recommended. This configuration has the following caveats:

  • Some metric names may differ to remain compatible with the collectd format.
  • Some OpenTelemetry metrics do not include labels.
  • Increased resource utilization to deliver metrics to both solutions.

Collectd is retired in version 3.23, so running both stacks is not possible after upgrading.

Post-upgrade checks

After the configuration run completes, whether you migrated manually or upgraded to version 3.23 or later:

Viewing historical collectd data

Since patches 3.22.3, 3.21.8, 3.20.10, and 3.19.14, the Monitoring tab includes collectd dashboards for viewing historical data. These dashboards display collectd metrics recorded up to the time the collectd stack was disabled.