data/release-notes/enterprise-server/3-16/1.yml

date: '2025-03-25'
intro: |
  {% warning %}

  **Warning**: We received a few reports of performance issues with GitHub Enterprise Server versions 3.15, 3.16, and 3.17 and have shipped performance fixes to the affected versions. You can now upgrade to 3.15.12, 3.16.8, 3.17.5, or later. We do not recommend upgrading to earlier releases of 3.15, 3.16, or 3.17. [Updated: 2025-08-25]

  {% endwarning %}
sections:
  security_fixes:
    - |
      Packages have been updated to the latest security versions.
  bugs:
    - |
      In Azure environments, running `ghe-single-config-apply` or `ghe-repl-setup` resulted in "Permission denied" errors during the pre-flight check.
    - |
      The `ghe-upgrade` command returned a zero exit code despite encountering errors.
    - |
      When performing an upgrade with an upgrade package, the process did not terminate when an invalid target partition was provided with the `-t` flag.
    - |
      Restoring from a backup did not always apply the latest data from GitHub Actions. All GitHub Actions data is now restored with a backup.
    - |
      For instances in a high availability configuration, Elasticsearch indices were deleted on failover and when `ghe-repl-teardown REPLICA_HOSTNAME` was run from the primary instance. All indices are recoverable except audit log indices, whose source of truth is Elasticsearch itself.
    - |
      On an instance with security product defaults set on organizations, users were unable to upgrade from versions earlier than 3.15 due to database migration failures.
    - |
      Domain entries could fail to load in the "Verified & Approves Domains" section of the site admin dashboard if one or more authoritative nameservers for the affected domain was unreachable or unresponsive due to inefficient DNS queries.
    - |
      On instances with a GitHub Advanced Security license, some secret scanning alerts were opened incorrectly despite the relevant folders or files being excluded from secret scanning.
    - |
      For appliances in a high availability configuration, Elasticsearch indices were deleted either on failover, or when running `ghe-repl-teardown <REPLICA_HOSTNAME>` from the primary instance.
  changes:
    - |
      Elasticsearch shards are excluded from the replica node when stopping replication via `ghe-repl-stop`. To prevent Elasticsearch from being stopped before all shards have been removed, Elasticsearch is polled until the shard count on the replica node is zero instead of waiting for a maximum timeout of 30 seconds.
    - |
      Update the bundled `actions/setup-dotnet` with the latest versions from https://github.com/actions/setup-dotnet.
  known_issues:
    - |
      Customers operating at high scale or near capacity may experience unexpected performance degradation, such as slow response times, background job queue spikes, elevated CPU usage, and increased MySQL load. Consider upgrading to 3.16 with caution. [Updated: 2025-07-28]
    - |
      Upgrades to 3.16.1 may fail due to an error while migrating a secret scanning database. The `ghe-migrations` command output will include a `DelegatedBypassConfigurationNotNil` error. To recover, you can restore from a backup, fail over to a replica, or reach out to GitHub Support if you encounter this issue. [Updated: 2025-04-03]
    - |
      Custom firewall rules are removed during the upgrade process.
    - |
      During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.
    - |
      If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see [AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account).
    - |
      On an instance with the HTTP `X-Forwarded-For` header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.
    - |
      {% data reusables.release-notes.large-adoc-files-issue %}
    - |
      Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.
    - |
      When following the instructions for [Replacing the primary database node](/admin/monitoring-and-managing-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-the-primary-database-node-mysql-or-mysql-and-mssql), `ghe-cluster-config-apply` might fail with errors. If this occurs, re-running `ghe-cluster-config-apply` is expected to succeed.
    - |
      Running a config apply as part of the steps for [Replacing a node in an emergency](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency) may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.
    - |
      {% data reusables.release-notes.2024-06-possible-frontend-5-minute-outage-during-hotpatch-upgrade %}
    - |
      When restoring data originally backed up from a 3.13 or greater appliance version, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance after at least one hour system uptime. [Updated: 2025-04-22]
    - |
      An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.
    - |
      When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.
    - |
      When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
    - |
      When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration.
    - |
      Admins setting up cluster high availability (HA) may encounter a spokes error when running `ghe-cluster-repl-status` if a new organization and repositories are created before using the `ghe-cluster-repl-bootstrap` command. To avoid this issue, complete the cluster HA setup with `ghe-cluster-repl-bootstrap` before creating new organizations and repositories.
    - |
      Some customers upgrading from 3.11.x or 3.12.x may experience a bug with the feature "Automatic update checks", filling the root disk with logs causing a system degradation. To prevent this, you can turn off the feature "[Enable automatic update check](/admin/upgrading-your-instance/preparing-to-upgrade/enabling-automatic-update-checks#enabling-automatic-update-checks)" in the management console.
    - |
      In a cluster, the host running restore requires access the storage nodes via their private IPs.
    - |
      On an instance hosted on Azure, commenting on an issue via email meant the comment was not added to the issue.
    - |
      After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance after at least one hour system uptime. [Updated: 2025-04-22]
    - |
      After a geo-replica is promoted to be a primary by running `ghe-repl-promote`, the actions workflow of a repository does not have any suggested workflows.
    - |
      The security risk page returns a 500 error when secret scanning is disabled. [Updated: 2025-04-25]
    - |
      Upgrading to this version from GHES 3.14.19 and higher or 3.15.14 and higher will cause the upgrade to fail due to this version containing an older version of MySQL. To avoid this issue please upgrade to GHES 3.16.10 or higher.

      [Updated: 2025-11-24]
  errata:
    - |
      Known issues incorrectly indicated that administrators could immediately run the `/usr/local/share/enterprise/ghe-es-search-repair` script after a restore to resolve indexing and collaborator access issues. In reality, this script must be run only after the system has been up for at least one hour. [Updated: 2025-04-22]