<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=585913) </details> <!--IssueSummary end--> ### Summary Currently, Geo's SSH proxying is incompatible with SSH certificate authentication. It has been so since the time when both SSH proxying and SSH certificate authentication existed, so technically it was an unimplemented feature addition, and now it's a bug. - https://docs.gitlab.com/administration/operations/ssh_certificates/ - https://docs.gitlab.com/user/group/ssh_certificates/ #### More details according to Duo When using SSH certificates for authentication (via `AuthorizedPrincipalsCommand`), Geo SSH proxy operations fail with "Geo push user is invalid" error. This occurs because `Geo::PushUser` only supports the `key-{ID}` format for `gl_id`, not the `username-{KEY_ID}` format used by SSH certificates. <details> <summary>Click to expand the flow in the codebase that leads to the error</summary> 1. Secondary's `GeoGitAccess#geo_custom_ssh_action` creates payload with `gl_id` in `username-{KEY_ID}` format (from certificate) 2. Primary's `GitHttpController#ssh_upload_pack` handles the request 3. `access_actor` method calls `Geo::PushUser.new(geo_gl_id)` 4. `Geo::PushUser#user` calls `identify_using_ssh_key(gl_id)` 5. `Gitlab::Identifier#identify_using_ssh_key` expects `key-{ID}` format and queries `User.find_by_ssh_key_id(key_id)` 6. This fails for SSH certificate users because there's no record in the `keys` table 7. Returns `nil` and `access_actor` raises `'Geo push user is invalid.'` </details> <details> <summary>Click to expand details on `gl_id` formats</summary> #### The Different `gl_id` Formats and Their Origins The `gl_id` format depends on **how the user authenticates via SSH**: **1. Traditional SSH Keys (`key-{ID}`):** When a user authenticates with an SSH key stored in GitLab's database, the `AuthorizedKeysCommand` returns: ```shell command="gitlab-shell upload-pack key_id=1" ``` GitLab Shell then uses `key-{ID}` as the `gl_id`, and `Gitlab::Identifier#identify_using_ssh_key` looks up the user via `User.find_by_ssh_key_id(key_id)`. **2. SSH Certificates (`username-{USERNAME}`):** When using SSH certificate authentication via `AuthorizedPrincipalsCommand`, the command outputs: ```shell command="/opt/gitlab/.../gitlab-shell username-{KEY_ID}",... {PRINCIPAL} ``` Here, `{KEY_ID}` is the **username** extracted from the certificate (not a database ID). There's no database lookup needed because the username comes directly from the cryptographically-signed certificate. **3. HTTP Push (`user-{ID}`):** For HTTP Git operations, the user is already authenticated via the web session/API, so `gl_id` uses `user-{ID}` with the numeric user ID. #### Why `Gitlab::Identifier#identify` Only Handles `user-*` and `key-*` The `identify` method was designed for the **internal API flow** where: - `user-{ID}` comes from HTTP authentication (user already identified) - `key-{ID}` comes from traditional SSH key lookup (key exists in database) The `username-{USERNAME}` format was added later for SSH certificates and **bypasses the need for database lookup** during initial authentication. The username is trusted because it's embedded in a certificate signed by a trusted CA. #### The Bug `Geo::PushUser` directly calls `identify_using_ssh_key(gl_id)`, which assumes `key-{ID}` format. When Geo proxies an SSH request from a certificate-authenticated user, the `gl_id` is `username-{USERNAME}`, which fails because there's no SSH key record to look up. </details> ### Steps to reproduce 1. Configure GitLab with SSH certificate authentication using `AuthorizedPrincipalsCommand` 2. Disable traditional SSH key uploads (common security policy) 3. Set up Geo with a secondary site configured as read-only (`GEO_SECONDARY_PROXY=0`) 4. Attempt to clone/pull a repository via SSH from the secondary when the repository is out-of-date ### What is the current bug behavior? When the secondary proxies an SSH request to the primary for an out-of-date repository: Error message: ``` remote: Geo push user is invalid. fatal: Could not read from remote repository. ``` ### What is the expected correct behavior? `Geo::PushUser` should support the `username-{KEY_ID}` format used by SSH certificate authentication, allowing users to be identified by username lookup. ### Potential solutions <details> <summary>Click to expand an idea from Duo</summary> This is Duo's idea which I have not vetted as of 13 Jan 2026. I added the security items. 1. Update `Geo::PushUser#user` to use the generic `identify()` method 2. Extend `Gitlab::Identifier#identify` to handle the `username-{KEY_ID}` pattern 3. Add an `identify_using_username` method to look up users by username 4. Consider security implications 5. Get app sec review #### Relevant code `ee/app/models/geo/push_user.rb`: ```ruby def user @user ||= identify_using_ssh_key(gl_id) end ``` `lib/gitlab/identifier.rb` already has a generic `identify()` method that handles multiple formats, but it does not include `username-*`: ```ruby def identify(identifier) case identifier when /\Auser-\d+\Z/ identify_using_user(identifier) when /\Akey-\d+\Z/ identify_using_ssh_key(identifier) end end ``` </details> ### Possible Workarounds 1. If you do not use CNH/CNG, you might try enabling `Feature.enable(:geo_proxy_fetch_ssh_to_primary) && Feature.enable(:geo_proxy_push_ssh_to_primary)` since I believe the unreleased change in flow does not run the offending code. This possible workaround has not been verified empirically, as of 13 Jan 2026. References https://gitlab.com/gitlab-org/gitlab/-/work_items/466045+, https://gitlab.com/gitlab-org/gitlab/-/work_items/466057+. (This is not working for CNH/CNG yet because you cannot disable proxy request buffering by path in the NGINX ingress. You could enable the FFs if you disable proxy request buffering completely. This is low risk if your env is not on the public internet.) ### Related issues Customer request for help: https://gitlab.com/gitlab-com/request-for-help/-/issues/4040