meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
git:gitlab:issues [2022/12/28 14:29] – created niziakgit:gitlab:issues [2024/10/01 10:24] (current) – niziak
Line 1: Line 1:
 ====== Gitlab issues ====== ====== Gitlab issues ======
 +
 +===== disallowed submodule url =====
 +
 +Repocheck failes:
 +
 +<file log /var/log/gitlab/gitlab-rails/repocheck.log>
 +ERROR -- : my_group/my_project: Could not fsck repository: error in blob eb87a57adefc985514ddc8ae1fa4e6749c7d8211: gitmodulesUrl: disallowed submodule url: https://gitlab.example.com:my_group/my_project.git
 +</file>
 +
 +  * [[https://stackoverflow.com/questions/55569644/how-to-fix-gitmodulespath-disallowed-submodule-path-on-git-fsck|How to fix 'gitmodulesPath: disallowed submodule path' on 'git fsck']]
 +  * [[https://gitlab.com/gitlab-org/gitlab/-/issues/462567|repocheck fails with "disallowed submodule url"]]
 +
 +<file ruby gitlab.rb>
 +    gitaly['configuration'] = {
 +         git: {
 +          config: [
 +            { key: "fetch.fsck.gitmodulesUrl", value: "ignore" },
 +            { key: "fsck.gitmodulesUrl", value: "ignore" },
 +
 +          ],
 +         },
 +    }
 +</file>
 +
 +<code bash>sudo gitlab-ctl reconfigure</code>
 +
 +===== One project failed its last repository check.  =====
 +
 +<code bash>cat /var/log/gitlab/gitlab-rails/repocheck.log</code>
 +
 +''error in blob 31ae455954e2ab24f917f0cad3c9dd9b0e029be4: gitmodulesMissing: unable to read .gitmodules blob''
 +
 +Solution:
 +<code bash>
 +su - git
 +cd /home/git-data/repositories/\@hashed\..\..\...git
 +git fsck
 +git gc
 +git fsck
 +</code>
 +
 +===== Failed to add control inotify watch descriptor for control group =====
 +<code bash>
 +gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add control inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
 +gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add memory inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
 +</code>
 +
 +===== Permission denied, please try again. =====
 +
 +rate_limit_gitlab_shell:
 +
 +Gitlab runner cannot clone repository:
 +
 +<code>
 +Permission denied, please try again.
 +Permission denied, please try again.
 +git@gitlab.example.com: Permission denied (publickey,password).
 +fatal: Could not read from remote repository.
 +Please make sure you have the correct access rights
 +and the repository exists.
 +</code>
 +
 +**Tracing**
 +Upon SSH connection ''gitlab-shell'' is executed. Shell makes internal API connection to ''http://unix/api/v4/internal/allowed'' endpoint. 
 +''Puma'' web server is responsible for handling request.
 +
 +[[https://docs.gitlab.com/ee/architecture/blueprints/pods/pods-feature-git-access.html#22-git-clone-over-ssh|2.2. Git clone over SSH]]
 +
 +**HINT:** [[https://docs.gitlab.com/ee/administration/operations/puma.html#gitlab-api-is-not-accessible|GitLab: API is not accessible]]
 +
 +**Reason:**
 +Rate limit introduced with v14.8 with (28 Jan, 2022) [[https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79419|Enable Gitlab Shell rate limit by default]].
 +
 +More info:
 +  * [[https://docs.gitlab.com/ee/security/rate_limits.html#git-operations-using-ssh|Git operations using SSH]]
 +  * [[https://docs.gitlab.com/ee/administration/feature_flags.html|Enable and disable GitLab features deployed behind feature flags]]
 +
 +How to disable rate limits - how to disable ''rate_limit_gitlab_shell'':
 +<code bash>gitlab-rails console</code>
 +and then
 +<code ruby>
 +
 +
 +Feature.enabled?(:rate_limit_gitlab_shell)
 +
 +Feature.disable(:rate_limit_gitlab_shell)
 +
 +Feature.enabled?(:rate_limit_gitlab_shell)
 +</code>
 +
 +Still problems, try to figure out:
 +  * Gitlab-shell rate limiter: [[https://docs.gitlab.com/ee/development/gitlab_shell/#rate-limiting|Rate limiting]]
 +  * There is another rate limiter in Gitaly: ''Gitaly also has a rate-limiter in place, but calls are never made to Gitaly if the rate limit is exceeded in GitLab Shell (Rails).''
 +    * It is disabled by default. Need to specify RPC endpoint and params to create Gitaly limiter: [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/middleware/limithandler/concurrency_limiter.go#L144|L144 of code]]
 +    * [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/backpressure.md|Request limiting in Gitaly]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#limit-rpc-concurrency|Limit RPC concurrency]]
 +      * ''When these limits are reached, users are disconnected.''
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-rate-limiting|Monitor Gitaly rate limiting]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-concurrency-limiting|Monitor Gitaly concurrency limiting]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/troubleshooting.html#500-and-fetching-folder-content-errors-on-repository-pages|Troubleshooting Gitaly and Gitaly Cluster]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/reference.html|Gitaly reference]]
 +    * [[https://gitlab.com/gitlab-org/gitaly/-/tree/master|Gitaly source]]
 +  * [[https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#allow-specific-users-to-bypass-authenticated-request-rate-limiting|Allow specific users to bypass authenticated request rate limiting]]