CI Issues

ERROR: Uploading artifacts as "archive" to coordinator... 413 Request Entity Too Large  correlation_id=234f5fd4a4f64a159d472ed978acdd68 id=1309185 responseStatus=413 Request Entity Too Large status=413 token=eyJraWQiO
FATAL: too large 

In Gitlab there are 2 instance settings:

• Maximum artifacts size (MB)
• Maximum artifacts archive size (bytes): https://gitlab.com/gitlab-org/gitlab/-/work_items/594232: This setting controls the maximum size of artifact files when they're extracted and read for inclusion in CI/CD configurations (specifically for dynamic child pipelines). It does NOT control the size of the artifact archive itself during upload.

On 22 CPU machine, when Buildroot is in parallel download mode (BR2_PER_PACKAGE_DIRECTORIES=y):

kex_exchange_identification: Connection closed by remote host
Connection closed by 192.168.64.240 port 22

In Gitlab server logs:

gitlab sshd[1800758]: error: beginning MaxStartups throttling
gitlab sshd[1800758]: drop connection #10 from [192.168.65.128]:37112 on [192.168.64.240]:22 past MaxStartups

Solution: ssh_exchange_identification

Cloning into '/builds/group/project/buildroot/src/maginc-submodule'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab.example.com/groups/project/magic-submodule.git': The requested URL returned error: 403
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.example.com/groups/project/magic-submodule.git' into submodule path '/builds/group/project/buildroot/src/maginc-submodule' failed

Reason: Default CI/CD job token (CI_JOB_TOKEN) scope changed

More: Allow access to your project with a job token

Solution:

• Go to Submodule to be cloned in Gitlab UI
• Settings –> CI/CD –> Token Access

fatal: No names found, cannot describe anything.

When git describe –tags is used to detect version name but repository doesn't contain tags.

Possible reasons of failure:

• shallow clone, to unshallow run sudo git fetch –unshallow
• gitlab ???

Synchronizing submodule url for 'xxx'
Cloning into 'xxx'...
error: cannot run ssh: No such file or directory
fatal: unable to fork
fatal: clone of 'git@gitlab.xxx:xxx/xxx.git' into submodule path 'xxx' failed

Looks like SSH client is not installed in Job image (install openssh-clients).

Reason: During clone operation Gitlab Runner is using small helper (around 66MB) image where ssh client is not installed:

docker run --rm -it registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:alpine-latest-x86_64-000bc602 ssh

It is possible to chose flavour of helper image: helper-image

Images source: runner-helper

Ubuntu helper flavour contains ssh client:

docker run --rm -it gitlab/gitlab-runner-helper:ubuntu-x86_64-bleeding ssh
docker run --rm -it gitlab/gitlab-runner-helper:ubuntu-x86_64-v15.11.1 ssh

Solution: use Ubuntu based helper image:

config.toml

  [runners.docker]
    helper_image = "gitlab/gitlab-runner-helper:ubuntu-x86_64-v16.0.2"
    helper_image_flavor = "ubuntu"

Workaround 1: use relative submodules as described in Using relative URLs. Gitlab runner will use https to clone submodules.

Workaround 2: Rewrite submodule URLs to HTTPS

fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.example.com/superproject/sandbox.git/': server certificate verification failed. CAfile: none CRLfile: none

Problem test:

git clone https://gitlab.example.com
 
fatal: unable to access 'https://gitlab.example.com/': server certificate verification failed. CAfile: none CRLfile: none

Solution:

sudo apt update; sudo apt install -yq libgnutls30 ca-certificates

Dockerfile fix:

RUN apt-get update && apt-get install -yq --no-install-recommends \
    ca-certificates \
    libgnutls30 \
    && apt-get clean && rm -rf /var/lib/apt/lists/*