avatar
tkat0.dev
JA
Published on

argocd-image-updater didn't work with Local Docker Registry

日本語で読む
Table of Contents

Summary

There is a issue that argocd-image-updater cannot update images in Application that has multiple images of Local Docker Registry (e.g. 192.168.0.100:32000). This article is a brief note on the issue and its solution.

In conclusion, don't use registry names containing colon :.

If you want to know more details, please let me know in the comments. Note that I am still learning kubernetes, so there may be some errors.

Background

  • I run my hobby kubernetes environment on a local machine.
  • I run Docker Registry locally, not ECR or GCR.
  • I use tekton for CI and argocd for CD.
  • I wanted to use argocd-image-updater to automate updating images in manifests used by argocd.

Environment

  • kubernetes
    • created by microk8s
    • work on local machine
  • Container Registry
    • Use local registry created by microk8s (microk8s enable registry)
    • work on NodePort of kubernetes like 192.168.0.100:32000 (default configurations)
  • argocd
    • v2.5.5+fc3eaec
  • argocd-image-updater
    • v0.12.1

Issue

There is an issue with an Application that uses Kustomize with more than one Local Registry image, and only one image is updated.

This manifest is a configuration to automatically update the following two images to the latest version.

  • 192.168.0.100:32000/api-server
  • 192.168.0.100:32000/app

When CI pushes an image, argocd-image-updater is expected to automatically deploy the latest image. The argocd-image-updater logs and Events show two images are updated as expected, but the result is that only 192.168.0.100:32000/app is updated.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: 192.168.0.100:32000/api-server,192.168.0.100:32000/app

    argocd-image-updater.argoproj.io/force-update: 'true'
    argocd-image-updater.argoproj.io/update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/write-back-target: kustomization
spec:
  project: myapp
  source:
    repoURL: git@github.com:tkat0/xxx.git
    targetRevision: HEAD
    path: kubernetes/myapp/development
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  destination:
    server: https://kubernetes.default.svc
    namespace: default

Solution

You can't use registry names containing colon :. Therefore, the following solutions are available.

  • Option1: Publish Local Registry with a URI that doesn't include a port, such as 192.168.0.101.
    • Used this option in this time.
  • Option2: Use GCR, ECR or other registries.

For Option 1, it would be sufficient to simply set the service type of Local Registry to LoadBalancer so that it can be accessed directly with an address such as 192.168.0.101, but I used Ingress to configure the Local Docker Registry as follows.

  • (required) Ingress exposes Service at /.
  • (required) nginx.ingress.kubernetes.io/proxy-body-size: "0"
  • (optional) Set my domain as host (cr.cluster.tkat0.dev)
    • For convenience (note that this host is configured to resolve local cluster in LAN, not public)
    • A wild card DNS such as https://sslip.io/ should work fine
  • (optional) Quit using microk8s registry and deploy another registry with helm
    • To manage manifests with Git as much as possible

Here is the manifest of Local Registry as above. I installed twuni/docker-registry via argocd and helm, plus added Ingress. (Depending on the configuration of the helm chart, it may be possible to complete all of this only in Application resource.)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cr
  namespace: ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    # NOTE: 413 Request Entity Too Large:
    # https://github.com/kubernetes/ingress-nginx/issues/4825#issuecomment-721911893
    # https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#custom-max-body-size
    # 0 means no upper bound
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
spec:
  ingressClassName: nginx
  rules:
    - host: cr.cluster.tkat0.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: docker-registry
                port:
                  number: 5000
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: docker-registry
  namespace: argocd
spec:
  project: default
  source:
    chart: docker-registry
    # https://github.com/twuni/docker-registry.helm
    repoURL: https://helm.twun.io
    targetRevision: 2.2.2
    helm:
      releaseName: docker-registry
      parameters:
        - name: service.type
          value: ClusterIP
        - name: garbageCollect.enabled
          value: 'true'
        - name: persistence.size
          value: 20Gi
        - name: ingress.enabled
          value: 'false'
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  destination:
    server: https://kubernetes.default.svc
    namespace: docker-registry

Cause

The cause is due to the implementation of argocd.

When comparing Kustomize image names, argocd checks the image name up to first : as an identifier, so it incorrectly recognizes 192.168.0.100:32000/api-server and 192.168.0.100:32000/app as the same image (192.168.0.100).

Specifically, argocd-image-updater updates Application in func SetKustomizeImage. A image should be added to Application each time this method is called.

However, func (*ApplicationSourceKustomize) MergeImage and func (KustomizeImage) Match overwrote the same image rather than added.

This is because, as mentioned above, if the image name is the same up to :, it is recognized as the same image.

In general, images such as gcr.io/project/image-1:v1.0 and gcr.io/project/image-2:v1.0 are recognized as different images, but this issue occurs in Local Registry that contains port number.

I understand that including a colon in registry URI is not common, especially in a production environment, but I could not find any documentation that says it should not be included as a specification (if anyone knows, please let me know).

At least, this documentation says including :8080 is fine.

https://docs.docker.com/engine/reference/commandline/tag/

The hostname must comply with standard DNS rules, but may not contain underscores. If a hostname is present, it may optionally be followed by a port number If a hostname is present, it may optionally be followed by a port number :8080.

Conclusion

This article describes the issue I encountered with argocd-image-updater not working with Local Registry.

I will organize the information and report to the community soon.

update(Jan 7): reported here https://github.com/argoproj-labs/argocd-image-updater/issues/516

Thank you for reading.