To override each image in all the applications you can use following parameters:

• --kubectl-image someRegistry/someImage:1.0.0
• --helm-image someRegistry/someImage:1.0.0
• --kubeval-image someRegistry/someImage:1.0.0
• --helmkubeval-image someRegistry/someImage:1.0.0
• --yamllint-image someRegistry/someImage:1.0.0

Images used by various tools and exercises can be configured using the following parameters:

• --grafana-image someRegistry/someImage:1.0.0
• --external-secrets-image someRegistry/someImage:1.0.0
• --external-secrets-certcontroller-image someRegistry/someImage:1.0.0
• --external-secrets-webhook-image someRegistry/someImage:1.0.0
• --vault-image someRegistry/someImage:1.0.0
• --nginx-image someRegistry/someImage:1.0.0
• --maven-image someRegistry/someImage:1.0.0

Note that specifying a tag is mandatory.

If you are using a remote cluster, you can set the --argocd-url parameter so that argocd-notification messages have a link to the corresponding application. Otherwise, argocd.$base-url is used.

You can specify email addresses for notifications (note that by default, MailHog will not actually send emails)

• --argocd-email-from: Sender E-Mail address. Default is argocd@example.org)
• --argocd-email-to-admin: Alerts send to admin. Default is infra@example.org)
• --argocd-email-to-user: Alerts send to user. Default is app-team@example.org)

Set the parameter --monitoring to enable deployment of monitoring and alerting tools like prometheus, grafana and mailhog.

See Monitoring tools for details.

You can specify email addresses for notifications (note that by default, MailHog will not actually send emails)

• --grafana-email-from: Sender E-Mail address. Default is grafana@example.org
• --grafana-email-to: Recipient E-Mail address. Default is infra@example.org

The gitops-playground uses MailHog to showcase notifications.
Alternatively, you can configure an external mailserver.

Note that you can't use both at the same time.
If you set --mail parameter, MailHog will be installed
If you set --smtp-* parameters, a external Mailserver will be used and MailHog will not be deployed.

Set the parameter --mail to enable MailHog.

This will deploy MailHog and configure Argo CD and Grafana to send mails to MailHog.
Sender and recipient email addresses can be set via parameters in some applications, e.g. --grafana-email-from or --argocd-email-to-user.

Parameters:

• --mail: Activate MailHog as internal Mailserver
• --mail-url: Specify domain name (ingress) under which MailHog will be served

If you want to use an external Mailserver you can set it with these parameters

• --smtp-address: External Mailserver SMTP address or IP
• --smtp-port: External Mailserver SMTP port
• --smtp-user: External Mailserver login username
• --smtp-password: External Mailservers login password. Make sure to put your password in single quotes.

This will configure Argo CD and Grafana to send mails using your external mailserver.
In addition you should set matching sender and recipient email addresses, e.g. --grafana-email-from or --argocd-email-to-user.

Set the parameter --vault=[dev|prod] to enable deployment of secret management tools hashicorp vault and external secrets operator.

--vault-url specifies domain name, Otherwise, vault.$base-url is used.

See Secrets management tools for details.

Is implemented by cert-manager. Set the parameter --cert-manager to enable cert-manager. For custom images use this parameters to override defaults:

• --cert-manager-image
• --cert-manager-webhook-image
• --cert-manager-cainjector-image
• --cert-manager-acme-solver-image
• --cert-manager-startup-api-check-image

i.e.

GOP includes some pre-defined profiles for easy usage. e.g. set --profile=full to start GOP with all features enabled.

Current existing profiles for argocd in non-operator mode:

• full - all features enabled
• minimal - starts only with ArgoCD and SCM-Manger
• content-examples - starts with ArgoCD, Jenkins, SCM-Manager and Petclinic

Follow profils for ArgoCD in Operator mode which has to be installed first:

• operator-full - all features enabled
• operator-minimal - starts only with ArgoCD and SCM-Manger
• operator-content-examples - starts with ArgoCD, Jenkins, SCM-Manager and Petclinic
• operator-mandant - starts mandant/tenant example

For k3d, you can just k3d cluster delete gitops-playground. This will delete the whole cluster. If you want to delete k3d use rm .local/bin/k3d.

To remove the playground without deleting the cluster, use the option --destroy. You need to pass the same parameters when deploying the playground to ensure that the destroy script can authenticate with all tools. Note that this option has limitations. It does not remove CRDs, namespaces, locally deployed SCM-Manager, Jenkins and registry, plugins for SCM-Manager and Jenkins

• In general: We cannot use the host network, so it's easiest to access via ingress controller and ingresses.
• --base-url=http://localhost --ingress should work on both Windows and Mac.
• In case of problems resolving e.g. jenkins.localhost, you could try using --base-url=http://local.gd or similar, as described in local ingresses.

On macOS and when using the Windows Subsystem Linux on Windows (WSL), you can just run our TL;DR command after installing Docker.

For Windows, we recommend using Windows Subsystem for Linux version 2 (WSL2) with a native installation of Docker Engine, because it's easier to set up and less prone to errors.

For macOS, please increase the Memory limit in Docker Desktop (for your DockerVM) to be > 10 GB. Recommendation: 16GB.

When you encounter errors with port 80 you might want to use e.g.

• init-cluster.sh) --bind-ingress-port=8080 and
• --base-url=http://localhost:8080 instead.

• As mentioned in the previous section, we recommend using WSL2 with a native Docker Engine.
• If you must, you can also run using Docker Desktop from native Windows console (see bellow)
• However, there seems to be a problem when the Jenkins Jobs running the playground access docker, e.g.

• In Docker Desktop, it's recommended to use WSL2 as backend.
• Using the Hyper-V backend should also work, but we experienced random CrashLoopBackoffs of running pods due to liveness probe timeouts.
  Same as for macOS, increasing the Memory limit in Docker Desktop (for your DockerVM) to be > 10 GB might help.
  Recommendation: 16GB.

Here is how you can start the playground from a Windows-native PowerShell console:

• Install k3d, see init-cluster.sh for K3D_VERSION, e.g. using winget

• Create k3d cluster. See K3S_VERSION in init-cluster.sh for $image, then execute

• Note that
  • You can ignore the warning about docker.sock
  • We're mounting the docker socket, so it can be used by the Jenkins Agents for the docker-plugin.
  • Windows seems not to provide a group id for the docker socket. So the Jenkins Agents run as root user.
  • If you prefer running with an unprivileged user, consider running on WSL2, Mac or Linux
  • You could also add -v gitops-playground-build-cache:/tmp@server:0 to persist the Cache of the Jenkins agent between restarts of k3d containers.
• Apply playground:
  Note that when using a $registry_port other than 30000 append the command --internal-registry-port=$registry_port bellow

As described above the GitOps playground creates a complete GitOps-based operational stack that can be used as an internal developer platform (IDP) on your Kubernetes cluster.

The stack is composed of multiple applications, where some of them can be accessed via a web UI.

• Argo CD
• Prometheus/Grafana
• Jenkins
• SCM-Manager
• Vault
• Ingress
• Cert-Manager

In addition, there are example applications that provide a turnkey solution for GitOps-Pipelines from a developer's point of view. See Example Applications.

We recommend using the --ingress and --base-url Parameters. With these, the applications are made available as subdomains of base-url.

For example, --base-url=http://localhost leads to `

• http://argocd.localhost
• http://grafana.localhost
• http://jenkins.localhost
• http://scmm.localhost
• http://vault.localhost

Of course, this would also work for production instances with proper domains, see Deploy Ingresses.

All applications are deployed via GitOps and can be found in the cluster-resources repository. See Argo CD for more details on the repository structure.

If deployed within the cluster, all applications can be accessed via: admin/admin

Note that you can change (and should for a remote cluster!) the password with the --password argument. There also is a --username parameter, which is ignored for argocd. That is, for now Argo CD's username always is admin.

Argo CD is installed in a production-ready way that allows for operating Argo CD with Argo CD, using GitOps and providing a repo per team pattern.

When installing the GitOps playground, the following Git repositories are created and initialized:

• example-apps - example GitOps repository for a developer / application team
• cluster-resources - example GitOps repository for a cluster admin or infra / platform team

Argo CD's own management and configuration, which previously lived in a dedicated argocd repository, is now part of the cluster-resources repo under apps/argocd:

When the GitOps playground is installed, Argo CD is bootstrapped as follows:

1. Argo CD is installed imperatively via a Helm chart.

2. Two resources are applied imperatively to the cluster:

   • an AppProject called argocd
   • an Application called bootstrap

   Both are stored in the cluster-resources repository under apps/argocd/applications.

From there, everything is managed via GitOps.

The following Argo CD Applications live in apps/argocd/applications:

• The bootstrap application manages the apps/argocd/applications folder (including itself). This allows changes to the bootstrap application and further application manifests to be managed via GitOps.
• The argocd application manages the folder apps/argocd/argocd, which contains Argo CD's resources as an umbrella Helm chart.
  • The umbrella chart pattern allows us to:
    • describe configuration in values.yaml
    • deploy additional resources (such as secrets and ingresses) via the templates folder
  • Chart.yaml declares the Argo CD Helm chart as a dependency.
  • Chart.lock pins the chart to a deterministic version from the upstream chart repository.
  • This mechanism can be used to upgrade Argo CD via GitOps (by updating the chart version and syncing).
• The projects application manages the folder apps/argocd/projects, which contains the following AppProject resources:
  • the argocd project (used for bootstrapping)
  • the built-in default project (restricted for security)
  • one project per team, for example:
    • cluster-resources (platform admins, more cluster permissions)
    • example-apps (application developers, fewer permissions)

Feature deployments (for example, monitoring, ingress, or other GOP features) are modeled as multi-source Argo CD Applications instead of using an App-of-Apps pattern.

For some features, the GitOps Playground Operator (GOP):

1. Writes values files into the cluster-resources repository under:

   apps/<feature>/
   <feature>-gop-helm.yaml
   <feature>-user-values.yaml

   The *-gop-helm.yaml file is managed by GOP, while *-user-values.yaml is intended for user overrides and is not overwritten.

2. Generates an Argo CD Application in apps/argocd/applications/.yaml that uses two sources:

   • Helm source (external chart)
     • repoURL: the external Helm repository
     • chart or path: the chart to deploy
     • targetRevision: the chart version
     • helm.valueFiles: includes the values from the cluster-resources repo via $values/... (for example $values/apps//-gop-helm.yaml and $values/apps//-user-values.yaml)
   • Git source (values and additional manifests)

     • repoURL: the cluster-resources repo

     • targetRevision: typically main

     • ref: set to values so the Helm source can reference $values/...

     • path: apps/

     • directory.recurse: true to pick up additional manifests in the feature folder

       If users create a misc subfolder under apps/ (for example apps//misc) and add additional Kubernetes manifests there, these manifests are automatically included and deployed as part of the feature.

Argo CD merges these sources, so each feature application is defined by:

• the external Helm chart (versioned and reproducible), and
• Git-managed configuration and manifests in the cluster-resources repo.

This multi-source pattern replaces the previous App-of-Apps based approach for feature deployments while still following the repo-per-team model.

The example-apps repository demonstrates how application teams can structure their own GitOps repositories. Its layout looks like this:

• The folder apps/argocd/applications contains Argo CD Application manifests for the example workloads:
  • petclinic-plain.yaml
• Each application implements the Environment per App Pattern::
  • separate folders for stagingand production
  • optional generatedResources/ subfolders where CI pipelines can write generated manifests (for example, templated messages or index files)

For example:

• apps/spring-petclinic-plain/production and apps/spring-petclinic-plain/staging contain plain Kubernetes manifests (deployment.yaml, service.yaml, ingress.yaml) plus generated resources.

The example-apps repo is thus a reference for how product teams can structure their GitOps repositories while still integrating cleanly with Argo CD and the multi-source pattern used by the platform.

To keep things simpler, the GitOps playground only uses one kubernetes cluster, effectively implementing the Standalone pattern. However, the repo structure could also be used to serve multiple clusters, in a Hub and Spoke pattern: Additional clusters could either be defined in the vaules.yaml or as secrets via the templates folder.

We're also working on an optional implementation of the namespaced pattern, using the Argo CD operator.

And advanced question: Why does the GitOps playground not use the argocd-autopilot?

The short answer is: As of 2023-05, version 0.4.15 it looks far from ready for production.

Here is a diagram that shows how the repo structure created by autopilot looks like:

Here are some thoughts why we deem it not a good fit for production:

• The version of ArgoCD is not pinned.
  • Instead, the kustomization.yaml (3 in the diagram) points to a base within the autopilot repo, which in turn points to the stable branch of the Argo CD repo.
  • While it might be possible to pin the version using Kustomize, this is not the default and looks complicated.
  • A non-deterministic version calls for trouble. Upgrades of Argo CD might happen unnoticed.
  • What about breaking changes? What about disaster recovery?
• The repository structure autopilot creates is more complicated (i.e. difficult to understand and maintain) than the one used in the playground
  • Why is the autopilot-bootstrap application (1 in the diagram) not within the GitOps repo and lives only in the cluster?
  • The approach of an ApplicationSet within the AppProject's yaml pointing to a config.json (more difficult to write than YAML) is difficult to grasp (4 and 6 in the diagram)
  • The cluster-resources ApplicationSet is a good approach to multi-cluster but again, requires writing JSON (4 in the diagram).
• Projects are used to realize environments (6 and 7 in the diagram).
  How would we separate teams in this monorepo structure?
  One idea would be to use multiple Argo CD instances, realising a Standalone pattern. This would mean that every team would have to manage its own ArgoCD instance.
  How could this task be delegated to a dedicated platform team? These are the questions that lead to the structure realized in the GitOps playground.

The playground installs cluster-resources (like prometheus, grafana, vault, external secrets operator, etc.) via the repo
argocd/cluster-resources.

When installing without Argo CD, the tools are installed using helm imperatively. We fall back to using imperative helm installation as a kind of neutral ground.

You can set an external jenkins server via the following parameters when applying the playground. See parameters for examples.

• --jenkins-url,
• --jenkins-username,
• --jenkins-password

The user has to have the following privileges:

• install plugins
• set credentials
• create jobs
• restarting

To apply additional global environments for jenkins you can use --jenkins-additional-envs "KEY1=value1,KEY2=value2" parameter.

Note that the example applications pipelines will only run on a Jenkins that uses agents that provide a docker host. That is, Jenkins must be able to run e.g. docker ps successfully on the agent.

You can choose between the following Git providers:

• SCM-Manager
• GitLab

For configuration details, see the CLI or configuration parameters above (SCM).

When using GitLab, you must provide a valid parent group ID. This group will serve as the main group for the GOP to create and manage all required repositories.

To authenticate with Gitlab provide a token token as password. More information can be found here or here The username should remain 'oauth2.0' to access the API, unless stated otherwise by GitLab documentation.

You can set an external SCM-Manager via the following parameters when applying the playground. See parameters for examples.

• --scmm-url,
• --scmm-username,
• --scmm-password

The user on the scm has to have privileges to:

• add / edit users
• add / edit permissions
• add / edit repositories
• add / edit proxy
• install plugins

Set the parameter --monitoring so the kube-prometheus-stack via its helm-chart is being deployed including dashboards for

• ArgoCD
• Traefik Ingress Controller
• Prometheus
• SCMManager
• Jenkins.

Grafana can be used to query and visualize metrics via prometheus. It is exposed via ingress, e.g. http://grafana.localhost. Prometheus is not exposed by default.

In addition, argocd-notifications is set up. Applications deployed with Argo CD now will alert via email to mailhog the sync status failed, for example.

Note that this only works with Argo CD so far

Via the vault parameter, you can deploy Hashicorp Vault and the External Secrets Operator into your GitOps playground.

With this, the whole flow from secret value in Vault to kubernetes Secret via External Secrets Operator can be seen in action:

For this to work, the GitOps playground configures the whole chain in Kubernetes and vault (when dev mode is used):

• In k8s namespaces argocd-staging and argocd-production:
  • Creates SecretStore and ServiceAccount (used to authenticate with vault)
  • Creates ExternalSecrets
• In Vault:
  • Create secrets for staging and prod
  • Create a human user for changing the secrets
  • Authorizes the service accounts on those secrets
• Creates an example app that uses the secrets

For testing you can set the parameter --vault=dev to deploy vault in development mode. This will lead to

• vault being transient, i.e. all changes during runtime are not persisted. Meaning a restart will reset to default.
• Vault is initialized with some fixed secrets that are used in the example app, see below.
• Vault authorization is initialized with service accounts used in example SecretStores for external secrets operator
• Vault is initialized with the usual admin/admin account (can be overriden with --username and --password)

The secrets are then picked up by the vault-backend SecretStores (connects External Secrets Operator with Vault) in the namespace argocd-staging and argocd-production namespaces

When using vault=prod you'll have to initialize vault manually but on the other hand it will persist changes.

If you want the example app to work, you'll have to manually

• set up vault, unseal it and
• authorize the vault service accounts in argocd-production and argocd-staging namspaces. See SecretStores and dev-post-start.sh for an example.

With vault in dev mode and ArgoCD enabled, the example app applications/nginx/argocd/helm-jenkins will be deployed in a way that exposes the vault secrets secret//nginx-secret via HTTP on the URL http:///secret, for example http://staging.nginx-helm.nginx.localhost/secret.

While exposing secrets on the web is a bad practice, it's good for demoing auto reload of a secret changed in vault.

To demo this, you could

• change the staging secret
• Wait for the change to show on the web, e.g. like so

This usually takes between a couple of seconds and 1-2 minutes.
This time consists of ExternalSecret's refreshInterval, as well as the kubelet sync period (defaults to 1 Minute)

• cache propagation delay

The following video shows this demo in time-lapse:

secrets-demo-video.mp4

The playground comes with example applications that provide a turnkey solution for GitOps-Pipelines
from a developer's point of view.

These can be enabled using --content-examples.
They require a registry, so locally use --registry or pass in an existing instance using registry-url.
The examples very much rely on jenkins. So it is recommended to enable it using --jenkins or pass in an existing instance using --jenkins-url.

The examples include staging and production environments, providing a ready-to-use solution for promotion.

All applications are deployed via separated application and GitOps repos:

• Separation of app repo (e.g. petclinic-plain) and GitOps repo (e.g. argocd/example-app)
• Config is maintained in app repo,
• CI Server writes to GitOps repo and creates PullRequests.

The applications implement a simple staging mechanism:

• After a successful Jenkins build, the staging application will be deployed into the cluster by the GitOps operator.
• Deployment of production applications can be triggered by accepting pull requests.
• For some applications working without CI Server and committing directly to the GitOps repo is pragmatic
  (e.g. 3rd-party-application like NGINX, like argocd/nginx-helm-umbrella)

Note that the GitOps-related logic is implemented in the gitops-build-lib for Jenkins. See the README there for more options like

• staging,
• resource creation,
• validation (fail early / shift left).

For further understanding, also take a look at our GitOps pattern repository cloudogu/gitops-patterns

Please note that it might take about a minute after the pull request has been accepted for the GitOps operator to start deploying. Alternatively, you can trigger the deployment via ArgoCD's UI or CLI.

We recommend using the --ingress and --base-url Parameters. With these, the applications are made available as subdomains of base-url.

For example, --base-url=http://localhost leads to http://staging.petclinic-plain.petclinic.localhost/.

The .petlinic. part can be overridden using --petclinic-base-domain (for the petlinic examples/exercises), or --nginx-base-domain (for the nginx examples/exercises).

Jenkinsfile for plain deployment

• Staging: http://staging.petclinic-plain.petclinic.localhost/
• Production: http://production.petclinic-plain.petclinic.localhost/
  Note that you have to accept a pull request for deployment

Jenkinsfile for helm deployment

• Staging: http://staging.petclinic-helm.petclinic.localhost/
• Production: http://production.petclinic-helm.petclinic.localhost/
  Note that you have to accept a pull request for deployment

Jenkinsfile

• Staging: http://staging.nginx-helm.nginx.localhost/
• Production: http://production.nginx-helm.nginx.localhost/
  Note that you have to accept a pull request for deployment

• http://production.nginx-helm-umbrella.nginx.localhost/

See docs/developers.md

Copyright (c) 2020 - present Cloudogu GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

See LICENSE for details.

GitOps Playground(c) for use with Argo(tm), Git(tm), Jenkins(r), Kubernetes(r), Grafana(r), Prometheus(r), Vault(r) and SCM-Manager

Argo(tm) is an unregistered trademark of The Linux Foundation(r)
Git(tm) is an unregistered trademark of Software Freedom Conservancy Inc.
Jenkins(r) is a registered trademark of LF Charities Inc.
Kubernetes(r) and the Kubernetes logo(r) are registered trademarks of The Linux Foundation(r)
K8s(r) is a registered trademark of The Linux Foundation(r)
The Grafana Labs Marks are trademarks of Grafana Labs, and are used with Grafana Labs' permission. We are not affiliated with, endorsed or sponsored by Grafana Labs or its affiliates.
Prometheus(r) is a registered trademark of The Linux Foundation(r)
Vault(r) and the Vault logo(r) are registered trademarks of HashiCorp(r) (http://www.hashicorp.com/)

Written Offer for Source Code:

Information on the license conditions and - if required by the license - on the source code is available free of charge on request.
However, some licenses require providing physical copies of the source or object code. If this is the case, you can request a copy of the source code. A small fee is charged for these services to cover the cost of physical distribution.

To receive a copy of the source code, you can either submit a written request to

Cloudogu GmbH
Garkuche 1
38100 Braunschweig

or you may email hello@cloudogu.com.

Your request must be sent within three years from the date you received the software from Cloudogu that is the subject of your request or, in the case of source code licensed under the AGPL/GPL/LGPL v3, for as long as Cloudogu offers spare parts or customer support for the product, including the components or binaries that are the subject of your request.