Helm charts are supposed to give you an easy way to install a set of kubernetes resources. In theory you just choose the chart you want to install, pass your variables and away you go. In practice there is usually a lot of different configuration values and some are not intuitive or explained so you end up having to dig through the templates to figure out what configuration you need.

When developing a helm chart myself I want the values to have descriptive self-explanatory names and if the name doesn’t say it all, then document it. The helm charts I’ve worked on are not public but here is an example of what you could do with commenting a values.yaml file: https://github.com/concourse/concourse-chart/blob/master/values.yaml. You won’t necessarily want to go quite this verbose but I think this is a useful example of the kind of information users want to know and how you can format it. For example – how do you indicate what values are acceptable if this is not self-evident?

I think the way Crossplane explains this could be clearer.

CompositeResourceDefinition (XRD, related but not exactly the same as a CRD) – this is like the interface for a function that defines the parameters, their types and which are required.

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xglobal-vault-accesses.azure.simbachain.com
spec:
  group: azure.simbachain.com
  # for the CompositeResource XR, the actual resource created by the claim
  names:
    kind: XGlobal-Vault-Access
    plural: xglobal-vault-accesses
  claimNames:
    kind: Global-Vault-Access
    plural: global-vault-accesses
  versions:
    - name: v1beta1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
<<snip>>

Composition – this is like the internal implementation code of the function, note however that there can be more than one implementation of a given function with labels used to distinguish them.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: global-vault-access
  labels:
    crossplane.io/xrd: xglobal-vault-accesses.azure.simbachain.com
    provider: azure
spec:
  compositeTypeRef:
    apiVersion: azure.simbachain.com/v1beta1
    kind: XGlobal-Vault-Access
  resources:
<<snip>>

Claim – this is a request like the invocation of the function, you pass in your parameters. Note: the kind is not Claim, it will be whatever was under claimNames > kind. The word Claim generally does not appear anywhere, you can add it to a comment if that helps

# below is a crossplane claim that requests an instance of XGlobal-Vault-Access and its managed resources
---
apiVersion: azure.simbachain.com/v1beta1
kind: Global-Vault-Access
metadata:
  annotations:
    crossplane.io/external-name: blocks-simba-kongtest
  # will be used to generate names for the resources in the composition
  name: blocks-simba-kongtest
  namespace: blocks-simba-kongtest
spec:
  compositionRef:
    name: global-vault-access
  parameters:
    location: eastus
<<snip>>

CompositeResource (XR) – Sometimes this is talked about as if it’s the same as the claim but it’s a separate Kubernetes resource and it’s important to differentiate them. This results from the resources above instead of being defined directly. You define an XRD, make a Composition that defines the internal implementation of that XRD, make a claim that passes parameters then a CompositeResource will be created automatically. When I am troubleshooting this is the first place I want to look because it will has Synced and Ready status and contains resource refs to all the individual resources. Note: the kind is not CompositeResource, it will be whatever was under names > kind. It is traditionally named starting with X in the XRD.

kubectl describe xGlobal-Vault-Access

There was no error showing on my Crossplane CompositeResource but one of the resources was not being created. No errors in the Crossplane log: kubectl -n crossplane-system logs -lapp=crossplane --since=1h

A helpful person on https://crossplane.slack.com/ssb/redirect pointed out that I needed to run crossplane with args --debug in order to see the error message. This surprised me, I am used to --debug being for relatively obscure information but now I know that in Crossplane this is not the case. Details here: https://github.com/crossplane/crossplane/discussions/4886

In the spirit of https://jvns.ca/blog/2023/10/06/new-talk–making-hard-things-easy/ and Matt Surabian’s DevOps Days Boston talk “Teaching and Learning When Words No Good Sense Make” – why is Kubernetes so hard?

In a word, complexity, but let’s break that out a couple dimensions of that complexity

Overloading of terms

Overloading is when a single term can have multiple meanings. For example, What The Heck Is Ingress? <— a whole episode of the Kubernetes Unpacked podcast devoted to that question

Very long, very nested resources defined in yaml

We have a single Kubernetes Composition in our codebase that is 529 lines long and has 5 levels of nesting. So the level of Cognitive Complexity is high, putting a strain on working memory.

Distributed system

Kubernetes is a distributed system because it spreads workloads, storage, and operations across multiple machines, coordinating them to appear as one cohesive system from the user’s perspective. This distributed nature allows Kubernetes to provide its core benefits of scalability, resilience, and flexibility. However,

Developing distributed utility computing services, such as reliable long-distance telephone networks, or Amazon Web Services (AWS) services, is hard. Distributed computing is also weirder and less intuitive than other forms of computing because of two interrelated problems. Independent failures and nondeterminism cause the most impactful issues in distributed systems. In addition to the typical computing failures most engineers are used to, failures in distributed systems can occur in many other ways. What’s worse, it’s impossible always to know whether something failed.

https://aws.amazon.com/builders-library/challenges-with-distributed-systems/

Declarative syntax

1. Indirectness of Action:
   • Unlike imperative languages where code flows sequentially, in Kubernetes, you specify a desired state. The system’s controllers then work to realize it. This indirect approach can make it difficult to pinpoint problems since you’re not commanding each step but relying on Kubernetes’ logic to interpret and act on your intent.
2. Asynchronous Behavior:
   • While imperative code executes in a predictable sequence, Kubernetes’ actions often run asynchronously. After declaring a desired state, the system might not immediately reflect that outcome. And it can be hard to tell if something has failed or whether it’s still processing.
3. Error Reporting:
   • In imperative programming, errors are usually tied to a specific action or code line. In Kubernetes, however, errors may be symptomatic of deeper issues. For example, a pod in a “CrashLoopBackOff” state might be due to various reasons, demanding a more in-depth examination of logs, events, and configurations to trace the root cause.

How do we make it easier?

There’s no one simple answer to this. But to start, I recommend:

• COMMENT your Kubernetes!
• learn the fundamentals: https://jvns.ca/blog/2017/06/04/learning-about-kubernetes/
• When you get discouraged, take a break but don’t give up. You’re not alone in your struggles!