Achiving Platform Engineering Multi-Tenancy with kcp and Crossplane

Code/Demo

An introductory talk to kcp and crossplanes by the companies maintaining both of them.

The basics

  • A platform should me automated and self-service driven to count as platform engineering
  • Provider teams: Certificates, databases, …
  • Consumer teams: Want to use a provided Service
  • IDP: Sits in the middle -> The real hard part

KCP

  • Idea: Why not use Kubernetes as our API-Layer? It tracks API ownership, versioning and resource managment and has built-in extensibility (CRD)
  • Problems:
    • APIs are always cluster-scoped (you advertise them to everyone) -> You could give everyone a cluster
    • Ramping up a new cluster takes time and resources -> Let’s just create a lightweight hosted control plane with it’s own datastore
    • Sharing APIs to multiple clusters is hard -> Leightweight control planes with a shared datastore
  • Solution: Workspaces that are organized in a tree and each workspace contains it’s own CRDs and RBAC -> All resources (e.g. namespaces) exist just in their own workspace
  • API-Sharing; APIExport CRD and APIBinding CRD (reference via the workspace path of the APIExport)
  • Running the operators that work on the APIs: Virtual Workspaces (virtually connects your operator to all of their resources across kcp via a magic kubeconfig) -> Controller needs to be built with multicluster-runtime (drop in replacement for controler runtime)
  • KCP API-Syncagent allows you to use a existing operator without modifying it for use with multicluster-runtime
graph
    KCP
    Datastore
    User
    subgraph Workspace
        APIs[API/CRD]
        RBAC
    end
    KCP-->|interact with|Datastore
    User-->|Create tenant|KCP
    KCP-->|Manages|Workspace
    KCP-->|Return kubeconfig|User
    User-->|Uses KCP like the apiserver|KCP

Crossplane

  • Providers for all kunds of resources (kubernetes or infra/cloud)
  • Compositions for higher level abstractions accross one or multiple providers
  • Uses the Kubernetes API (aka CRDs) as it’s api to enable integration with standardized tooling (like GitOps)
apiVersion: ...
kind: CompositeResourceDefinition
spec:
    compositetyperef:
        group: my.exdample/v1aplha1
        kind: Test
    mode: pipeline
    pipeline:
        - ...

The demo

I recommend watching the recording but thul shall serve as a overview of the scenario. Or run it locally (code linked above).

  • User whants to order a new database in their workspace a
  • Database team offers their API through their database workspace
  • Database team runs their operator in their own cluster
  • kcp api-syncagent swyncs the database crd from workspace a into the db team’s cluster and the connection-secrets back to the workspace