Solving Operator Extensibility: A gRPC Plugin Framework for kubernetes

From the guys at EDB (the comercial offerin connected to CNPG). If you haven’t hear of CNPG: Use it, please

Why

  • Modify beyond CRDs (Inject Sidecars, run modifications, …)
  • Smarter reconciliation: Pause/Skip on evernal logic
  • Integrate with third party sdks (backup management, storeage, …)

Current Options

  • Patch it upstream -> and hope they agree with your PR)
  • Fork it -> now you have to maintain it
  • Mutating Webhooks -> Doesn’t know about the working of the operator ans just blindly forces resource modifications

Sample Use-Case

  • Project: CloudNativePG
  • Goal: Extend backup functionality
  • Solution: CNPG-I (inspired by CSI)
    • Protobuf contract
    • Decoule of lifecycle/deplyoment: Plugins run as pods in the cnpg namespace
    • Capability based dicovocery: Automatic registration and it will be called only if needed
    • Orchestrated delegation: Operator is still “the brain” but can delegate

TODO: Steal diagram from slides

Operator Hooks

  • Lifecycle: During reconcile -> Modify
  • Reconciler: Interaction during the reconciliation loop -> Stop, Requeue, Continue
  • Metrics Collection: Register additional metrics
  • Backups: Execute the backup
  • WAL Archiving: Handle WAL archiving externally

Takeaways

  • A good way to move from a monolith to modularity (like csi did for storage maintainance in kubernetes core)
  • Interface first is usually the better choice for long term extensibility
  • (From Q&A): External Secrets is also thinking about going the plugin way and this would increase security by only deploying the providers you need and also moving stuff out of the main codebase