00. About what?
Let’s consider a seemingly harmless situation. You deployed a new Kubernetes cluster, connected the grid and storage, rolled out monitoring and quotas. It would seem that all that remains is to cut up namespaces and hand them over to colleagues in development for use. However, you are aware that developers will run commands through kubectl, which means that, in an amicable way, you need to impose at least “basic” restrictions on their commands, because this is the only way to protect yourself from most problems during the upcoming operation of the cluster.
But, somehow, there was no ready-made guide at hand with a list of “basic” restrictions on the cluster, and the good old senior admin went for a promotion to a large corporation? And so, with a quick search, you land on this article. Well… let’s not make your developers wait too long, just about half a day. Although, nothing can stop you from skipping the theoretical platitudes of this friendly conversation and, using the examples posted at the very end, describe the security policies you need in an hour.
Of course you are right, this will not be like a ready-made tutorial. However, no matter how complicated setting up a cluster may seem, if you have some time, then you are welcome to briefly discuss the capabilities of Kubernetes controllers and practical ways to use Validating Admission Policy.
ИНСТРУКЪЦЫЯ ПО НАСТРОЙКЕ КУБЪ-КЛАСТЕРА
...
шаг 998 Подключите Validating Admission Policies
шаг 999 Отдайте, наконецъ, кластеръ в эксплуатацию
...
01. Why?
When developers do kubectl apply -f my-deployment.yaml
-
this command is sent as a request to the Kubernetes API server
-
if authn / authz are completed successfully, then the request is forwarded to admin controllers
-
the request will pass through the chain all the necessary controllers
-
each “suitable” controller will carry out its own validation
-
if all controllers are OK, then eventually it will come to Scheduler
-
Scheduler will plan resources and trigger the creation of pods based on their specifications
When the request reaches Scheduler, then the specifications for the resource will already be, as it were, formed, and the quotas for its creation will almost be allocated. There is no way to change anything “on the fly”; all that remains is to screw in something additional and launch it as a new process, which in itself seems to hint: any of our undertakings at this stage will cause a guaranteed overhead on the cluster resources. Therefore, we are encouraged to intervene a little earlier.
Admission controllers are an important part of Kubernetes that manages resources and implements critical security policies even before important actions and events occur. Admissions are intended:
-
filter the creation and modification of resources at the request stage
// only those requests to the API server that satisfy our rules will be executed successfully -
apply security policies for different types of resources
// dynamically “on the fly” change the resource specs to suit the required specifics (labels, annotations, health checks, injectors…) -
monitor events that may affect cluster runtime
// “dangerous” changes to quotas, nodes, storage, system services…
02. But how?
To do this, administration rules are described, which indicate which resources they should be applied to. For example, the command to create a deployment involves a chain of several controllers, each of which will go through its own admins (which, in turn, are also 2 types of controllers – mutate and validate):
…
DEPLOYMENT CONTROLLER (mutate, validate, persist to etcd, request to api)
REPLICASET CONTROLLER (mutate, validate, persist to etcd, request to api)
POD CONTROLLER (mutate, validate, persist to etcd, request to api)
…
mutate, validate – these are the types of administration controllers that will take care of your request if it produces one ACTION over that RESOURCE, which this same controller “controls” (binding). And exactly in this sequence:
. mutate webhook modifies the manifest from your request
// and evaluates the output for validity with Object Schema Validation
// this is how sidecars are injected, labels, annotations, etc. are added.
. validate hooks run the final manifesto through their rules
// the request will be canceled if the final version of the manifest does not match at least one of the rules,
// although, instead of canceling, individual controllers can be skipped.
This is how built-in (BUILD-IN) admins work for each of the controllers that run in the controller manager. They work according to standard rules with standard cluster resources, that is, they call standard webhooks. Recently, you can view their list, connect, disconnect…
kube-apiserver -h | grep enable-admission-plugins
kube-apiserver --enable-admission-plugins=.., .., ...
kube-apiserver --disable-admission-plugins=.., .., ...
But what if we want to expand the list of such rules a little, say, prohibit the creation of services like NodePort, or, for example, limit the creation of certain resources in production environments under a number of conditions?
To implement this… PREVIOUSLY you had to use Third-party-tool.
This is expensive, but necessary in cases where it is important:
. automatically inject sidecars for service mesh (Linkerd, Istio)
. add annotations for observability (Prometheus-like in specific installations)
To implement this… NOW (starting from version 1.26)
. no need for Third-party-tool with all the bonuses
. you don’t have to wait for overhead because you don’t need to create new webhooks
. no need to bother with authorizations, certificates…
. but you still have to learn the expression language (CEL https://github.com/google/cel-spec)
03. So how?
Maximum customizable and optionally parametric. No more difficult than writing rules in Alert Manager.
First, check whether admission plugins and the required api are connected to your clustercontrolplane$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
containers:
- command:
- kube-apiserver
- ...
- --enable-admission-plugins=NodeRestriction #1
- --runtime-config=admissionregistration.k8s.io/v1alpha1 #2
- --feature-gates=ValidatingAdmissionPolicy=true #3
- ...
If the API server was installed as a systemd service, then the settings are expected to be edited in the systemd unit file.
So, the plugins are connected and tested. Now you need to create at least 2 resources
1. Validating Admission Policy
Полиси - это по сути список правил, описанных с помощью языка выражений CEL (см.выше)
2.ValidatingAdmissionPolicyBinding Биндинг - это по сути скоуп ресурсов, на который мы хотим распространить данные правила
3. * Параметр - это всего лишь опция, но вероятно game-changer кандидат.
не исключено, что однажды вам понадобится избавиться от внешних источников для приложений, например, пошарить стейт занятости вашей GPU между разными сервисами внутри кластера.
Вам понадобится создать отдельный ресурс для хранения нужного GPU стейта в виде параметра и продумать логику - кто, как и при каких условиях будет изменять его значение.
Готово. вы привязали поведение разных сервисов к одному общему параметру внутри кластера
Try creating test policies and bindings using the official doc
https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
Following the example of test policies, create your own. To do this, we will present several options as an example. Just remember to update the bindings accordingly along with them.
Hidden text
// запретить в деплойменте использовать неизвестные адреса
// реджистри для пулла имиджей в кластер
---
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
variables:
- name: containers
expression: "object.spec.template.spec.containers"
validations:
- expression: "!(variables.containers.filter(c, c.image.contains('company.ru/'))"
messageExpression: "'only approved company.ru images are allowed'"
// запретить создавать сервисы с типом NodePort
---
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["*"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["services"]
validations:
- expression: "object.spec.type == 'NodePort'"
reason: Invalid
// запретить создание новых джобов в кастомном ресурсе, пока GPU находится в статусе >0
---
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apiextensions.k8s.io"]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["jobs"]
validations:
- expressions: "object.spec.needState < params.stateGPU"
messageExpression: "'GPU is busy with state ' + string(params.stateGPU)"
If everything worked out, then you are ready to cover the cases you need with security policies and ensure the protection of your cluster. Hurry up, add “basic” policies and give this cluster to the developers, business won’t wait!
Note that you didn’t have to install third-party-tools and calculate their load.
And if you need to impose new restrictions, then you already know how to do it.
04. That’s all?
Wait… What are the disadvantages?
In the first stages, apart from the restrictions that you need, in principle, there are no disadvantages. But as the number of new policies increases, the load on webhooks will increase, and then, most likely, you will want to scale them. What can be done for this?
-
increase the availability of webhooks by deploying several webhook server instances into a separate webhook-namespace
-
wrap admin webhooks (more precisely, webhook server) in monitoring to track their performance and availability
-
consider reading and analyzing /var/log/webhook-server.log in order to have confidence in the security of using the policy
-
use dry-run validator of YAML files for admin controllers, for example, kubeconform
Do you have anything to suggest, add, or object to? You are welcome to comment, I will be glad to see interesting options from your experience, unexpected opinions and fantastic hypotheses.
Thank you very much for your attention if you have read this far. Yes, and whether it’s the technical part in the operation of Kubernetes components, the architecture and security of deploying applications in k8s, feel free to show healthy criticism of my research in the comments.
PS including, do not skimp on criticism in such a difficult area as the spelling of the article text and the correctness of presentation of complex concepts in simple Russian, in order to eliminate ambiguity and misleading and help make texts better.
P.P.S. If you already have a ready-made “bicycle” or just an idea to develop something useful for working with cuber, allowing you to simplify the work on the configuration of cube applications, including expanding the capabilities of the cube for specific tasks ML/AI-operations, in-memory db, custom balancers and schedulers, or simply automate some part of the routine, then do not hesitate to write in a personal message with suggestions for mutually beneficial cooperation.
Acknowledgement and Usage Notice
The editorial team at TechBurst Magazine acknowledges the invaluable contribution of the author of the original article that forms the foundation of our publication. We sincerely appreciate the author’s work. All images in this publication are sourced directly from the original article, where a reference to the author’s profile is provided as well. This publication respects the author’s rights and enhances the visibility of their original work. If there are any concerns or the author wishes to discuss this matter further, we welcome an open dialogue to address potential issues and find an amicable resolution. Feel free to contact us through the ‘Contact Us’ section; the link is available in the website footer.