Git-Mirrors/constellation
1
0
Fork 0
mirror of https://github.com/edgelesssys/constellation.git synced 2024-12-13 18:04:20 -05:00
Code Issues Packages Projects Releases Wiki Activity
949186e5d7
constellation/internal/helm/charts/aws-load-balancer-controller/README.md

259 lines
36 KiB
Markdown
Raw Normal View History

aws: use new LB controller to fix SecurityGroup cleanup on K8s service deletion (#2090) * add current chart add current helm chart * disable service controller for aws ccm * add new iam roles * doc AWS internet LB + add to LB test * pass clusterName to helm for AWS LB * fix update-aws-lb chart to also include .helmignore * move chart outside services * working state * add subnet tags for AWS subnet discovery * fix .helmignore load rule with file in subdirectory * upgrade iam profile * revert new loader impl since cilium is not correctly loaded * install chart if not already present during `upgrade apply` * cleanup PR + fix build + add todos cleanup PR + add todos * shared helm pkg for cli install and bootstrapper * add link to eks docs * refactor iamMigrationCmd * delete unused helm.symwallk * move iammigrate to upgrade pkg * fixup! delete unused helm.symwallk * add to upgradecheck * remove nodeSelector from go code (Otto) * update iam docs and sort permission + remove duplicate roles * fix bug in `upgrade check` * better upgrade check output when svc version upgrade not possible * pr feedback * remove force flag in upgrade_test * use upgrader.GetUpgradeID instead of extra type * remove todos + fix check * update doc lb (leo) * remove bootstrapper helm package * Update cli/internal/cmd/upgradecheck.go Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com> * final nits * add docs for e2e upgrade test setup * Apply suggestions from code review Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com> * Update cli/internal/helm/loader.go Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com> * Update cli/internal/cmd/tfmigrationclient.go Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com> * fix daniel review * link to the iam permissions instead of manually updating them (agreed with leo) * disable iam upgrade in upgrade apply --------- Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com> Co-authored-by: Malte Poll
2023-07-24 04:30:53 -04:00
# AWS Load Balancer Controller
AWS Load Balancer controller Helm chart for Kubernetes
## TL;DR:
```sh
helm repo add eks https://aws.github.io/eks-charts
# If using IAM Roles for service account install as follows - NOTE: you need to specify both of the chart values `serviceAccount.create=false` and `serviceAccount.name=aws-load-balancer-controller`
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
# If not using IAM Roles for service account
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system
```
## Introduction
AWS Load Balancer controller manages the following AWS resources
- Application Load Balancers to satisfy Kubernetes ingress objects
- Network Load Balancers to satisfy Kubernetes service objects of type LoadBalancer with appropriate annotations
## Security updates
**Note**: Deployed chart does not receive security updates automatically. You need to manually upgrade to a newer chart.
#### Node isolation
As a security best practice, we recommend isolating the controller deployment pods to specific node groups which run critical components. The helm chart provides parameters ```nodeSelector```, ```tolerations``` and ```affinity``` to configure node isolation. For more information, please refer to the guidance [here](https://aws.github.io/aws-eks-best-practices/security/docs/multitenancy/#isolating-tenant-workloads-to-specific-nodes).
## Prerequisites
- Kubernetes >= 1.19
- IAM permissions
- Helm v3
- Optional dependencies
- cert-manager
- Prometheus Operator
The controller runs on the worker nodes, so it needs access to the AWS ALB/NLB resources via IAM permissions. The
IAM permissions can either be setup via IAM roles for ServiceAccount or can be attached directly to the worker node IAM roles.
#### Setup IAM for ServiceAccount
1. Create IAM OIDC provider
```
eksctl utils associate-iam-oidc-provider \
--region <aws-region> \
--cluster <your-cluster-name> \
--approve
```
1. Download IAM policy for the AWS Load Balancer Controller
```
curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
```
1. Create an IAM policy called AWSLoadBalancerControllerIAMPolicy
```
aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam-policy.json
```
Take note of the policy ARN that is returned
1. Create a IAM role and ServiceAccount for the Load Balancer controller, use the ARN from the step above
```
eksctl create iamserviceaccount \
--cluster=<cluster-name> \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
--approve
```
#### Setup IAM manually
If not setting up IAM for ServiceAccount, apply the IAM policies from the following URL at minimum.
```
https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/main/docs/install/iam_policy.json
```
#### Upgrading from ALB ingress controller
If migrating from ALB ingress controller, grant [additional IAM permissions](https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy_v1_to_v2_additional.json).
#### Upgrading from AWS Load Balancer controller v2.1.3 and earlier
- Additional IAM permissions required, ensure you have granted the [required IAM permissions](https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json).
- CRDs need to be updated as follows
```shell script
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
```
- you can run helm upgrade without uninstalling the old chart completely
#### Installing cert-manager
If you are setting `enableCertManager: true` you need to have installed cert-manager and it's CRDs before installing this chart; to install [cert-manager](https://artifacthub.io/packages/helm/cert-manager/cert-manager) follow the installation guide.
The controller helm chart requires the cert-manager with apiVersion `cert-manager.io/v1`.
Set `cluster.dnsDomain` (default: `cluster.local`) to the actual DNS domain of your cluster to include the FQDN in requested TLS certificates.
#### Installing the Prometheus Operator
If you are setting `serviceMonitor.enabled: true` you need to have installed the Prometheus Operator ServiceMonitor CRD before installing this chart and have the operator running to collect the metrics. The easiest way to do this is to install the [kube-prometheus-stack](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack) Helm chart using the installation guide.
## Installing the Chart
**Note**: You need to uninstall aws-alb-ingress-controller. Please refer to the [upgrade](#Upgrade) section below before you proceed.
**Note**: Starting chart version 1.4.1, you need to explicitly set `clusterSecretsPermissions.allowAllSecrets` to true to grant the controller permission to access all secrets for OIDC feature. We recommend configuring access to individual secrets resource separately [[link](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/examples/secrets_access/)].
Add the EKS repository to Helm:
```shell script
helm repo add eks https://aws.github.io/eks-charts
```
Install the TargetGroupBinding CRDs:
```shell script
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
```
Install the AWS Load Balancer controller, if using iamserviceaccount
```shell script
# NOTE: The clusterName value must be set either via the values.yaml or the Helm command line. The <k8s-cluster-name> in the command
# below should be replaced with name of your k8s cluster before running it.
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<k8s-cluster-name> --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
```
Install the AWS Load Balancer controller, if not using iamserviceaccount
```shell script
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<k8s-cluster-name>
```
## Upgrade
The new controller is backwards compatible with the existing ingress objects. However, it will not coexist with the older aws-alb-ingress-controller.
The old controller must be uninstalled completely before installing the new version.
### Kubectl installation
If you had installed the previous version via kubectl, uninstall as follows
```shell script
$ kubectl delete deployment -n kube-system alb-ingress-controller
$ kubectl delete clusterRole alb-ingress-controller
$ kubectl delete ClusterRoleBinding alb-ingress-controller
$ kubectl delete ServiceAccount -n kube-system alb-ingress-controller
# Alternatively you can find the version of the controller and delete as follows
$ kubectl describe deployment -n kube-system alb-ingress-controller |grep Image
Image: docker.io/amazon/aws-alb-ingress-controller:v1.1.8
# You can delete the deployment now
$ kubectl delete deployment -n kube-system alb-ingress-controller
# In this case, the version is v1.1.8, the rbac roles can be removed as follows
$ kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.8/docs/examples/rbac-role.yaml
```
### Helm installation
If you had installed the incubator/aws-alb-ingress-controller Helm chart, uninstall as follows
```shell script
# NOTE: If installed under a different chart name and namespace, please specify as appropriate
$ helm delete aws-alb-ingress-controller -n kube-system
```
If you had installed the 0.1.x version of eks-charts/aws-load-balancer-controller chart earlier, the upgrade to chart version 1.0.0 will
not work due to incompatibility of the webhook api version, uninstall as follows
```shell script
$ helm delete aws-load-balancer-controller -n kube-system
```
## Uninstalling the Chart
```sh
helm delete aws-load-balancer-controller -n kube-system
```
If you setup IAM Roles for ServiceAccount, you can cleanup as follows
```
eksctl delete iamserviceaccount --cluster <cluster-name> --namespace kube-system --name aws-load-balancer-controller
```
## HA configuration
Chart release v1.2.0 and later enables high availability configuration by default.
- The default number of replicas is 2. You can pass`--set replicaCount=1` flag during chart installation to disable this. Due to leader election, only one controller will actively reconcile resources.
- The default priority class for the controller pods is `system-cluster-critical`
- Soft pod anti-affinity is enabled for controller pods with `topologyKey: kubernetes.io/hostname` if you don't configure custom affinity and set `configureDefaultAffinity` to `true`
- Pod disruption budget (PDB) has not been set by default. If you plan on running at least 2 controller pods, you can pass `--set podDisruptionBudget.maxUnavailable=1` flag during chart installation
## Configuration
The following tables lists the configurable parameters of the chart and their default values.
The default values set by the application itself can be confirmed [here](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/deploy/configurations/#controller-configuration-options).
| Parameter | Description | Default |
|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
| `image.repository` | image repository | `public.ecr.aws/eks/aws-load-balancer-controller` |
| `image.tag` | image tag | `<VERSION>` |
| `image.pullPolicy` | image pull policy | `IfNotPresent` |
| `clusterName` | Kubernetes cluster name | None |
| `cluster.dnsDomain` | DNS domain of the Kubernetes cluster, included in TLS certificate requests | `cluster.local` |
| `securityContext` | Set to security context for pod | `{}` |
| `resources` | Controller pod resource requests & limits | `{}` |
| `priorityClassName` | Controller pod priority class | system-cluster-critical |
| `nodeSelector` | Node labels for controller pod assignment | `{}` |
| `tolerations` | Controller pod toleration for taints | `{}` |
| `affinity` | Affinity for pod assignment | `{}` |
| `configureDefaultAffinity` | Configure soft pod anti-affinity if custom affinity is not configured | `true` |
| `topologySpreadConstraints` | Topology Spread Constraints for pod assignment | `{}` |
| `deploymentAnnotations` | Annotations to add to deployment | `{}` |
| `podAnnotations` | Annotations to add to each pod | `{}` |
| `podLabels` | Labels to add to each pod | `{}` |
| `additionalLabels` | Labels to add to all components | `{}` |
| `rbac.create` | if `true`, create and use RBAC resources | `true` |
| `serviceAccount.annotations` | optional annotations to add to service account | None |
| `serviceAccount.automountServiceAccountToken` | Automount API credentials for a Service Account | `true` |
| `serviceAccount.imagePullSecrets` | List of image pull secrets to add to the Service Account | `[]` |
| `serviceAccount.create` | If `true`, create a new service account | `true` |
| `serviceAccount.name` | Service account to be used | None |
| `terminationGracePeriodSeconds` | Time period for controller pod to do a graceful shutdown | 10 |
| `ingressClass` | The ingress class to satisfy | alb |
| `createIngressClassResource` | Create ingressClass resource | true |
| `ingressClassParams.name` | IngressClassParams resource's name, default to the aws load balancer controller's name | None |
| `ingressClassParams.create` | If `true`, create a new ingressClassParams | true |
| `ingressClassParams.spec` | IngressClassParams defined ingress specifications | {} |
| `region` | The AWS region for the kubernetes cluster | None |
| `vpcId` | The VPC ID for the Kubernetes cluster | None |
| `awsApiEndpoints` | Custom AWS API Endpoints | None |
| `awsApiThrottle` | Custom AWS API throttle settings | None |
| `awsMaxRetries` | Maximum retries for AWS APIs | None |
| `defaultTargetType` | Default target type. Used as the default value of the `alb.ingress.kubernetes.io/target-type` and `service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" annotations.`Possible values are `ip` and `instance`. | `instance` |
| `enablePodReadinessGateInject` | If enabled, targetHealth readiness gate will get injected to the pod spec for the matching endpoint pods | None |
| `enableShield` | Enable Shield addon for ALB | None |
| `enableWaf` | Enable WAF addon for ALB | None |
| `enableWafv2` | Enable WAF V2 addon for ALB | None |
| `ingressMaxConcurrentReconciles` | Maximum number of concurrently running reconcile loops for ingress | None |
| `logLevel` | Set the controller log level - info, debug | None |
| `metricsBindAddr` | The address the metric endpoint binds to | "" |
| `webhookBindPort` | The TCP port the Webhook server binds to | None |
| `webhookTLS.caCert` | TLS CA certificate for webhook (auto-generated if not provided) | "" |
| `webhookTLS.cert` | TLS certificate for webhook (auto-generated if not provided) | "" |
| `webhookTLS.key` | TLS private key for webhook (auto-generated if not provided) | "" |
| `webhookNamespaceSelectors` | Namespace selectors for the wekbook | None |
| `keepTLSSecret` | Reuse existing TLS Secret during chart upgrade | `true` |
| `serviceAnnotations` | Annotations to be added to the provisioned webhook service resource | `{}` |
| `serviceMaxConcurrentReconciles` | Maximum number of concurrently running reconcile loops for service | None |
| `targetgroupbindingMaxConcurrentReconciles` | Maximum number of concurrently running reconcile loops for targetGroupBinding | None |
| `targetgroupbindingMaxExponentialBackoffDelay` | Maximum duration of exponential backoff for targetGroupBinding reconcile failures | None |
| `syncPeriod` | Period at which the controller forces the repopulation of its local object stores | None |
| `watchNamespace` | Namespace the controller watches for updates to Kubernetes objects, If empty, all namespaces are watched | None |
| `disableIngressClassAnnotation` | Disables the usage of kubernetes.io/ingress.class annotation | None |
| `disableIngressGroupNameAnnotation` | Disables the usage of alb.ingress.kubernetes.io/group.name annotation | None |
| `defaultSSLPolicy` | Specifies the default SSL policy to use for HTTPS or TLS listeners | None |
| `externalManagedTags` | Specifies the list of tag keys on AWS resources that are managed externally | `[]` |
| `livenessProbe` | Liveness probe settings for the controller | (see `values.yaml`) |
| `env` | Environment variables to set for aws-load-balancer-controller pod | None |
| `hostNetwork` | If `true`, use hostNetwork | `false` |
| `dnsPolicy` | Set dnsPolicy if required | `ClusterFirst` |
| `extraVolumeMounts` | Extra volume mounts for the pod | `[]` |
| `extraVolumes` | Extra volumes for the pod | `[]` |
| `defaultTags` | Default tags to apply to all AWS resources managed by this controller | `{}` |
| `replicaCount` | Number of controller pods to run, only one will be active due to leader election | `2` |
| `podDisruptionBudget` | Limit the disruption for controller pods. Require at least 2 controller replicas and 3 worker nodes | `{}` |
| `updateStrategy` | Defines the update strategy for the deployment | `{}` |
| `enableCertManager` | If enabled, cert-manager issues the webhook certificates instead of the helm template, requires cert-manager and it's CRDs to be installed | `false` |
| `enableEndpointSlices` | If enabled, controller uses k8s EndpointSlices instead of Endpoints for IP targets | `false` |
| `enableBackendSecurityGroup` | If enabled, controller uses shared security group for backend traffic | `true` |
| `backendSecurityGroup` | Backend security group to use instead of auto created one if the feature is enabled | `` |
| `disableRestrictedSecurityGroupRules` | If disabled, controller will not specify port range restriction in the backend security group rules | `false` |
| `objectSelector.matchExpressions` | Webhook configuration to select specific pods by specifying the expression to be matched | None |
| `objectSelector.matchLabels` | Webhook configuration to select specific pods by specifying the key value label pair to be matched | None |
| `serviceMonitor.enabled` | Specifies whether a service monitor should be created, requires the ServiceMonitor CRD to be installed | `false` |
| `serviceMonitor.additionalLabels` | Labels to add to the service account | `{}` |
| `serviceMonitor.interval` | Prometheus scrape interval | `1m` |
| `serviceMonitor.namespace` | Namespace in which Prometheus is running | None |
| `clusterSecretsPermissions.allowAllSecrets` | If `true`, controller has access to all secrets in the cluster. | `false` |
| `controllerConfig.featureGates` | set of `key: value` pairs that describe AWS load balance controller features | `{}` |
| `ingressClassConfig.default` | If `true`, the ingressclass will be the default class of the cluster. | `false` |
| `enableServiceMutatorWebhook` | If `false`, disable the Service Mutator webhook which makes all new services of type LoadBalancer reconciled by the lb controller | `true` |
Reference in New Issue Copy Permalink