Skip to main content

Open Telemetry

This laboratory focuses on installing the OpenTelemetry collector into that cluster and forwarding metrics to Amazon Cloudwatch.

Requirements​

To configure observability you will need:

  • Amazon EKS cluster running Retool
  • OpenTelemetry agent
  • IAM Policy(s) to PutEvent to Amazon CloudWatch

The following steps are outlined in the previous lab Self-hosted / Kubernetes / Kubernetes Helm Installation:

  • Spinning up an Amazon EKS Cluster using Terraform
  • Deploying Retool without an Ingress via Helm
  • Deploying Let’s Encrypt infrastructure for SSL/TLS certificates
  • Deploying NGINX Ingress Controller with an Ingress manifest that uses Let’s Encrypt
  • Updating Amazon Route53 to have an A Record with a domain
  • Updating Amazon Route53 A Record to direct traffic to ELB

Steps​

Install the AWS Distro for OpenTelemetry (ADOT) as a DaemonSet using the following download / apply command:

curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml |
kubectl apply -f -

This should result in the following creation activites:

namespace/aws-otel-eks created
serviceaccount/aws-otel-sa created
clusterrole.rbac.authorization.k8s.io/aoc-agent-role created
clusterrolebinding.rbac.authorization.k8s.io/aoc-agent-role-binding created
configmap/otel-agent-conf created
daemonset.apps/aws-otel-eks-ci created

The next step is to update the IAM role on Amazon EKS Cluster Managed Worker Nodes to include the CloudWatchAgentServerPolicy.

  • Go to AWS Console > Amazon EC2 and select a Worker Node Instance.
  • Select Actions > Security > Modify IAM Role.
  • Select Add Permissions and select the check box near CloudWatchAgentServerPolicy and then select Attach Policies.

Confirm ADOT Collector is running on each EKS node:

kubectl get pods -l name=aws-otel-eks-ci -n aws-otel-eks

This will display n-number of pods.

NAME                    READY   STATUS    RESTARTS   AGE
aws-otel-eks-ci-bwcp5   1/1     Running   0          3m59s
aws-otel-eks-ci-h42qw   1/1     Running   0          3m59s

Confirm successful execution of the ADOT Collector by examining the logs.

kubectl logs <pod_name> -n aws-otel-eks

View the container insights using Amazon CloudWatch > Insights > Container Insights to examine available metrics for Managed Worker Nodes and Pods deployed. The following image shows node_cpu_utilization, pod_cpu_utilization, node_number_of_running_containers, and node_memory_utilization.

CSV Import

Cleanup of Retool Installation​

The following steps can be used to cleanup the Open Telemetry install.

Delete DaemonSet​

kubectl get daemonset -n aws-otel-eks
...
kubectl delete daemonset <name> -n aws-otel-eks