Logging and Troubleshooting

Exercise 12.1: Review Log File Locations

Overview

In addition to various logs files and command output, you can use journalctl to view logs from the node perspective. We will view common locations of log files, then a command to view container logs. There are other logging options, such as the use of a sidecar container dedicated to loading the logs of another container in a pod.

Whole cluster logging is not yet available with Kubernetes. Outside software is typically used, such as Fluentd, part of https://fluentd.org/, which is another member project of CNCF, like Kubernetes.

Review Log File Locations

Take a quick look at the following log files and web sites. As server processes move from node level to running in containers the logging also moves.

1. If using a systemd based Kubernetes cluster view the node level logs for kubelet, the local Kubernetes agent. Each node will have different contents as this is node specific.

   
   student@lfs458-node-1a0a:~$ journalctl -u kubelet |less
   <output_omitted>
   

2. Major Kubernetes processes now run in containers. You can view them from the container or the pod perspective. Use the find command to locate the kube-apiserver log. Your output will be different, but will be very long. Once you locate the files use the diff utility to compare them. There should be no difference, as they are symbolic links to /var/log/pods/. If you follow the links the log files are unique.

   
   student@lfs458-node-1a0a:~$ sudo find / -name "*apiserver*log"
   /var/log/containers/kube-apiserver-u16-12-1-dcb8_kube-system_kube-apiserver-
   eddae7079382cd382cd55f8f46b192565dd16b6858206039d49b1ad4693c2a10.log
   /var/log/containers/kube-apiserver-u16-12-1-dcb8_kube-system_kube-apiserver-
   d00a48877af4ed4c7f8eedf2c7805c77cfabb31fcb453f7d89ffa52fc6ea5f36.log
   
   student@lfs458-node-1a0a:~$ sudo diff /var/log/containers/kube-apiserver-u16-
   12-1-dcb8_kube-system_kube-apiserver-eddae7079382cd382cd55f8f46b192565dd16b68
   58206039d49b1ad4693c2a10.log /var/log/containers/kube-apiserver-u16-12-1-
   dcb8_kube-system_kube-apiserver-d00a48877af4ed4c7f8eedf2c7805c77cfabb31fcb453
   f7d89ffa52fc6ea5f36.log
   <output_omitted>
   

3. Take a look at the log file.

   
   student@lfs458-node-1a0a:~$ sudo less /var/log/containers/kube-apiserver-u16-
   12-1-dcb8_kube-system_kube-apiserver-d00a48877af4ed4c7f8eedf2c7805c77cfabb31f
   cb453f7d89ffa52fc6ea5f36.log
   

4. Search for and review other log files for kube-dns, kube-flannel, and kube-proxy.
5. If not on a Kubernetes cluster using systemd you can view the text files on the master node.
   1. /var/log/kube-apiserver.log
      Responsible for serving the API
   2. /var/log/kube-scheduler.log
      Responsible for making scheduling decisions
   3. /var/log/kube-controller-manager.log
      Controller that manages replication controllers
6. /var/log/containers
   Various container logs
7. /var/log/pods/
   More log files for current Pods
8. Worker Nodes Files (on non-systemd systems)
   1. /var/log/kubelet.log
      Responsible for running containers on the node
   2. /var/log/kube-proxy.log
      Responsible for service load balancing
9. More reading: https://kubernetes.io/docs/tasks/debug-application-cluster/\debug-service/ and https: //kubernetes.io/docs/tasks/debug-application-cluster/\determine-reason-pod-failure/

Exercise 12.2: Viewing Logs Output

Container standard out can be seen via the kubectl logs command. If there is no standard out, you would not see any output.In addition, the logs would be destroyed if the container is destroyed.

1. View the current Pods in the cluster. Be sure to view Pods in all namespaces.

   
   student@lfs458-node-1a0a:~$ kubectl get po --all-namespaces
   NAMESPACE 	NAME 					READY 	STATUS 		RESTARTS 	AGE
   default 	ds-one-qc72k 				1/1 	Running 	0 		3h
   default 	ds-one-z31r4 				1/1 	Running 	0 		3h
   ....
   kube-system 	etcd-lfs458-node-1a0a 			1/1 	Running 	2 		9h
   kube-system	kube-apiserver-lfs458-node-1a0a 	1/1 	Running 	2 		9h
   kube-system 	kube-controller-manager-lfs458-node-1a0a 1/1 	Running 	2 		9h
   kube-system 	kube-dns-2425271678-w80vx 		3/3 	Running 	6 		9h
   kube-system 	kube-scheduler-lfs458-node-1a0a 	1/1 	Running 	2 		9h
   

2. View the logs associated with various infrastructure pods. Using the Tab key you can get a list and choose a container. Then you can start typing the name of a pod and use Tab to complete the name.

   
   student@lfs458-node-1a0a:~$ kubectl -n kube-system logs <Tab><Tab>
   calico-etcd-n6h2q
   etcd-lfs458-1-11-1update-cm35
   calico-kube-controllers-74b888b647-9ds42
   kube-apiserver-lfs458-1-11-1update-cm35
   calico-node-6j8hc
   kube-controller-manager-lfs458-1-11-1update-cm35
   calico-node-dq6kf
   kube-proxy-8sn6f
   coredns-78fcdf6894-7fpfp
   kube-proxy-wf5dr
   coredns-78fcdf6894-g6k99
   kube-scheduler-lfs458-1-11-1update-cm35
   
   student@lfs458-node-1a0a:~$ kubectl -n kube-system logs \
   	kube-apiserver-lfs458-1-11-1update-cm35
   Flag --insecure-port has been deprecated, This flag will be
   removed in a future version.
   I0729 21:29:23.026394 	1 server.go:703] external host
   was not specified, using 10.128.0.2
   I0729 21:29:23.026667 	1 server.go:145] Version: v1.11.1
   I0729 21:29:23.784000 	1 plugins.go:158] Loaded 8 mutating
   admission controller(s) successfully in the following order:
   NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,
   Priority,DefaultTolerationSeconds,DefaultStorageClass,
   MutatingAdmissionWebhook.
   I0729 21:29:23.784025	 1 plugins.go:161] Loaded 6 validating
   admission controller(s) successfully in the following order:
   LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,
   ValidatingAdmissionWebhook,ResourceQuota.
   <output_omitted>
   

3. View the logs of other Pods in your cluster.

---

Avail Rajesh Kumar as trainer at 50% Discount Puppet Online Training Enroll Now Puppet Classroom TrainingEnroll Now