If you are running etcd, and need to check a few thing / see the status of your cluster, use the included hacks.
Check the Endpoint Status and DB Size
If you want to see some key details for your cluster, you can run the etcdctl:
$ etcdctl -w table endpoint status
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+
| ENDPOINT | ID |VERSION| DB SIZE |LEADER |LEARNER|RT |RAFTINDEX|RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+
| https://1.2.3.3:2379 | e97ca8fed9268702 | 3.5.3 | 128 MB | false | false | 8 | 2766616 | 2766616 | |
| https://1.2.3.2:2379 | 82c75b78b63b558b | 3.5.3 | 127 MB | true | false | 8 | 2766616 | 2766616 | |
| https://1.2.3.1:2379 | afa5e0b54513b116 | 3.5.3 | 134 MB | false | false | 8 | 2766616 | 2766616 | |
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+
Check the Revision and Count for All the Keys
If you need to see how many keys you have, you can execute the following command, and you get 5061 keys.
$ etcdctl get / --prefix --count-only=true --write-out=fields
"ClusterID" : 1232296676125618033
"MemberID" : 9423601319307597195
"Revision" : 2712993
"RaftTerm" : 8
"More" : false
"Count" : 5061
Check the 5 Highest ModRevisions for a Key/Value
If you need to find the Highest utilized (Updated), keys you can use this hack:
$ for KEY in $(etcdctl get / --prefix --keys-only=true | grep -v leases)
do
if [ ! -z "${KEY}" ]
then
COUNT=$(etcdctl get ${KEY} --prefix --write-out=fields | grep \"ModRevision\" | awk '{print $NF}')
echo "${COUNT} ${KEY}"
fi
done | sort -nr | head -n 5
2732087 /kubernetes.io/validatingwebhookconfigurations/performance-addon-operator
2731785 /kubernetes.io/resourcequotas/openshift-host-network/host-network-namespace-quotas
2731753 /kubernetes.io/validatingwebhookconfigurations/multus.openshift.io
2731549 /kubernetes.io/network.openshift.io/clusternetworks/default
2731478 /kubernetes.io/configmaps/openshift-service-ca/service-ca-controller-lock
Calculating the Theoretical Memory Pressure
Per the website, you can calculate memory pressure as:
The theoretical memory consumption of watch can be approximated with the formula:
etcd benchmark sitememory = c1 * number_of_conn + c2 * avg_number_of_stream_per_conn + c3 * avg_number_of_watch_stream
Command to be added in the future
References
- etcd Cheat Sheet
- GitHub etcd
- etcd Metrics such as etcd_debugging_mvcc_total_put_size_in_bytes
- etcd benchmarks