Paul Bastide

etcdctl hacks

Written by

Paul

in

If you are running etcd, and need to check a few thing / see the status of your cluster, use the included hacks.

Check the Endpoint Status and DB Size

If you want to see some key details for your cluster, you can run the etcdctl:

$ etcdctl -w table endpoint status
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+
|        ENDPOINT      |        ID        |VERSION| DB SIZE |LEADER |LEARNER|RT |RAFTINDEX|RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+
| https://1.2.3.3:2379 | e97ca8fed9268702 | 3.5.3 |  128 MB | false | false | 8 | 2766616 | 2766616 |   |
| https://1.2.3.2:2379 | 82c75b78b63b558b | 3.5.3 |  127 MB |  true | false | 8 | 2766616 | 2766616 |   |
| https://1.2.3.1:2379 | afa5e0b54513b116 | 3.5.3 |  134 MB | false | false | 8 | 2766616 | 2766616 |   |
+----------------------+------------------+-------+---------+-------+-------+---+---------+-----+--------+

Check the Revision and Count for All the Keys

If you need to see how many keys you have, you can execute the following command, and you get 5061 keys.

 $ etcdctl get / --prefix --count-only=true --write-out=fields
"ClusterID" : 1232296676125618033
"MemberID" : 9423601319307597195
"Revision" : 2712993
"RaftTerm" : 8
"More" : false
"Count" : 5061

Check the 5 Highest ModRevisions for a Key/Value

If you need to find the Highest utilized (Updated), keys you can use this hack:

$ for KEY in $(etcdctl get / --prefix --keys-only=true | grep -v leases)
do 
    if [ ! -z "${KEY}" ]
    then 
        COUNT=$(etcdctl get ${KEY} --prefix --write-out=fields | grep \"ModRevision\" | awk '{print $NF}')
        echo "${COUNT} ${KEY}"
    fi
done | sort -nr | head -n 5

2732087 /kubernetes.io/validatingwebhookconfigurations/performance-addon-operator
2731785 /kubernetes.io/resourcequotas/openshift-host-network/host-network-namespace-quotas
2731753 /kubernetes.io/validatingwebhookconfigurations/multus.openshift.io
2731549 /kubernetes.io/network.openshift.io/clusternetworks/default
2731478 /kubernetes.io/configmaps/openshift-service-ca/service-ca-controller-lock

Calculating the Theoretical Memory Pressure

Per the website, you can calculate memory pressure as:

The theoretical memory consumption of watch can be approximated with the formula: memory = c1 * number_of_conn + c2 * avg_number_of_stream_per_conn + c3 * avg_number_of_watch_stream

etcd benchmark site

Command to be added in the future

References

More posts