Setup vmoperator with only vmagent and vmservicescrape
Step by step guide on how to setup vmoperator with only vmagent and vmservicescrape to scrape your Kubernetes svcs and remote write metrics to Levitate
Introduction
The vmoperator streamlines the deployment and management of vmagent on Kubernetes, optimizing for ease of use while retaining native configuration options inherent to Kubernetes environments. It achieves this by introducing various custom resource definitions (CRDs) into the Kubernetes ecosystem and send metrics semalessly to Levitate.
Custom Resource Definitions (CRDs)
- vmagent
- vmnodescrapes
- vmservicescrapes
- vmprobes
- vmpodscrapes
- vmstaticscrapes
These CRDs empower users to effortlessly create and manage vmagent instances along with scrape configurations like VMServiceScrape, VMPodScrape, etc. These configurations closely resemble the Prometheus Operator’s ServiceMonitor and PodMonitor. This eliminates the need for manual setup of vmagent deployment, image configuration, and other intricate details.
Prerequisites
Make sure you have the following prerequisites installed:
-
kubectl - Kubernetes command-line tool
-
Clone this repository to your local machine:
git clone https://github.com/last9/vmagent-operator-levitate.git
cd vmagent-operator-levitate
Install Custom Resource Definitions (CRDs)
-
Navigate to the
crds/
directory:cd ./crds/
-
Install the CRDs using
kubectl
:$ kubectl apply -f ./crd.yaml
-
Verify that the CRDs are successfully installed:
$ kubectl get crd --sort-by=.metadata.creationTimestamp
# Output
NAME CREATED AT
vmagents.operator.victoriametrics.com 2023-12-26T12:02:48Z
vmnodescrapes.operator.victoriametrics.com 2023-12-26T12:02:49Z
vmservicescrapes.operator.victoriametrics.com 2023-12-26T12:02:50Z
vmprobes.operator.victoriametrics.com 2023-12-26T12:02:50Z
vmpodscrapes.operator.victoriametrics.com 2023-12-26T12:02:50Z
vmstaticscrapes.operator.victoriametrics.com 2023-12-26T12:02:51ZYou should see the names of the installed CRDs in the output.
Install vmoperator
-
Navigate to the
operator/
directory:cd ./operator/
-
Install the operator and rbac using
kubectl
:$ kubectl apply -f ./manager.yaml -f rbac.yaml
infoThis creates a separate
last9-monitoring
namespace to not clash with your existing namespaces. -
Verify the status of the operator:
$ kubectl get pods -n monitoring-system
# Output
NAME READY STATUS RESTARTS AGE
vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s 2023-12-26T12:02:51ZYou should see the vmoperator pod in running status.
Install vmagent
-
Navigate to the
vmagent/
directory:cd ./vmagent/
-
You will need to obtain your Levitate cluster's Remote Write URL and its credentials. Here is a quick way to create your cluster and obtain your credentials. Run the below command to list all the placeholder values in this file vmagent.yaml
$ cat ./vmagent.yaml | grep -n "Todo"
# Output
11: levitate_cluster_username: "<levitate_cluster_username>" # Todo: append levitate cluster username
12: levitate_cluster_password: "<levitate_cluster_password>" # Todo: append levitate cluster password
31: via_cluster: <via_cluster> # Todo: add a relevant cluster name. e.g: k8s cluster name
33: - url: <levitate_remote_write_url> # Todo: append levitate remote write URL
55: storage: 20Gi # Todo: Default is 20Gi. Scale up after you have provisioned more if you need more
58:# Todo: Below configs need to be enabled depending upon your affinity towards nodegroups.
59:# Todo: Ensure that the below selector terms and tolerations are exactly same as the metadata of the nodegroups itself. -
Proceed to installation once you have replaced the placeholder values with actual values
-
Install vmagent in the
last9-monitoring
namespace usingkubectl
:$ kubectl apply -f ./vmagent.yaml -n last9-monitoring
-
Verify the status of the vmagent and ensure that it's running:
$ kubectl get pods -n last9-monitoring -l "last9_monitoring_agent=vmagent"
# Output
NAME READY STATUS RESTARTS AGE
vmagent-demo-6785f7d7b9-zpbv6 2/2 Running 0 72s
Install VMServiceScrape (i.e ServiceMonitor and PodMonitor)
Navigate to the vmservicescrape/
directory:
cd ./vmservicescrape/
Caveats
VMServiceScrape works similar to Prometheus Operator's ServiceMonitor and PodMonitor where you can define scrape selectors to do service and pod discovery.
In this file vmservicescrape.yaml you can override default scrape selectors to suit your requirements. By default this assumes the default labels that are applied to the exporter as part of their installations.
Below is the generic command to find you the labels of your K8s Services.
$ kubectl get services -n <namespace> <service-name> -o jsonpath='{.metadata.labels}'
Once, you have inspected the labels for the services that you chose to scrape, you can then proceed to modify this file vmservicescrape.yaml and match the scrape selector labels with the labels of your services.
Another caveat to note here is to ensure that the namespaces are also declared correctly for the scrape selectors to correctly perform service discovery.
This file also includes scrape configs for Common Exporters such as Kafka, Redis, Node, RabbitMQ, Prometheus Pushgateway etc.
Run this command to list all the Todo
comments which will guide you to customize this
file vmservicescrape.yaml
as required.
$ cat ./vmservicescrape.yaml | grep -n "Todo"
# Output
48: matchNames: [ "kube-system" ] # Todo: append namespaces here
63: matchNames: [ "kube-system" ] # Todo: append namespaces here
86:# Todo: Uncomment this if you have custom application enabled svcs running and you want to scrape them
97:# matchNames: [ ] # Todo: Append more namespaces here
105:# app: "" # Todo: Append app name label here
107:# Todo: Uncomment this if you have node exporters svcs running and you want to scrape them
118:# matchNames: [ ] # Todo: Append more namespaces here
129:# Todo: Uncomment this if you have rabbitMQ exporters svcs running and you want to scrape them
140:# matchNames: [ ] # Todo: Append more namespaces here
151:# Todo: Uncomment this if you have kafka exporters svcs running and you want to scrape them
162:# matchNames: [ ] # Todo: Append more namespaces here
173:# Todo: Uncomment this if you have redis exporters svcs running and you want to scrape them
184:# matchNames: [ ] # Todo: Append more namespaces here
195:# Todo: Uncomment this if you have pushgateway svcs running and you want to scrape them
206:# matchNames: [ ] # Todo: Append more namespaces here
Install VMServiceScrape using kubectl
:
$ kubectl apply -f ./vmservicescrape.yaml -n last9-monitoring
# Output
vmservicescrape.operator.victoriametrics.com/last9-vmservicescrape-vmagent-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-k8s-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-metrics-server-01 created
Troubleshooting
Feel free to contact us through your preferred communication channels, or hop on our Discord and give us a ping.