Skip to main content

How to setup vmoperator with only vmagent and vmservicescrape

Step by step guide on how to setup vmoperator with only vmagent and vmservicescrape to scrape your Kubernetes svcs and remote write metrics to Levitate

Introduction​

The vmoperator streamlines the deployment and management of vmagent on Kubernetes, optimizing for ease of use while retaining native configuration options inherent to Kubernetes environments. It achieves this by introducing various custom resource definitions (CRDs) into the Kubernetes ecosystem and send metrics semalessly to Levitate.

Custom Resource Definitions (CRDs)

  • vmagent
  • vmnodescrapes
  • vmservicescrapes
  • vmprobes
  • vmpodscrapes
  • vmstaticscrapes

These CRDs empower users to effortlessly create and manage vmagent instances along with scrape configurations like VMServiceScrape, VMPodScrape, etc. These configurations closely resemble the Prometheus Operator’s ServiceMonitor and PodMonitor. This eliminates the need for manual setup of vmagent deployment, image configuration, and other intricate details.

Prerequisites​

Make sure you have the following prerequisites installed:

  • kubectl - Kubernetes command-line tool.

  • Clone this repository to your local machine:

    git clone https://github.com/last9/vmagent-operator-levitate.git
    cd vmagent-operator-levitate

Install Custom Resource Definitions (CRDs)​

  1. Navigate to the crds/ directory:

    cd ./crds/

  2. Install the CRDs using kubectl:

    $ kubectl apply -f ./crd.yaml

  3. Verify that the CRDs are successfully installed:

    $ kubectl get crd --sort-by=.metadata.creationTimestamp
    # Output
    NAME                                                         CREATED AT
    vmagents.operator.victoriametrics.com                        2023-12-26T12:02:48Z
    vmnodescrapes.operator.victoriametrics.com                   2023-12-26T12:02:49Z
    vmservicescrapes.operator.victoriametrics.com                2023-12-26T12:02:50Z
    vmprobes.operator.victoriametrics.com                        2023-12-26T12:02:50Z
    vmpodscrapes.operator.victoriametrics.com                    2023-12-26T12:02:50Z
    vmstaticscrapes.operator.victoriametrics.com                 2023-12-26T12:02:51Z

    You should see the names of the installed CRDs in the output.

Install vmoperator​

  1. Navigate to the operator/ directory:

    cd ./operator/

  2. Install the operator and rbac using kubectl:

    $ kubectl apply -f ./manager.yaml -f rbac.yaml
    info

    This creates a separate last9-monitoring namespace to not clash with your existing namespaces.

  3. Verify the status of the operator:

    $ kubectl get pods -n monitoring-system
    # Output
    NAME                           READY   STATUS    RESTARTS   AGE
    vm-operator-667dfbff55-cbvkf   1/1     Running   0          101s          2023-12-26T12:02:51Z

    You should see the vmoperator pod in running status.

Install vmagent​

  1. Navigate to the vmagent/ directory:

    cd ./vmagent/

  2. You will need to obtain your Levitate cluster's Remote Write URL and its credentials. Here is a quick way to create your cluster and obtain your credentials. Run the below command to list all the placeholder values in this file vmagent.yaml

    $ cat ./vmagent.yaml | grep -n "Todo"
    # Output
    11:  levitate_cluster_username: "<levitate_cluster_username>" # Todo: append levitate cluster username
    12:  levitate_cluster_password: "<levitate_cluster_password>" # Todo: append levitate cluster password
    31:    via_cluster: <via_cluster> # Todo: add a relevant cluster name. e.g: k8s cluster name
    33:    - url: <levitate_remote_write_url> # Todo: append levitate remote write URL
    55:            storage: 20Gi # Todo: Default is 20Gi. Scale up after you have provisioned more if you need more
    58:# Todo: Below configs need to be enabled depending upon your affinity towards nodegroups.
    59:# Todo: Ensure that the below selector terms and tolerations are exactly same as the metadata of the nodegroups itself.

  3. Proceed to installation once you have replaced the placeholder values with actual values.

  4. Install vmagent in the last9-monitoring namespace using kubectl:

    $ kubectl apply -f ./vmagent.yaml -n last9-monitoring

  5. Verify the status of the vmagent and ensure that it's running:

    $ kubectl get pods -n last9-monitoring -l "last9_monitoring_agent=vmagent"
    # Output
    NAME                            READY   STATUS    RESTARTS   AGE
    vmagent-demo-6785f7d7b9-zpbv6   2/2     Running   0          72s

Install VMServiceScrape (i.e ServiceMonitor and PodMonitor)​

Navigate to the vmservicescrape/ directory:

cd ./vmservicescrape/

Caveats

VMServiceScrape works similar to Prometheus Operator's ServiceMonitor and PodMonitor where you can define scrape selectors to do service and pod discovery.

In this file vmservicescrape.yaml you can override default scrape selectors to suit your requirements. By default this assumes the default labels that are applied to the exporter as part of their installations.

Below is the generic command to find you the labels of your K8s Services.

$ kubectl get services -n <namespace> <service-name> -o jsonpath='{.metadata.labels}'

Once, you have inspected the labels for the services that you chose to scrape, you can then proceed to modify this file vmservicescrape.yaml and match the scrape selector labels with the labels of your services.

Another caveat to note here is to ensure that the namespaces are also declared correctly for the scrape selectors to correctly perform service discovery.

This file also includes scrape configs for Common Exporters such as Kafka, Redis, Node, RabbitMQ, Prometheus Pushgateway etc.

Run this command to list all the Todo comments which will guide you to customize this file vmservicescrape.yaml as required.

$ cat ./vmservicescrape.yaml | grep -n "Todo"
# Output
48:    matchNames: [ "kube-system" ] # Todo: append namespaces here
63:    matchNames: [ "kube-system" ] # Todo: append namespaces here
86:# Todo: Uncomment this if you have custom application enabled svcs running and you want to scrape them
97:#    matchNames: [ ] # Todo: Append more namespaces here
105:#      app: "" # Todo: Append app name label here
107:# Todo: Uncomment this if you have node exporters svcs running and you want to scrape them
118:#    matchNames: [ ] # Todo: Append more namespaces here
129:# Todo: Uncomment this if you have rabbitMQ exporters svcs running and you want to scrape them
140:#    matchNames: [  ] # Todo: Append more namespaces here
151:# Todo: Uncomment this if you have kafka exporters svcs running and you want to scrape them
162:#    matchNames: [  ] # Todo: Append more namespaces here
173:# Todo: Uncomment this if you have redis exporters svcs running and you want to scrape them
184:#    matchNames: [ ] # Todo: Append more namespaces here
195:# Todo: Uncomment this if you have pushgateway svcs running and you want to scrape them
206:#    matchNames: [ ] # Todo: Append more namespaces here

Install VMServiceScrape using kubectl:

$ kubectl apply -f ./vmservicescrape.yaml -n last9-monitoring
# Output
vmservicescrape.operator.victoriametrics.com/last9-vmservicescrape-vmagent-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-k8s-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-metrics-server-01 created

Troubleshooting​

Feel free to contact us through your preferred communication channels, or hop on our Discord and give us a ping.