<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=160269078105920&amp;ev=PageView&amp;noscript=1">
This service has been removed!
Service successfully added!
Sign up for our blog, talk to a specialist, or just send us an email

Balancing your Jenkins Workload on a Master/Slave Setup.

Tuesday 16 of October, 2018./ Reading Time: 8 minutes./ By Alejandro Rojas

Overview.

Implementing Continuous Integration / Continuous Delivery services into our development process has become a very typical expectation when boosting our production line. It is now very common to see multiple environments required along production lines.

Jenkins is well known as an orchestration tool for CI/CD that can be extended by using nodes machines and a wide variety of plugins. This gives us some great features that help when it comes to supporting multiple jobs. However, properly balancing the workload among nodes becomes key to getting the most out of this approach.

This article will show you how to implement a load balancing approach on a master/slave Jenkins infrastructure.

Definitions.

Let’s set up a common knowledge ground to work with.

Master and Metadata.

A Jenkins Master is the machine where our Jenkins service is running. This master server will contain all the metadata related the job, like build history, job description, and steps to accomplish the designated task.

The information regarding the task itself, like files pulled from repository and compiled binaries, is stored in the project workspace (i.e. JENKINS_HOME/workspace/myjenkinsproject ), which is found inside the machine in charge of the project execution.

Execution Nodes.

Execution nodes (slaves) can give our CI/CD environment a suite of operating systems to execute what we please. A Jenkins master can have as many execution nodes as you need. There are multiple methods to connect them: SSH, Java agents or Remote Management. Machines with pretty much any OS can be used as slaves.

Both masters and nodes are able to run jobs (pipeline or freestyle projects), which is a finite set of tasks, such as pulling a repository, creating a file, building a project, etc. Execution nodes will also provide the workspace for the jobs we run.

Executors.

Jobs are built using executors. An executor is basically a process and both the master and the slaves can have any number of executors.

If we decide to give our master two executors, that means Jenkins will be able to create two different processes at any given time in order to build two different tasks. We might end up executing two tasks of the same job, in the two different executors in the same machine.

Building our scenario.

To start demonstrating the benefits of a master/slave setup, we will assume we have three Centos7 nodes connected to a Jenkins master.

You can identify your nodes based on assigned labels. Defining a label scheme is important, as this will help Jenkins delegate job execution efficiently across our platform. Aspects such as Operative System, physical location, or purpose can be used to define our label scheme.

In this case, our nodes will be identified based on the kind of technology they support, like this:

Screen Shot 2018-10-16 at 10.20.45 AM

From a label perspective, when a job or a task explicitly requests a node capable of building and running a “NodeJS” app, Jenkins should assign the execution to a node identified with a “Node JS” label (Node 1 is the only one that meets this requirement).

On the other hand, if we would like to run a task with the “Linux” label, Jenkins will pick any of the 3 nodes available, depending on the number of idle executors on each node.

Configuring Our Nodes.

Once we have defined our label scheme, we can start adding execution nodes to our Jenkins master. Let’s assume we have three Centos7 idle machines up and running on our network with Java and Git installed and we would like to use them as execution nodes.

On the left side of the Jenkins dashboard screen, we can see our nodes and add new ones. Click on Build Executor Status -> New Node.

Screen Shot 2018-10-16 at 10.22.14 AM

 

This step creates a new node instance on our Jenkins Master server (a placeholder to receive the actual node connection). We will need to add some parameter values:

Screen Shot 2018-10-16 at 10.24.34 AM

 

Screen Shot 2018-10-16 at 10.26.03 AM

 

  1. Define a name for your node (something simple but descriptive).
  2. Select “Permanent Agent”.
  3. Define the # of executors you would like on each node.
  4. Add the Labels according to the scheme we defined before.
  5. In Usage, select Only build jobs that explicitly request a node with this node’s label.

On Unix systems (Ubuntu, Centos and MacOS among others) the standard connection method is SSH. We will want to connect via SSH to our execution nodes with a user that has execution privileges in the workspace folder we are using for Jenkins (i.e. /opt/jenkins) and avoid using root, as we don’t want to expose root permissions through Jenkins jobs on our machines.

Repeat this process for the other machines you will be using as execution nodes. After that, your list of nodes should look like this:

Screen Shot 2018-10-16 at 10.27.20 AM

Testing performance without load balancing.

To test the performance request handling in Jenkins, we will use this declarative pipeline script, simulating a simple pull, with a build stage (just a sleep command) and workspace cleansing to avoid running out of disk space.

pipeline{
        agent { label 'linux'}
         stages {
                 stage("Pull Repository"){
                       steps{
                         git credentialsId: '78307373-e926-xxxx-xxxx-2b065a6a52d7',
                         url: 'http://git.url/user.name/project.git'
                         }
                    }
                    stage("Wait Stage"){
                           steps{
                               sh """
                               #!/bin/bash
                               sleep 60;
                               """
                               }
                     }
                     stage("Clean directory"){`
                             steps{
                                     deleteDir()
                               }
                       }
 

                }
}

In order to understand the performance of the execution, let’s run two scenarios:

  1. Run the job in just one node with 10 executors available, triggering 10 builds at the same time (10 executors, 1 node).
  2. Remove 5 executors on the original node, add one more node with 5 executors (2 nodes, 5 executors each) and trigger the same 10 builds at the same time.

Your results will probably look similar to the ones described in the table:

Screen Shot 2018-10-16 at 10.28.31 AM

 

Using 2 nodes instead of one decreased the average execution time of the job. What if we add a load balancing approach into our process?

Expanding the solution with Load Balancing.

As our requests for executors grow, we will have to define a criteria to assign slaves and considering performance while allocating resources on our limited stack. We will use this plugin https://plugins.jenkins.io/scoring-load-balancer for that purpose. As its name states, it will score each matching node based on defined circumstances, like having an idle executor, or if the job failed previously on a certain node.

To add this plugin, got to Manage Jenkins -> Manage Plugin, select the “Available” Tab, look for the “Scoring Load Balancer” plugin, select and install it.

Adding Logging to the Load Balancing.

Once installed, we will add a logger to our jenkins to check the scoring algorithm in action.

To add the logger:
Go to Manage Jenkins -> System log.
Then go to “New Log Recorder”.
Once there, name the log however you want.

The important part is to set the log level to “INFO” and set the logger value as “jp.ikedam.jenkins.plugins.scoringloadbalancer.ScoringLoadBalancer”.

Screen Shot 2018-10-16 at 10.30.18 AM

 

This will give us some info related to the node scores everytime we fire a build, it will be useful when debugging and understanding the node assignment.

Adding Criteria to the Load Balancing.

Go from Manage Jenkins -> Configure System and scroll to Scoring Load Balancer to set our scoring rules.

By default, the plugin offers 2 types of scoring
  • Node Based scoring: Scores each idle executor and each busy executor, multiplied by a factor.


Screen Shot 2018-10-16 at 10.31.20 AM
  • Job based scoring: Use the history of builds on each node to define where the job should be assigned. This is useful if we need to set some kind of affinity for certain nodes.

Screen Shot 2018-10-16 at 10.32.05 AM

Here we would like to penalize busy executors. As seen previously, each busy executor adds build time to each concurrent run.

By default, Jenkins doesn’t assign builds based on any criteria, so we want to make this behaviour predictable and loggable. For this example, we will be using the first scoring type but both can be used together.

Testing the solution:

The first image on the left shows five concurrent builds with no load balancer up front, which, in this case, left a completely idle node and assigned three executions to one node.

In the image on the right, Jenkins managed to assign our requests to both of the nodes equally. Jenkins could have filled all the executors on a node, leaving two idles nodes, which is exactly why we want to control the node assignment.

Screen Shot 2018-10-16 at 10.34.20 AM

 

Now, if we go to Manage Jenkins -> System Logs -> Node Score Logger, we can see the scoring for each execution.

Screen Shot 2018-10-16 at 10.35.21 AM

 

Conclusions.

In environments where a master/slave scheme applies, having a load balancer is what we want to enable proper and predictable resource usage along with good performance.

Defining a label pattern for your nodes including the information mentioned above will make an efficient usage of your CI/CD platform.

About Avantica.

If you are looking for a software partner who will work towards your own business goals and success, then Avantica is your solution. We offer dedicated teams, team augmentation, and individual projects to our clients, and are constantly looking for the best methodologies in order to give you the best results.

 

Let's Start a Project Together 

 

 

 

 

 

 

 

 

 

 

 

PREVIOUS
How Important Is Security Testing?
NEXT
Integration of Stand-Alone QA Teams Into Mature Software Development Companies.