Jenkins

Jenkins is an open-source automation server used primarily for continuous integration (CI) and continuous delivery (CD) in software development. It automates tasks related to building, testing, and deploying code, making it easier for teams to work together on software projects.

Jenkins is widely used in DevOps and software development workflows to speed up delivery cycles, ensure quality through automation, and reduce the need for manual intervention in repetitive tasks.

Key Features

  • Extensibility: Jenkins supports a vast array of plugins that allow for integration with various tools, including Git, Docker, and Kubernetes.
  • Distributed Builds: Jenkins can distribute workloads across multiple machines to speed up the CI/CD process.
  • Pipeline as Code: Jenkins provides a way to define CI/CD pipelines as code using Groovy-based DSL (Domain Specific Language), allowing for complex workflows.
  • Community and Plugins: Being open-source, Jenkins has a large and active community. Its plugin ecosystem allows for integration with various development, deployment, and operations tools.

Typical Use Cases

  • Build Automation: Automatically building code upon each commit.
  • Testing Automation: Running automated tests to ensure code quality.
  • Deployment Automation: Automating the process of deploying code to production or other environments.


👉 Setup Jenkins Server (step-by-step guide)

Step1: Install Java
  • Jenkins requires Java to run. First, install Java on your system.
sudo apt update -y
sudo apt upgrade -y

sudo apt install openjdk-17-jdk -y
java -version

  • Add the Jenkins repository and key
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
  
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
  • Install Jenkins
sudo apt-get update -y
sudo apt-get install jenkins

Step3: Start Jenkins (You can start/stop Jenkins service with systemctl commands)
  • Starts the Jenkins service
sudo systemctl start jenkins

  • Enable Jenkins service to start automatically when the system reboot (so always enable Jenkins service after start)

sudo systemctl enable jenkins

Disable Jenkins service from starting automatically when the system reboot but does not stop it if already running 

sudo systemctl disable jenkins
  • Check the status of Jenkins service
sudo systemctl status jenkins
  • Stop Jenkins service if already running but does not prevent it from starting on reboot
sudo systemctl stop jenkins

Step4: Access Jenkins
  • Open your web browser and go to http://your_server_ip_or_domain:8080
  • Unlock Jenkins: Jenkins will ask for an initial password. You can retrieve it using
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
  • Complete the setup wizard
    1. Install suggested plugins or select plugins manually
    2. Create your first admin user account

Step5: Configure Jenkins
Once Jenkins is up and running
  1. Configure Jenkins Global Settings: Go to Manage Jenkins > Configure System to configure system-wide settings.
  2. Set up Jenkins Nodes (Optional): If you need to configure distributed builds, go to Manage Jenkins > Manage Nodes and Clouds.
Step6: Install Plugins
Jenkins supports many plugins to extend its functionality. To install plugins,
  1. Go to Manage Jenkins > Manage Plugins.
  2. Install the required plugins (e.g., Git, Docker, Maven, etc.)


👉 Jenkins Jobs : A Jenkins job refers to a task or project that Jenkins executes to automate different aspects of software development, such as building, testing, and deploying applications.

Types of Jenkins Jobs
  • Freestyle Job:
    • The most basic type of Jenkins job.
    • Allows you to define a set of steps like shell scripts, build triggers, and post-build actions.
    • Useful for simple tasks or when flexibility is needed in defining custom build processes.
  • Pipeline Job:
    • Defines a set of tasks (or stages) that are executed in a sequence using Groovy scripts.
    • Supports both Declarative Pipelines (easier to read and maintain) and Scripted Pipelines (more flexibility for complex logic).
    • Can manage complex workflows with multiple steps, parallel processing, and integration with source control.
  • Multibranch Pipeline Job:
    • Automates the creation of pipeline jobs for branches in a source control repository.
    • Automatically discovers new branches in a repository and runs pipeline jobs on them, making it easier to handle multiple branches in a project.
  • Folder Job:
    • Organizes multiple jobs into folders.
    • Useful for managing large-scale Jenkins environments with many jobs by grouping related jobs together.
  • GitHub Organization Job:
    • Automatically sets up Jenkins jobs for every repository in a GitHub organization.
    • Jenkins will discover and create jobs for repositories and branches that contain Jenkinsfile.
  • Matrix (Multi-configuration) Job:
    • Designed for testing across multiple environments or configurations.
    • You can specify parameters (e.g., different OSes, browser types), and Jenkins will run the job with each combination.
Key Concepts in Jenkins Jobs
  • Build Triggers: Define when the job should be run, such as after a code commit, at scheduled intervals, or when another job finishes.
  • Build Steps: The individual tasks or commands executed during the job, such as compiling code, running tests, or deploying the software.
  • Post-build Actions: Actions taken after the job completes, such as sending notifications, archiving artifacts, or triggering other jobs.
Managing Jenkins Jobs
  • Jobs can be created manually via the Jenkins UI or automatically using Jenkinsfiles.
  • Jenkinsfile: A file that defines the job's pipeline in code, allowing for version control and easy collaboration.
Jenkins Jobs Advanced Features
  • Parameters:
    • Jenkins jobs can be parameterized, meaning they can accept inputs when the job is triggered (e.g., environment, branch name, or user-specified values).
    • Common parameters include string, boolean, choice, or file parameters.
    • Useful for reusing the same job across different environments or configurations.
  • Triggers and Scheduling:
    • Jobs can be triggered by:
      • SCM changes: Automatically runs when changes are committed to a source control repository (e.g., Git, Subversion).
      • Polling SCM: Jenkins periodically checks the repository for changes and triggers a job if there are updates.
      • Build after other jobs are completed: You can chain jobs together so that one job starts after another finishes.
      • Scheduled Jobs: Using CRON syntax, Jenkins jobs can run at regular intervals (e.g., nightly builds or weekly tests).
  • Build Agents (Nodes):
    • Jenkins can distribute jobs across multiple agents (nodes) to balance the load or run jobs on specific machines/environments.
    • Master is the central node where Jenkins is installed, and agents are additional machines where jobs can be run.
    • You can define labels for nodes and assign jobs to specific labels to ensure they run in the correct environment.
  • Post-Build Actions:
    • After a job is completed, Jenkins allows you to define actions based on the result of the job (success, failure, or unstable build):
      • Publish JUnit test results: Jenkins can display test results and trends over time.
      • Archive artifacts: Store important files (e.g., build outputs, logs) from a job's execution.
      • Send notifications: Jenkins can notify team members via email, Slack, or other messaging platforms about the status of a job.
      • Deploy: Automatically deploy an application to a production or testing environment.
  • Build History and Console Output:
    • Jenkins maintains a history of all job runs, providing logs for each build.
    • You can inspect the console output of each run to view logs, errors, or any information printed during the build steps.
    • Build history helps in debugging issues by allowing you to compare successful and failed runs.
  • Integrations:
    • Jenkins integrates with many tools and services via plugins, including:
      • Version control systems (Git, GitHub, Bitbucket, SVN)
      • Testing frameworks (JUnit, TestNG, Selenium)
      • CI/CD tools (Docker, Kubernetes)
      • Notification services (Slack, Email)
      • Cloud providers (AWS, Azure, Google Cloud)
    • Jenkins has a vast ecosystem of plugins to extend its functionality.
  • Blue Ocean:
    • Jenkins offers a modern user interface called Blue Ocean that provides a more user-friendly way to visualize and manage pipelines.
    • It displays a graphical representation of jobs and pipelines, showing which stages passed or failed.
  • Declarative vs Scripted Pipelines:
    • Declarative Pipelines are simpler and use a structured syntax that makes it easier for users to define pipeline jobs in a human-readable format.
    • Scripted Pipelines provide more flexibility and are written in a full-fledged Groovy script. These are useful for more complex automation scenarios.


👉 Example 1: Create a Jenkins Freestyle Job that builds a Java (Maven) Project
- You will be able to run this job in two ways, either manually by clicking "Build Now" button on Jenkins UI or it will be triggered automatically on pushing a updated code into GitHub repository.

Step 1: Configure webhook in project GitHub repository
Under project GitHub repository -> Settings -> Webhooks -> Payload URL http://<Jenkins-Server-IP>:8080/github-webhook/ -> content type application/json -> Add webhook


Step 2: Create a New Freestyle Project
  • Open Jenkins.
  • Click on "New Item."
  • Give the job a name (e.g., Build_MyApp) and select "Freestyle Project."
  • Click "OK."
Step 3: Configure Source Code Management (SCM)
  • Under "Source Code Management," select "Git."
  • Enter the GitHub repository URL https://github.com/SirajChaudhary/springboot-helloworld-service.git
  • If its private github repository, provide credentials for accessing the repository.

Step 4: Configure Build Trigger
  • Scroll down to the "Build Trigger" section.
  • Select "GitHub hook trigger for GITScm polling"

Step 5: Configure the Build Steps
  • Scroll down to the "Build Steps" section.
  • Select "Add build step" → "Invoke top-level Maven targets."
  • In the "Goals" field, enter clean install to clean the existing build and compile the project.

Step 6: Post-Build Actions
  • Scroll down to the "Post-build Actions" section.
  • Add an action like "Archive the artifacts" if you want to save build artifacts (e.g., target/*.jar files).
  • Optionally, you can configure email notifications to notify users about the build status.
Step 7: Save and Build the Job
  • Click "Save" to store the job configuration.
  • You can now either trigger the job manually by clicking "Build Now" or set up automatic triggers like:
    • Click "Build Now" or
    • Change and push the code in the GitHub repository to trigger Jenkins job automatically
Step 8: Monitor the Build
  • After triggering the build, Jenkins will display a build progress bar.
  • You can monitor the console output in real-time to see the build process, including Maven compiling your Java code, running tests, and packaging the output (usually a .jar or .war file).








👉 Example 2: Create a Jenkins Pipeline Job to automate the process of building, testing, deploying, and running a Spring Boot application on a remote machine.

-For this example we consider having two machines (EC2 instances). One for Jenkins and another for project deployment and let's call them as Jenkins Server and Remote Server respectively.

Step 1: Configure webhook in project GitHub repository
Under project GitHub repository -> Settings -> Webhooks -> Payload URL http://<Jenkins-Server-IP>:8080/github-webhook/ -> content type application/json -> Add webhook

Step 2: Install JDK and Maven in Remote Server where spring boot application will be deployed
sudo apt update -y
sudo apt upgrade -y
sudo apt install openjdk-17-jdk -y
sudo apt install maven

Step 3: Create a new Pipeline Job (Project) in Jenkins server
  • Open Jenkins.
  • Click on "New Item."
  • Give the job a name (e.g., Build_Deploy_MyApp) and select "Pipeline"
  • Click "OK."
- Add GitHub project URL
https://github.com/SirajChaudhary/springboot-helloworld-service.git

- Configure Build Trigger
Scroll down to the "Build Trigger" section.
Select "GitHub hook trigger for GITScm polling"



Note: You need to install and configure a SSH plugin 'sshagent' which is required to deploy and run application jar to remote host. I had also installed and configure 'Publish over SSH' plugin (added .pem key in its configuration).



Step 4: Write a Jenkins Pipeline groovy script (Declarative Pipeline)
Pipeline Steps Overview
  • Checkout the Code from the repository.
  • Build the Spring Boot application using Maven/Gradle.
  • Run Tests to ensure that the application is working as expected.
  • Deploy the application to a remote machine using SSH or SCP.
  • Stop (Kill process) already running application instance (port:8080) if any
  • Run the application on the remote machine.

pipeline {
    agent any
    environment {
        REMOTE_USER = 'ubuntu'
        REMOTE_HOST = '3.109.227.52'
        REMOTE_DIR = '/home/ubuntu'
        JAR_NAME = 'springboot-helloworld-service-0.0.1-SNAPSHOT.jar'
        PORT = 8080
    }
    stages {
        stage('Checkout Code') {
            steps {
                git url: 'https://github.com/SirajChaudhary/springboot-helloworld-service.git', branch: 'main'
            }
        }
        stage('Build Application') {
            steps {
                sh 'mvn clean package'
            }
        }
        stage('Run Tests') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Deploy to Remote Host') {
            steps {
                sshagent(['3.109.227.52']) {
                    // Copy the JAR file to the remote machine
                    sh """
                    
                    scp target/${JAR_NAME} ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_DIR}
                    
                    """
                }
            }
        }
        stage('Stop Running Application (Kill Process) on Remote Host') {
            steps {
                script {
                    sshagent(['3.109.227.52']) {
                        sh """
                        ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} '
                            PID=\$(lsof -t -i:${PORT})
                            if [ ! -z "\$PID" ]; then
                                echo "Killing process on port ${PORT} with PID \$PID"
                                kill -9 \$PID
                            else
                                echo "No process running on port ${PORT}"
                            fi'
                        """
                    }
                }
            }
        }
        stage('Run Application on Remote Host') {
            steps {
                sshagent(['3.109.227.52']) {
                    sh """
                    
                    ssh ${REMOTE_USER}@${REMOTE_HOST} 'nohup java -jar ${REMOTE_DIR}/${JAR_NAME} > /dev/null 2>&1 &'
                    
                    """
                }
            }
        }
    }
    post {
        success {
            echo 'Application deployed and running successfully!'
        }
        failure {
            echo 'Build or deployment failed.'
        }
    }
}


Note: If you notice here just below Pipleline Script editor there is link 'Pipeline Syntax' which you can use to generate your jenkins pipeline groovy scripts

Step 6: Run the Jenkins pipeline job either by clicking "Build Now" button on Jenkins UI or by pushing a new code into your GitHub repository so job will be triggered automatically. 






👉 Blue Ocean Plugin is a modern user interface for Jenkins, designed to simplify the process of continuous integration and continuous delivery (CI/CD). It provides an intuitive, visual interface compared to the more traditional Jenkins UI, making it easier to manage and monitor pipeline jobs

You can install the Blue Ocean plugin via the Jenkins Plugin Manager or by searching for it in Jenkins’ plugin repository. Once installed, it will be accessible via the Jenkins dashboard.





👉 Some more specific definitions and key concepts related to Jenkins

1. Jenkins Pipeline

  • Definition: A Jenkins Pipeline is a suite of plugins that support the implementation of continuous delivery pipelines. A pipeline defines the sequence of stages that a piece of code moves through in the CI/CD process, such as building, testing, and deploying.
  • Types:
    • Declarative Pipeline: A more structured format for defining pipelines using a Groovy-based DSL.
    • Scripted Pipeline: More flexible and powerful, but also more complex. It is written entirely in Groovy.

2. Jenkinsfile

  • Definition: A Jenkinsfile is a text file that stores the definition of a Jenkins Pipeline. It is usually placed in the root of a project’s repository and defines the entire CI/CD process in code. This allows version control of the build pipeline and promotes transparency.

3. Freestyle Project

  • Definition: A Freestyle Project in Jenkins is the simplest form of a build job, offering a flexible and easy way to automate basic tasks, such as running shell scripts or compiling code. Unlike Jenkins Pipelines, Freestyle Projects lack the robust support for complex workflows.

4. Build

  • Definition: In Jenkins, a build is the process of compiling and/or packaging software from source code. A build may also include steps like running automated tests and pushing artifacts to a repository.
  • Triggered by:
    • Source control commits (e.g., Git commits)
    • Scheduled tasks (e.g., cron jobs)
    • Manual triggers

5. Node (Agent)

  • Definition: A node (or agent) is a machine that Jenkins uses to run tasks. The main Jenkins instance is often referred to as the Master (or Controller), and it can distribute jobs to different nodes (agents) to balance the workload.

6. Master (Controller)

  • Definition: The Master (now often referred to as the Controller) is the central server in Jenkins, responsible for managing build processes, scheduling jobs, dispatching builds to agents (nodes), and handling their outputs.

7. Job

  • Definition: In Jenkins, a job is a task or unit of work that Jenkins performs. Jobs could be as simple as running a script, or as complex as running an entire CI/CD pipeline. Examples include Freestyle jobs or Pipeline jobs.

8. Executor

  • Definition: Executors are the resources that Jenkins uses to run builds. Each node has a set number of executors, which indicates how many jobs it can run simultaneously.

9. Plugin

  • Definition: Plugins are extensions for Jenkins that add new features or integrate with other tools. Jenkins has an extensive plugin ecosystem, with over a thousand plugins for SCM (Source Code Management), build tools, UI, notifications, and more.

10. Blue Ocean

  • Definition: Blue Ocean is an alternative user interface for Jenkins that provides a modern, intuitive way to visualize the pipeline process. It makes it easier to see the status of builds and the stages in a pipeline.

11. SCM (Source Code Management)

  • Definition: SCM in Jenkins refers to the source control systems (e.g., Git, SVN) that manage code. Jenkins can interact with these systems to check out code before building and testing.

12. Webhook

  • Definition: A webhook is an HTTP callback that Jenkins can use to trigger builds automatically. For example, a webhook can be set up to trigger a Jenkins build whenever new code is pushed to a repository.

13. Slave (Deprecated)

  • Definition: A term previously used for the nodes that run jobs delegated by the master. Jenkins now uses the term Agent instead of Slave.

14. Pipeline Stages

  • Definition: In a Jenkins pipeline, stages represent major steps in the process (e.g., Build, Test, Deploy). Each stage contains one or more steps that execute during the pipeline run.

15. Artifacts

  • Definition: Artifacts are files generated by a build, such as compiled code or test results. Jenkins can store and manage artifacts as part of the build history, making them available for future reference or deployment.
Here are some additional Jenkins-related definitions and key concepts

16. Build Trigger
  • Definition: A build trigger is an event or condition that automatically initiates a Jenkins job or pipeline. Triggers can be configured to run builds based on certain criteria, such as:
    • SCM polling: Jenkins checks the source control system at regular intervals to detect changes.
    • Webhook: A push-based trigger where an external system (e.g., GitHub) notifies Jenkins of changes, triggering a build.
    • Scheduled builds: Builds can be set to run at specific times, similar to cron jobs.
    • Manual trigger: A user can start the build manually through the Jenkins interface.
17. Pipeline Stages and Steps
  • Stages: These define logical segments of the pipeline, such as Build, Test, Deploy, and help organize the workflow.
  • Steps: Steps are the actual commands or functions that are executed within each stage. For example, a "Build" stage might have steps for compiling the code and running unit tests.
18. Workspace
  • Definition: A workspace is a directory on the Jenkins node where a job executes. It contains the checked-out source code and any files or data created during the build process. Each build has its own workspace, ensuring isolated environments.
19. Build Executor Status
  • Definition: Jenkins nodes have a certain number of executors, which indicate how many builds or jobs they can handle simultaneously. The build executor status shows how busy a node is and whether it has free capacity to take on new builds.
20. Declarative vs. Scripted Pipelines
  • Declarative Pipeline: A simpler and more readable way of writing Jenkins pipelines using a predefined structure. It is ideal for most use cases and enforces a cleaner syntax.
Example: groovy

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Test') {
      steps {
        sh 'make test'
      }
    }
  }
}
  • Scripted Pipeline: A fully programmable pipeline that provides more flexibility but requires a deep understanding of Groovy scripting. It’s generally more complex and is less structured compared to declarative pipelines.
21. Post Actions
  • Definition: Post actions are blocks within a Jenkins pipeline that define actions to take once the pipeline has finished executing, depending on the result of the build (e.g., success, failure, or unstable).
Example of post actions: groovy

post {
  always {
    echo 'This will always run'
  }
  success {
    echo 'This will run only if the build succeeds'
  }
  failure {
    echo 'This will run if the build fails'
  }
}

22. Environment Variables

  • Definition: Jenkins allows the use of environment variables within jobs and pipelines. These variables store dynamic data (such as build numbers, Git commit hashes, and user inputs) that can be used in various build steps.

Example: groovy

pipeline {
  environment {
    MY_VAR = 'Hello World'
  }
  stages {
    stage('Print') {
      steps {
        echo "${MY_VAR}"
      }
    }
  }
}

23. Multibranch Pipeline

  • Definition: A multibranch pipeline job is a special type of Jenkins job that automatically creates and runs pipelines for multiple branches in a repository. It dynamically discovers Jenkinsfiles in each branch and creates a job for each.
  • Use Case: Useful for repositories with many branches, as it ensures each branch has its own CI/CD pipeline.
24. Pipeline Libraries
  • Definition: Jenkins Pipeline libraries allow developers to define shared code that can be reused across multiple pipelines. These libraries are defined in a central repository, making it easier to manage common functions, steps, or configurations.
Example: You could create a shared library for common deployment steps and reuse it across various projects.

25. Parallel Execution

  • Definition: Jenkins allows stages or steps in a pipeline to be run in parallel. This is useful for speeding up the pipeline by running independent tasks simultaneously (e.g., different types of tests or builds).
Example: groovy

stage('Test') {
  parallel {
    stage('Unit Tests') {
      steps {
        sh 'make test-unit'
      }
    }
    stage('Integration Tests') {
      steps {
        sh 'make test-integration'
      }
    }
  }
}

26. Upstream and Downstream Jobs

  • Upstream Job: A job whose successful completion triggers another job. For example, a build job may trigger a downstream deployment job.
  • Downstream Job: A job that is triggered after an upstream job finishes. This can help create a chain of dependent jobs in a CI/CD pipeline.
27. Console Output
  • Definition: The console output in Jenkins provides a real-time view of what’s happening during the build process. It shows logs, outputs, errors, and any other messages generated by the steps within the pipeline or job.
28. Parameterized Build
  • Definition: A parameterized build is a Jenkins job or pipeline that takes input parameters when it's triggered. This allows users to customize the behavior of the build, such as providing different environment configurations or build options.
Example of parameterized Jenkinsfile: groovy

pipeline {
  parameters {
    string(name: 'ENV', defaultValue: 'dev', description: 'Target environment')
  }
  stages {
    stage('Deploy') {
      steps {
        sh "deploy --env=${params.ENV}"
      }
    }
  }
}

29. Artifacts Archive

  • Definition: Artifacts that are generated as part of a build (e.g., compiled binaries, logs, reports) can be archived in Jenkins for future use. Archiving artifacts ensures they are stored and accessible after the build process is completed.
30. Post-build Actions
  • Definition: Actions or steps that Jenkins can take after a job or build completes. Examples include:
    • Sending notifications (email, Slack, etc.)
    • Archiving build artifacts
    • Publishing reports (such as test results)
    • Triggering downstream jobs.

By